Thermal imaging with view-point correction

Abstract

To solve a problem that it is possibly a task liable to spend time and ready to cause errors to image industrial property in order to assure a correct operation of industrial property or identify a trouble.SOLUTION: Property is made possible to be swiftly and correctly identified by automating an imaging-capture process and identifying property with utilization of a neural network, and this is true for the current operation state of them. In order to identify property, determine whether or not property is operating in an anticipated parameter or, if not operating in an anticipated parameter, start a corrective action, property imaging may be performed through visual imaging, infrared imaging or a combination thereof.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention generally relates to systems and methods for imaging industrial assets, and more particularly to using neural networks, machine learning models, and classical computer vision to analyze infrared and / or visual images to identify and evaluate industrial assets.

Background Art

[0002] Imaging industrial assets (or more simply "assets") provides a means for identifying a particular asset or type of asset and some characteristics of the asset. Characteristics may include those presented in the visible and non-visible spectra of the asset itself as well as identifying attributes of the asset (e.g., leaks, operating / non-operating, location of components, attached equipment, etc.). Infrared imaging can be useful for identifying the temperature of an asset or part of an asset.

[0003] In many facilities (e.g., factories, chemical plants, oil refineries, etc.), users are tasked with periodically taking images of assets, such as daily. Other assets may require more or less frequent imaging. In facilities with a large number of assets such as machines, pumps, etc., this can be an important task.

[0004] When capturing an image of an asset, the user may perform a cursory inspection, such as looking at a display screen presenting the captured image. When the imaging is infrared, the user may determine whether the image was captured correctly and whether the expected hot spots are shown in the image. The user may then record such results and / or associate a particular image with a particular asset.

[0005] Typically, an image is downloaded from a camera to another computer for subsequent storage and analysis. For example, a user may draw a bounding box around an area of interest that comprises a portion of an asset. One asset may have one or more bounding boxes. The user may then observe the image or a particular bounding box to look for anomalies in each image.

[0006] Such techniques enable many facilities to operate as designed and catch problems early, but imaging assets for analysis is a time-consuming and error-prone process. Additionally, changes due to environmental or other factors, including changes over time, are difficult for a human user to accurately account for when determining whether an asset is functioning as expected. SUMMARY OF THE INVENTION PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] To identify assets and / or the operating attributes of assets, visual spectral imaging of the assets may be used. The imaging may identify a particular asset being imaged (e.g., an identifier, model, and serial number, etc.) or the category or type of the asset. Thermal imaging (i.e., infrared) of the assets can provide additional, alternative, or supplementary information compared to visual imaging. Thermal imaging can assist in identifying an asset when heat from the motor of the asset at a certain location is identified as being able to match the first asset or the first type of asset, as opposed to detecting heat from a motor located at a second location that matches a second asset or a second type of asset. Thermal imaging can also identify parts of an asset based on thermal characteristics, such as the motor part of a pump or the bearings of a shaft driven by the pump. Thermal imaging of the assets may be used to determine the current state of the assets, such as the operating state, standby state, transient state, steady state, or a state that changes over time, and whether the asset or a part of the asset is operating normally or abnormally. The state of the assets may be determined by thermal imaging of the assets when generating thermal images that are actively operating and indicative of their operations, such as in a standby state where heat generated by a motor or other electrical equipment and / or heat generated by supply inventory or shipping inventory is detected, or an endothermic or exothermic reaction within the asset when the supply inventory is converted to shipping inventory, or the temperature characteristics of the heater / cooler part used to prepare for the transition to the operating state, etc.

[0008] Being able to quickly and accurately image assets is often difficult due to the time available to capture an image, variations in the imaging device, and variations in the use of the imaging device. Further, there are usually a vast amount of different types and numbers of assets in an industrial plant, and it can be extremely difficult / impossible for a human to identify which area / region of an asset should be imaged (e.g., for temperature monitoring) and / or for other purposes. An asset may be imaged from one location with respect to that asset and then from another location. For example, subsequent images may vary with respect to the imaging location, the height of the imaging device, the zoom setting, the angle, the rotation, the vibration, etc. Thermal imaging may show variations due to the operation of the asset (e.g., the load in the asset, the most recent startup with respect to an ongoing operation, the supply inventory attributes, etc.). As a result, the images taken over time vary, complicating the analysis of the images for identifying abnormal behavior.

[0009] These and other needs are addressed by various embodiments and configurations of the present invention. The present invention can provide several advantages depending on the particular configuration. These and other advantages will become apparent from the present disclosure of the present invention contained herein.

[0010] In one embodiment, computer vision algorithms, deep learning neural networks, and machine learning models are used to improve the accuracy of temperature calculations and to account for differences in imaging (e.g., angle, rotation). A trained neural network is provided to enable subsequent operations of the asset, such as automated image processing for identifying the asset and / or identifying the operating attributes of the asset, and continuing the current operation, scheduling maintenance, or performing an emergency operation (e.g., a shutdown).

[0011] Accurately imaging assets in a real-world environment, such as large factories, chemical plants, oil distillation facilities, etc., is error-prone. Some assets are portable and may be moved daily, while others are fixed but may operate in a dynamic environment. The surrounding environment may change seasonally, etc., or the production environment may change, such as when processing different supply inventories or operating under different loads, for example, when processing one supply inventory while another is being processed on the production line.

[0012] The imaging challenge is further complicated by inconsistent image capture. Variations may occur due to height, angle, zoom settings, etc. Therefore, in imaging where assets and their associated areas (e.g., areas of interest) are automatically captured and identified for temperature monitoring, asset management, and / or other purposes, it is desirable to establish a method or system that can take into account all those variations. Converting raw images (e.g., captured thermal images) to processed images may be necessary to mitigate imaging errors. For example, after comparison with a reference image / template image related to the asset, the thermal image can be transformed / processed through several conversions. The conversions may include affine transformations and homographic transformations.

[0013] Given two different images (e.g., a first image and a second image), techniques such as image alignment can be used to align the two images. The two different images may be taken from different heights, angles, zooms, or origins, but may contain a common object within the image. One or more common assets are identified in both images, and image alignment may be used to automatically align the coordinate space of the second image with the coordinate space of the first image. During the alignment process, a transformation that maps the images to each other (e.g., maps the second image to the first image) is found. This transformation can include an affine transformation or a homography transformation.

[0014] Since the transformed image maintains straight lines and parallelism, affine transformations are commonly used in image processing, computer vision applications, and computer graphics. Affine transformations preserve parallel lines, the ratio of distances, and the ratio of areas. Affine transformations include translation, rotation, scaling, and shearing.

[0015] In a homographic transformation, the homography matrix (sometimes called a projective transformation or a perspective transformation) is a more general transformation that can handle both affine and perspective transformations. Perspective transformations introduce vanishing points and do not necessarily preserve parallel lines, collinearity, or the ratio of areas.

[0016] Both affine transformations and homography matrices are represented by 3×3 matrices. The main difference is that the elements of the homography matrix can take any non-zero value, while some of the elements in the affine transformation matrix have specific constraints, which means a smaller degree of freedom for affine transformations.

[0017] In some other embodiments, other transformations may be utilized, such as thin-plate spline (TPS) transformation, piecewise affine transformation (warping), radial basis function (RBF) transformation, projective transformation, similarity transformation (i.e., scaling), Euclidean transformation, and / or deformable image registration transformation.

Means for Solving the Problem

[0018] In some aspects, the techniques described herein relate to a device for monitoring the temperature of assets in an environment, the device comprising: a conveyance device that is instructed to patrol the environment and that moves autonomously or semi-autonomously within the environment; a thermal image capture device disposed on the conveyance device that assists in the autonomous capture of a thermal image of the environment; and a processor configured to process the thermal image to identify an asset from a database of assets and to determine at least one region of interest of the asset where a temperature measurement is to be performed, wherein the conveyance device is configured to move according to the results from the processing of the thermal image.

[0019] In some aspects, the techniques described herein relate to a device, wherein the conveyance device is configured to move autonomously such that an image capture is performed at an angle suitable for a successful processing of the thermal image.

[0020] In some aspects, the techniques described herein relate to a device, wherein the conveyance device is configured to move autonomously to adjust the distance between the asset and the thermal image capture device for a successful processing of the thermal image.

[0021] In some aspects, the techniques described herein relate to a device, wherein the conveyance device is configured to move autonomously to adjust the position of the thermal image capture device for a successful processing of the thermal image according to machine learning.

[0022] In some aspects, the techniques described herein relate to a device, wherein the conveyance device being configured to move according to the results from the processing of the thermal image includes moving an internal portion of the conveyance device that is configured to hold the thermal image capture device while remaining stationary at a predetermined position.

[0023] In some aspects, the techniques described herein relate to a device, wherein the thermal image includes an integrated image from a visual image capture device and the thermal image capture device.

[0024] In some aspects, the techniques described herein relate to a method for monitoring the temperature of assets located within an environment, the method comprising receiving a thermal image of the environment containing the assets by using a thermal image capture device disposed on a conveyance device instructed to patrol the environment, wherein the conveyance device moves autonomously or semi-autonomously within the environment, identifying the assets from a database of assets, determining at least one region of interest of the assets where temperature measurements are to be taken, adjusting the movement of the conveyance device for successful processing of the thermal image, and calculating at least one temperature value of the assets.

[0025] In some aspects, the techniques described herein further relate to a method comprising providing the thermal image to a machine learning model and receiving an output from a machine learning network in response to the machine learning model related to the processed result of the thermal image being at least partially based on a detection model, the output including one or more regions of interest of the assets.

[0026] In some aspects, the techniques described herein relate to a method, wherein the detection model includes a region of interest (ROI) detection model trained at least partially based on a set of base thermal images related to the type of the assets, which are prepared before capturing the thermal image.

[0027] In some aspects, the techniques described herein relate to a method, wherein each of the one or more regions of interest is prepared from one or more predetermined image positions within the thermal image.

[0028] In some aspects, the techniques described herein relate to a method, wherein identifying and determining the temperature for one region of interest of the assets includes calculating a temperature value based on the pixel values of each of the one or more regions of interest.

[0029] In some aspects, the techniques described herein relate to a method, wherein the temperature includes at least one of a maximum temperature value, a minimum temperature value, an average temperature value, or a median temperature value.

[0030] In some embodiments, the techniques described herein relate to a method, the thermal image includes at least one object located within an environment that is used for asset identification purposes or for identifying the location of an asset, and the processed result is obtained by mapping at least one object included in a portion of the thermal image to a reference image, at least partially based on a machine learning model.

[0031] In some embodiments, the techniques described herein relate to a method, the reference image is obtained at least partially based on a machine learning network, the machine learning model is trained based on a set of images related to an asset, and the set of images is transformed via at least one of cropping, blurring, rotating, or adjusting the brightness.

[0032] In some embodiments, the techniques described herein relate to a method, the reference image includes one or more regions of interest, and each of the one or more regions of interest corresponds to a predetermined portion of an asset.

[0033] In some embodiments, the techniques described herein relate to a method, the reference image includes a digitally created image for asset identification purposes that is significantly reduced in size compared to a corresponding color image in order to speed up the identification process.

[0034] In some embodiments, the techniques described herein relate to a method, and the reference image is created from a machine learning model.

[0035] In some embodiments, the techniques described herein relate to a method, and mapping at least one object includes mapping the image coordinates of the thermal image to the image coordinates of the reference image, at least partially based on a machine learning model.

[0036] In some embodiments, the techniques described herein relate to a method, and mapping is performed based on a transformation of the captured thermal image via a projective transformation.

[0037] In some embodiments, the techniques described herein relate to methods, and the affine transformation includes a homography matrix transformation.

[0038] In some embodiments, the techniques described herein relate to methods, and the conveyance device is configured to move according to whether mapping is successfully achieved after transformation.

[0039] In some embodiments, the techniques described herein relate to methods, and mapping at least one object includes a remapping process in which the image coordinates of the thermal image are updated to a different set of coordinates, resulting in different regions of the thermal image being mapped.

[0040] In some embodiments, the techniques described herein relate to methods, and the conveyance device is configured to adjust the angle of image capture according to the result of the processed thermal image.

[0041] In some embodiments, the techniques described herein relate to methods, and the conveyance device is configured to adjust the distance of image capture according to the result of the processed thermal image.

[0042] In some embodiments, the techniques described herein relate to methods, and the conveyance device is configured to adjust the location of image capture according to the result of the processed thermal image.

[0043] In some embodiments, the techniques described herein relate to methods, and further include capturing a color image of the environment by using an additional image capture device disposed on the conveyance device.

[0044] In some embodiments, the techniques described herein relate to methods, and identifying an asset from an asset database includes processing a color image.

[0045] In some aspects, the techniques described herein relate to methods, and determining at least one area of interest of an asset includes processing a color image.

[0046] In some aspects, the techniques described herein relate to methods, and a thermal image includes an integrated image from a visual image capture device and a thermal image capture device.

[0047] In some aspects, the techniques described herein relate to methods, and identifying an asset from a database of assets includes processing a thermal image.

[0048] In some aspects, the techniques described herein relate to methods, and determining at least one area of interest of an asset includes processing a thermal image.

[0049] In some aspects, the techniques described herein relate to methods, and identifying an asset from a database of assets includes mapping a thermal image to an image template generated based on a machine learning model.

[0050] In some aspects, the techniques described herein relate to methods, and calculating at least one temperature value of an asset includes processing an ambient temperature of an environment in which the asset is located.

[0051] In some aspects, the techniques described herein relate to methods, and calculating at least one temperature value of an asset includes calculating an operating time indicating how long the asset has been in an operating mode.

[0052] In some aspects, the techniques described herein relate to a computer-implemented method for training a neural network for asset detection, comprising collecting a set of digital asset images from a database, and mirroring, rotating, smoothing, sharpening, blurring, increasing contrast, increasing saturation, increasing brightness, decreasing contrast, decreasing saturation, decreasing brightness, cropping, zooming in, or zooming out, to create a modified set of digital asset images, applying one or more transformations to each digital asset image, creating a first training set comprising the collected set of digital asset images, the modified set of digital asset images, and a set of digital non-asset images, training a neural network in a first stage using the first training set, creating a second training set for a second stage of training comprising the first training set and digital non-asset images that are erroneously detected as asset images after the first stage of training, and training the neural network in the second stage using the second training set.

[0053] A system-on-chip (SoC) comprising any one or more of the above aspects or aspects of the embodiments described herein.

[0054] One or more means for performing any one or more of the above aspects or aspects of the embodiments described herein.

[0055] Any aspect in combination with any one or more other aspects.

[0056] Any one or more of the features disclosed herein.

[0057] Any one or more of the features substantially as disclosed herein.

[0058] One or more of the features substantially disclosed herein, in combination with one or more of any other features substantially as disclosed herein.

[0059] Any one of the aspects / features / embodiments, in combination with one or more of any other aspects / features / embodiments.

[0060] Use of one or more of the aspects or features as disclosed herein.

[0061] Any of the above aspects in which the data storage comprises a non-transitory memory device may further comprise at least one of on-chip memory within the processor, registers of the processor, on-board memory placed together on a processing substrate with the processor, memory accessible to the processor via a bus, magnetic media, optical media, solid-state media, input / output buffers, memory of input / output components communicating with the processor, network communication buffers, and network-type components communicating with the processor via a network interface.

[0062] It should be understood that any feature described herein may be claimed in combination with such other features, regardless of whether such other features are from the same embodiment described.

[0063] The phrases "at least one of", "one or more of", "or", and "and / or" are open-ended expressions that are both conjunctive and disjunctive in operation. For example, each of the expressions "at least one of A, B, and C", "at least one of A, B, or C", "one or more of A, B, and C", "one or more of A, B, or C", "A, B, and / or C", and "A, B, or C" means only A, only B, only C, A and B together, A and C together, B and C together, or A, B, and C together.

[0064] The term "a" or "an" entity refers to one or more of that entity. Thus, the terms "a" (or "an"), "one or more", and "at least one" may be used interchangeably herein. Note also that the terms "comprising", "including", and "having" may be used interchangeably.

[0065] As used herein, the term "automatic" and variations thereof refer to any process or operation that is typically continuous or semi - continuous and is performed without material human input when the process or operation is executed. However, if input is received prior to the execution of the process or operation, the process or operation may be automatic even if it uses material or immaterial human input. Human input is considered to be material if such input affects how the process or operation is executed. Human input in response to the execution of a process or operation is not considered to be "material".

[0066] Aspects of the present disclosure may take the form of embodiments that are entirely hardware, embodiments that are entirely software (including firmware, resident software, microcode, etc.), or embodiments that combine software aspects and hardware aspects that may generally be referred to herein as "circuits", "modules", or "systems". Any combination of one or more computer - readable media may be utilized. The computer - readable media may be a computer - readable signal medium or a computer - readable storage medium.

[0067] A computer-readable storage medium may be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples (non-exhaustive listing) of computer-readable storage media include the following, namely, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this specification, a computer-readable storage medium may be any tangible non-transitory medium that can include or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0068] A computer-readable signal medium may include a propagated data signal having computer-readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including but not limited to electromagnetic form, optical form, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The program code embodied on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0069] As used herein, the terms "determine," "calculate," "compute," and variations thereof are used interchangeably and include any type of method, process, mathematical operation, or technique.

[0070] As used herein, the term "means" shall be given its broadest possible interpretation under 35 U.S.C. § 112(f) and / or § 112, paragraph 6. Accordingly, claims incorporating the term "means" shall cover all structures, materials, or acts described herein, and all equivalents thereof. Further, structures, materials, or acts, and equivalents thereof, shall include all those described in the Summary of the Invention, Brief Description of the Drawings, Detailed Description of the Invention, Summary, and the claims themselves.

[0071] The foregoing is a simplified summary of the present invention for the purpose of providing an understanding of some aspects of the present invention. This summary is neither an extensive nor an exhaustive overview of the present invention and its various embodiments. It is not intended to identify key or critical elements of the present invention, nor to delineate the scope of the present invention. Instead, its purpose is to present selected concepts of the present invention in a simplified form as an introduction to the more detailed description presented below. It is understood that other embodiments of the present invention are possible by using one or more of the features described above or detailed below, either alone or in combination. Also, while the present disclosure is presented with respect to exemplary embodiments, it is understood that individual aspects of the present disclosure may be separately claimed.

[0072] The present disclosure is described in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0073]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

DETAILED DESCRIPTION OF THE INVENTION

[0074] The following description is merely provided to present embodiments and is not intended to limit the scope, applicability, or configuration of the claims. Rather, the following description provides an enabling description for those skilled in the art to implement the present embodiments. It will be understood that various changes may be made to the functions and configurations of the elements without departing from the spirit and scope of the appended claims.

[0075] Any reference in the description that includes a numerical reference number without an alphabetical sub-reference identifier when there is a sub-reference identifier in the drawings, when used in the plural, is a reference to any two or more elements having the same reference number. Such a reference, when made in the singular without identifying a sub-reference identifier, is not limiting with respect to a particular one of the elements being referred to, but is a reference to one of the numbered similar elements. Any explicit use in this specification that provides an opposite or additional qualification or identification shall take precedence.

[0076] Exemplary systems and methods of the present disclosure are also described with respect to analysis software, modules, and related analysis hardware. However, to avoid unnecessarily obscuring the present disclosure, the following description omits well-known structures, components, and devices, which may be omitted from the drawings or may be illustrated or summarized in the drawings in simplified form.

[0077] For purposes of explanation, numerous details are set forth in order to provide a thorough understanding of the present disclosure. It should be understood, however, that the present disclosure may be practiced beyond the specific details described herein in a variety of ways.

[0078] FIG. 1 shows a system 100 according to an embodiment of the present disclosure. Image capture devices 106A - B may be utilized to capture images of asset 102. Image capture devices 106A - B may be configured to capture images in the visible spectrum and / or the infrared spectrum. Image capture devices 106A - B may be embodied as a single device or as a system of multiple devices such that one device captures visible images and one device captures infrared images. Image capture devices 106A - B may further comprise a display, a communication interface, a user input / output interface, and / or other computing components, communication components, data storage components, and utility components.

[0079] In one embodiment, the image capture device 106A is operated by the conveyance device 104. The conveyance device 104 may be operated remotely, such as by a human-controlled conveyance device 104 via a wireless communication link or other communication link, may be semi-autonomous (e.g., autonomously perform some navigation or movement functions), or may be fully autonomous (e.g., receive a task and perform all actions required to execute the task). In another embodiment, the conveyance device 104 may comprise a device carried by a human operator / user to patrol the environment. The device may further comprise a visual image capture device, a thermal image capture device, and / or other image capture devices. The conveyance device 104 may be configured with components for traveling over various media, such as over the surface of the earth or within a facility, in the air, underwater or on water, or within an industrial environment (e.g., an internal pipeline).

[0080] When fully or partially autonomous, the conveyance device 104 may instruct or otherwise control the image capture device 106A, such as to position the image capture device 106A at a target location or a suitable location for imaging the asset 102.

[0081] In another embodiment, motive power is applied to the image capture device 106B via the human 108. The human 108 may carry or otherwise move the image capture device 106B. Accordingly, the image capture device 106B may be configured to image under ambient conditions or in a marine environment, an aerial environment, or other environments. The image capture device 106B may be inherently autonomous (e.g., capturing an image when arrival at a target location is detected, or when it is detected that it is at a certain distance / angle from the asset to be imaged, etc.). In other embodiments, the image capture device 106B may advise the human 108 to capture an image and / or adjust the placement or orientation of the image capture device 106B as a prerequisite for capturing an image of the asset, which may be performed automatically or via input from the human 108. In yet another embodiment, the conveyance device 104 may be configured to wait or remain in an area for a predetermined amount of time. Specifically, the conveyance device 104 may intelligently remain stationary for a predetermined amount of time to achieve a successful image capture and / or for data processing purposes. Additionally or alternatively, the partially captured image of the asset 102 may direct the conveyance device 104 to move or position the image capture device 106A in a manner that enables capture of the entire asset 102 or a target (e.g., region of interest) of the asset 102. Similarly, the image capture device 106B may provide auditory feedback, visual feedback, or tactile feedback to the human 108 to direct the human 108 to position or move to a location where the entire or target area of the asset 102 can be imaged.

[0082] Generally, the image capture device 106 is optimized to capture images within its field of view and may not have sufficient data processing or storage capacity. Thus, in another environment, the image capture device 106 may be configured to communicate with the server 112, data storage 114, or other computing or storage devices. The communication may include receiving an image from the image capture device 106 and / or providing imaging instructions to the image capture device 106 (either directly or via the carrier device 104 or the human 108). The communication between the image capture devices may be continuous or intermittent (e.g., when physically connected to a data cable, when radio frequency communication is active, etc.).

[0083] In addition to capturing an image of the asset 102, the image capture device 106 may be equipped with sensors, instrumentation, or input components for capturing input from the human 108, ambient conditions (e.g., temperature, humidity, air pressure, etc.), time information (e.g., time / date), location information (e.g., GPS coordinates, x-y coordinates of a facility, distance to the asset, etc.), orientation information (e.g., altitude, angle, etc.), settings (e.g., zoom level, alpha correction, contrast, brightness, color enhancement, etc.). In the case of infrared imaging, the settings may further include pseudo-color conversion settings for converting infrared values to human-understandable colors, such as a blue overlay applied to the image to indicate a lower temperature and a red overlay applied to indicate a higher temperature. The pseudo-color settings may be set for the asset 102, or the environment of the asset 102, or the environment. For example, a steel foundry may consider 1000°F (537°C) to be a low temperature and thus may be displayed as a blue overlay, but a petroleum cracking tower operating at 1000°F (537°C) may become abnormally hot and thus may be displayed as red or white. In another example, the pseudo-color settings may be determined for each image, such as applying blue to the cold parts of the image and red or white to the hottest parts of the image.

[0084] In another embodiment, the observed temperature of asset 102 or portions thereof may be evaluated against alarm conditions, such as a high temperature alarm. The image capture device 106 and / or the server 112 may determine that the observed temperature meets the alarm condition and may automatically issue an alarm of the condition to other systems, such as causing the system to take an action that will mitigate, reduce the alarm condition.

[0085] FIG. 2 shows an asset image 200 according to an embodiment of the present disclosure. In one embodiment, asset 102 is imaged to generate asset image 200. Asset image 200 is further overlaid using bounding boxes, specifically, bounding box 202, bounding box 204, bounding box 206, and bounding box 208. Each of the bounding boxes 202-208 corresponds to an area of interest, which may include a portion of asset 102, such as a motor, bearing, reduction gear, pump, etc. The bounding boxes 202, 204, 206, and 208 are illustrated as rectangles, but it should be understood that other closed geometric shapes of bounding boxes, such as regular geometric shapes (e.g., circular, triangular, elliptical, etc.) or irregular geometric shapes (e.g., freeform shapes), may be utilized.

[0086] In another embodiment, each of the bounding boxes 202-208 may be determined by a human operator. In another embodiment, each of the bounding boxes 202-208 may be determined by an automated process, such as a neural network trained to identify components of asset 102.

[0087] Coloring (illustrated as cross-hatching) may be applied to visually indicate the temperature (highest, central, lowest, etc.) of a portion of the bounding boxes 202-208.

[0088] Figure 3 shows a data flow 300 according to an embodiment of the present disclosure. In one embodiment, some steps of the data flow 300, when read by a machine such as a processor of a server, cause the machine to execute instructions, thereby causing a portion of the data flow 300 to be executed, and are embodied as machine-readable instructions maintained in a non-transitory memory. The processor of the server may include, without limitation, at least one processor of the server 112.

[0089] In one embodiment, the data flow 300 may be segmented into phases such as a positioning / training phase and a testing phase. During the positioning phase, the user uploads one or more images of the asset at step 302 and defines one or more regions of interest and corresponding identifiers in the image at step 304. The regions of interest may be identified for each image, and the portions imaged within the regions of interest may be labeled. If there are two or more regions of interest in each of the images, one portion may be obscured (e.g., blacked out) to avoid errors when the second portion is the subject of training or analysis.

[0090] In the major feature generation step 306, several major features are generated by computer vision. The major features (or major points) are important points determined by the machine that may indicate points indicating specific assets or features of the assets in the image. Step 308 enables the user to define the region of interest.

[0091] As is known in the art, and in one embodiment, a neural network self-configures a layer of logical nodes having inputs and outputs. If the output is below a self-determined threshold level, the output is omitted (i.e., the input is within the non-active response portion of the scale and no output is provided). If the self-determined threshold level exceeds the threshold, an output is provided (i.e., the input is within the active response portion of the scale and an output is provided). As one or more training steps, a particular arrangement of active and non-active wireframes is performed. Multiple inputs into the nodes generate a multi-dimensional plane (e.g., a hyperplane) for wiring combinations of inputs that are active or non-active.

[0092] Model 1 (314) receives images, bounding boxes, identifiers, key features, and regions of interest (steps 304, 306, and 308) for training. Model 1 (314) may include training of a deep learning network, such as via a neural network, for a neural network to later determine an asset identifier (e.g., a tag) when an image of an asset is provided. Model 1 (314) may have access to a database of assets and their descriptions to automatically identify an asset or to remove irrelevant assets from further consideration. For example, an asset having a motor and impeller and connected to piping may be identified as a pump or may be identified as not being at least a drive for a belt conveyor. The image may undergo transformations, including, but not limited to, rotation, contrast change, brightness change (image width and image portion), blurring, sharpening, zooming in, zooming out, partially obscuring, cropping, etc., either manually and / or automatically. During the test phase, Model 1 (314) receives a new image, such as an image captured by image capture device 106, at step 312 and then automatically identifies the asset and location at step 316 based on Model 1. After determining the asset identifier, positioning information is provided to database 322 for use in subsequent models.

[0093] Model 1 (314) automatically identifies an asset from image information, thereby avoiding the need for manual lookup of the asset. If the asset is not identified in detail, the class of the asset may be identified (e.g., one of 20 recirculation pumps may be identified). Additionally or alternatively, asset tags or other indicators attached to or in proximity to the asset may be imaged and analyzed to identify the asset. In another embodiment, thermal images (alone or in addition to visual images) and heat patterns therein are used to identify the asset.

[0094] In another embodiment, in model 2 (318), a positioning phase and a testing phase may be performed. During the positioning phase, an image is taken for each asset, and a domain expert labels the region of interest (e.g., the winding area for a pump) using a shape (e.g., a box, polygon, circle, etc.) that is to be monitored and saved in a database for each asset.

[0095] For each asset to be presented to the neural network for training by model 2 (318), a computer vision algorithm such as scale invariant feature transform (SIFT) automatically generates features (or key points). Model 1 (314) and model 2 (318) may be the same but differently trained neural networks, or separate neural networks. During training within model 2 (318), image transformations are performed including, but not limited to, rotation, contrast change, brightness change (image width and image portion), blurring, sharpening, zooming in, zooming out, partially covering, cropping, etc. Model 2 (318) receives a test image with visible imaging and / or infrared imaging for use in identifying the key features 320.

[0096] Model 1 (314) may automatically identify an asset (e.g., a device) and the location of the asset in step 316, such as coordinates on a grid, GPS coordinates, etc., based on positioning information. This positioning information includes the position (x, y) coordinates of each key point in the template image / reference image and the training image. A one-to-one mapping of the (x, y) coordinates is performed between the template image key points (saved in the database) and the test image key points (identified by model 2 (318)). Based on mathematical characteristics such as a mathematical matrix composed of information for converting a visible light test image to identify the best match to one of several stored images 324 accessed from database 322, poor matches may be filtered. This matrix may also be applied to an infrared red test image to similarly convert the position of temperature values. For example, the image may need to be translated (up, down, left, right), de-rotated, and / or transformed in other ways, such as in step 326, to then align with the template image.

[0097] After the transformation in step 326, each region of interest is defined, and temperature information for each pixel value in the region of interest (e.g., pixel location, line, key point, etc.) is obtained from the test infrared red image, and the temperature value (e.g., minimum, average, maximum, etc.) is the calculated and adjusted temperature determined in step 330. The temperature value comprises a statistical measure related to temperature, which is a mathematical analysis used to summarize the characteristics of the dataset. The statistical measure related to temperature may be, for example, but not limited to, an average temperature value, a median temperature value, a mode temperature value, a percentile temperature value, a range of temperature values, a variance of temperature values, and a standard deviation of temperature values.

[0098] In another embodiment, the asset comprises livestock such as cows. The region of interest may include various body parts such as the head (brain) and / or ears. Since the cows may be in a different orientation or in a different body position, lighting, etc. from the image capture device 106, each photo of the captured cows may look different. Each thermal image is adjusted so that the final region of interest is correctly detected for accurate temperature calculation without human observation or assessment of the collected data.

[0099] Model 2 (318) generates a mathematical matrix that automatically detects the asset key points and corrects the visible light image and the infrared red image to correct for changes in image orientation and perspective that do not affect the temperature calculation. Referring to the livestock example above, Model 2 (318) can correct images taken from different distances to the cows, the orientation of the cows to the image capture device 106, etc.

[0100] In step 330, after Model 1 (314) and Model 2 (318) identify the asset and adjust the image (e.g., the patrol image), such as by adjusting the temperature values calculated from each region of interest, it is executed.

[0101] Model 3 (328) is equipped with a machine learning model such as a neural network and is used to adjust the temperature calculated for each region of interest identified by Model 2 (318). During the training phase, over a specified training period of time, the user collects thermal images of each asset over many days. Metadata for each image, such as the time taken, the user's distance to the device, the current room temperature, and other factors affecting the temperature measurement, are saved. These metadata are treated as independent variables (X) acting on the current temperature (Y) of each region of interest. Using a suitable mathematical formula, when the metadata are provided, a model is constructed to predict the average temperature to be predicted. Next, a reference point is defined using one selected data point out of all the data collected during the training period, which is determined by the domain expert identifying ideal metadata Z that occur regularly and / or are easy to understand.

[0102] For each thermal image during the test, the third model successfully uses the current temperature (Y) and the current metadata to calculate an adjusted temperature (A) similar to the reference point during the training phase. The adjusted temperature for each region of interest is saved to the server at step 330.

[0103] The user may define a set of key performance indicators for the thermal image at step 310. Model 3 (328) uses the adjusted temperature at step 330, and a set of metrics or key performance indicators (KPIs) are automatically calculated at step 332 to detect whether there are any problems or anomalies with the device at step 334. An example of a KPI is the temperature difference between one region of interest and another, and whether this temperature difference exceeds N degrees Celsius. Since all the recorded temperatures have been adjusted or corrected by the three models, the final calculated KPI is fully automated, reliable, and does not require human supervision or intervention.

[0104] In step 334, in the post-model analysis, since the adjusted temperature for each region of interest already takes into account both metadata fluctuations and image capture fluctuations, it can be directly used for KPI calculation by the user and further analysis. Step 334 may trigger an alarm or an automatic action (such as load reduction, plant shutdown, etc.) if the temperature condition exceeds a previously determined threshold.

[0105] Figure 4 shows a process 400 according to an embodiment of the present disclosure. In one embodiment, the process 400 is embodied as machine-readable instructions maintained in a non-transitory memory that, when read by a machine such as a processor of a server, cause the machine to execute instructions, thereby causing the process 400 to be executed. The processor of the server may include, without limitation, at least one processor of server 112.

[0106] In one embodiment, process 400 begins and in step 402, a set of digital asset images is collected from a database or other data repository. In step 404, one or more transformations are applied to each digital asset image, including mirroring, rotating, smoothing, sharpening, blurring, increasing contrast, increasing saturation, increasing brightness, decreasing contrast, decreasing saturation, decreasing brightness, cropping, zooming in, or zooming out, to create a modified set of digital asset images.

[0107] Step 406 creates a first training set comprising a collected set of digital asset images, a modified set of digital asset images, and a set of digital non-asset images. Step 408 trains a neural network in a first training stage using the first training set. Step 410 creates a second training set for a second training stage comprising the first training set and digital non-asset images that are erroneously detected as asset images after the first training stage. Step 412 trains the neural network in the second training stage using the second training set.

[0108] In another embodiment, the images of the set of digital asset images and / or the images of the modified set of digital images may be visible spectrum images or infrared images with temperature indicators such as pseudo-color for converting relative heat levels into visible colors.

[0109] FIG. 5 shows a process 500 according to an embodiment of the present disclosure. In one embodiment, the process 500 is embodied as machine-readable instructions maintained in a non-transitory memory that, when read by a machine such as a processor of a server, cause the machine to execute instructions and thereby execute the process 500. The processor of the server may include, without limitation, at least one processor of the server 112.

[0110] The process 500 begins, and step 502 receives a thermal image of an environment containing assets by using a thermal image capture device disposed on a conveyance device commanded to patrol the environment, the conveyance device moving within the environment autonomously or semi-autonomously.

[0111] Step 504 identifies the assets from an asset database.

[0112] Step 506 determines at least one region of interest of the assets where temperature measurements are to be performed.

[0113] Step 508 performs adjusting the movement of the conveying device for successful processing of the thermal image.

[0114] Step 510 performs calculating at least one temperature value of the asset.

[0115] Figure 6 shows device 602 within system 600 according to an embodiment of the present disclosure. In one embodiment, it includes various components, as well as connections to other components and / or systems. The components may be embodied in various ways and may include a processor 604. As used herein, the term "processor" exclusively refers to an electronic hardware component that includes an electrical circuit configuration having connections (e.g., pinouts) for transmitting encoded electrical signals between the electrical circuit configuration. Processor 604 may have programmable logic functionality, at least in part, as determined by accessing machine-readable instructions maintained in non-transitory data storage, which may be embodied as a circuit configuration, on-chip read-only memory, computer memory 606, data storage 608, etc., that cause the processor 604 to execute steps of the instructions. Processor 604 may be further embodied as a single electronic microprocessor or multiprocessor device (e.g., multi-core) having an electrical circuit configuration therein, and the electrical circuit configuration may further include other components that access information (e.g., data, instructions, etc.) received via a control unit, an input / output unit, an arithmetic logic unit, registers, primary memory, and / or a bus 614, execute instructions, and output data such as via the bus 614 again. In other embodiments, processor 604 may include a shared processing device that can be utilized by other processes and / or process owners, such as within a processing array (e.g., blade, multiprocessor board, etc.) within the system or a distributed processing system (e.g., "cloud," farm, etc.). It should be understood that processor 604 is a non-transitory computing device (e.g., an electro-mechanical device having a circuit configuration and connections for communicating with other components and devices).Processor 604 may operate a virtual processor to process machine instructions that are not native to the processor (e.g., convert a VAX operating system and VAX machine instruction code set to Intel® 9xx chipset code to enable a VAX-specific application to run on a virtual VAX processor). However, as those skilled in the art will understand, such a virtual processor is an application that is executed by hardware, and more particularly, by the electrical circuit configuration and other hardware of the underlying processor (e.g., processor 604). Processor 604 may be executed by a virtual processor, such as when an application (i.e., a Pod) is organized by Kubernetes. The virtual processor may appear to be static and / or a dedicated processor that executes the instructions of the application, but the underlying non-virtual processor is executing the instructions, may be dynamic, and / or may be divided among several processors.

[0116] In addition to the components of the processor 604, the device 602 may utilize computer memory 606 and / or data storage 608 for the storage of accessible data such as instructions, values, etc. The communication interface 610 may facilitate communication with components such as the processor 604 via the bus 614 when the components are not accessible via the bus 614, and may be embodied as a network interface (e.g., Ethernet card, wireless networking components, USB port, etc.). The communication interface 610 may be embodied as a network port, card, cable, or other configured hardware device. Additionally or alternatively, the human input / output interface 612 connects to one or more interface components to receive and / or present information (e.g., instructions, data, values, etc.) between humans and / or electronic devices. Examples of input / output devices 630 that may be connected to the input / output interface include, but are not limited to, keyboards, mice, trackballs, printers, displays, sensors, switches, relays, speakers, microphones, still cameras and / or video cameras, etc. In another embodiment, the communication interface 610 may include the human input / output interface 612, or may be included by the human input / output interface 612. The communication interface 610 may be configured to communicate directly with network-type components, or may be configured to utilize one or more networks such as network 620 and / or network 624.

[0117] Network 620 may be a wired network (e.g., Ethernet), a wireless (e.g., WiFi, Bluetooth, cellular, etc.) network, or a combination thereof, and may enable the device 602 to communicate with network-type component 622.

[0118] Additionally or alternatively, one or more other networks may be utilized. For example, network 624 may represent a second network that facilitates communication with components utilized by device 602. For example, network 624 may be an internal network to a business entity or other organization, whereby components are more trusted (i.e., at least more trusted) than network type component 622 that may be connected to network 620 that comprises a public network (e.g., the Internet) that may not be trusted.

[0119] Components attached to network 624 may include computer memory 626, data storage 628, input / output device 630, and / or other components that may be accessible to computer memory 606, data storage 608, input / output device 612, and / or processor 604. For example, computer memory 626 and / or data storage 628 may augment or replace computer memory 606 and / or data storage 608, either in whole or for a particular task or purpose. As another example, computer memory 626 and / or data storage 628 may be an external data repository (e.g., a server farm, array, "cloud", etc.) that may enable device 602 and / or other devices to access data thereon. Similarly, input / output device 630 may be accessed by processor 604 either directly via network 624, via network 620 alone (not shown), or via networks 624 and 620, via human input / output interface 612, and / or via communication interface 610. Each of computer memory 606, data storage 608, computer memory 626, and data storage 628 comprises non-transitory data storage that comprises a data storage device.

[0120] It should be understood that computer-readable data may be sent, received, stored, processed, and presented by various components. It should also be understood that, whether illustrated herein or not, the illustrated components may control other components. For example, an input / output device 630 may be a router, switch, port, or input / output device 630 that may be associated with network 620 and / or network 624, and may allow (or not allow) communication between two or more nodes on network 620 and / or network 624, such that a particular output of processor 604 enables (or does not enable) it. Without departing from the scope of the present embodiment, those skilled in the art will understand that other communication devices may be utilized in addition to or as an alternative to those described herein.

[0121] FIG. 7 shows a user interface 700 having an asset image 702 according to an embodiment of the present disclosure. In one embodiment, the user interface 700 provides an interface to a computer (e.g., server 112) for performing a positioning phase. During positioning, a user (e.g., user 108) uploads a plurality of images with devices such as pumps. The user defines a plurality of regions of interest using a region shape tool and selects a location type (e.g., "1", "2", "3", "4", "5", and "6") based on a predefined region of interest, such as through the use of a VGG image annotator.

[0122] The user may also define a region that includes the device (e.g., pump_polygon location). For example, if the positioning image includes two pumps, an appropriate pump is defined with reference to the temperature calculation to correctly transform the image. In another embodiment, the background outside the pump_polygon location is removed and blacked out before automatically generating the main features for model 2, ensuring that the test image is aligned with the appropriate pump for accurate data transformation.

[0123] Figure 8 shows a user interface 800 having an asset image 802 according to an embodiment of the present disclosure. In one embodiment, a plurality of pumps can be saved during the positioning process. FIGS. 9-10 show the selection of a CT pump, with the panel pump being selected and no selection of the center pump being shown.

[0124] Figure 9 shows a user interface 900 having an asset image 902 according to an embodiment of the present disclosure. In one embodiment, the asset image 902 comprises an image of a CT pump.

[0125] Figure 10 shows a user interface 1000 having an asset image 1002 according to an embodiment of the present disclosure. In one embodiment, the asset image 1002 comprises an image of a center pump.

[0126] Figure 11 shows a user training image 1100 according to an embodiment of the present disclosure. In one embodiment, a training image for "Model 1" (such as Model 1 (314)) for identifying the equipment / pump training image 1100. The training image 1100 comprises asset images of three different pump identifiers (e.g., "center_pump", "ct_pump", and "panel_pump"). A computer vision deep learning model is trained on images taken at multiple pump identifiers, and the accuracy is evaluated using test images. For each pump, the user takes various images from different angles and vantage points based on where a person normally stands before taking a thermal image of a particular pump. The computer vision algorithm can also be used to randomly crop, rotate, change the brightness, and transform the original photo in multiple ways to increase the accuracy of Model 1.

[0127] The training image 1100 comprises a series of images used to teach a machine learning model to identify the ID "center_pump". When a test image from a camera is received by Model 1 to predict a pump ID based on the historical images on which the machine learning model was trained.

[0128] Figure 12 shows a user training image 1200 according to an embodiment of the present disclosure. In one embodiment, the training image 1200 is provided to train Model 1 to identify "ct_pump".

[0129] In one embodiment, FIGS. 13 to 16 show an example.

[0130] Figure 13 shows an asset image 1300 according to an embodiment of the present disclosure. In one embodiment, the asset image 1300 is shown, such as an image of an asset when the asset image is captured in the visible spectrum. The asset image 1300 may look the same, and even identical, when captured manually and when captured automatically, but may be different due to small-scale image processing.

[0131] Figure 14 shows an asset image 1400 according to an embodiment of the present disclosure. In one embodiment, the asset image 1400 is shown, such as an image of an asset when the asset image is captured in the infrared spectrum. The asset image 1400 may look the same, and even identical, when captured automatically and when captured manually, but may be different due to small-scale image processing.

[0132] Figure 15 shows an asset image 1500 according to an embodiment of the present disclosure. In one embodiment, the asset image 1500 is shown, such as an image of an asset when a visible image and an infrared image (for example, the asset image 1300 and the asset image 1400) are combined / integrated.

[0133] The domain expert may then turn to a composite of a visible light image and an infrared red image, which is created by blending images (e.g., asset image 1300 and asset image 1400) and is used to identify the pump type or ID. Based on an understanding of the pump, a region of interest (shown using rectangles or polygons or others) is selected in the image. In this case, there are five regions of interest called MNDE, winding area, MDE, PDE, and PNDE. The user selects a box and the minimum temperature, average temperature, and maximum temperature are calculated.

[0134] Figure 16 shows an asset image 1600 according to an embodiment of the present disclosure. In one embodiment, an asset image 1600 is shown, such as an image of an asset when a visible image and an infrared image (e.g., asset image 1300 and asset image 1400) are combined.

[0135] In one embodiment, an identifier (the "ID") of the asset (e.g., "center_pump") is identified using Model 1. Configuration data is retrieved from a database based on the pump ID. The configuration saved five regions of interest (e.g., five red rectangles). Using the thermal data under each region of interest, the maximum temperature (in red) and the average temperature (in green) are calculated. Other measurements such as the minimum temperature, the central temperature, or the temperature distribution for each region may also be obtained.

[0136] In one embodiment, FIGS. 17-22 show another example.

[0137] The test image only covers the images (motor areas) shown in FIGS. 17, 19, and 21.

[0138] Figures 18, 20, and 22 each show asset images 1700, 1900, and 2100 that have been transformed according to embodiments of the present disclosure. Although the pump is positioned in the overall pump within the field of view, the user only takes a photo of the left side (motor side) of the pump where the MNDE, winding area, and MDE area are present (the captured asset image 1700 of FIG. 17), as shown in FIG. 18. The invention first uses Model 1 to identify the pump type as "center_pump". Next, Model 2 identifies the important key points / main features. Using the key points / main features of the test image and the template image, the visible light image of the test image is transformed to obtain a mathematical transformation. This transformation is similarly applied to the infrared red image. Finally, the region of interest (in red) is retrieved from the database (see FIG. 22) to calculate the required temperature.

[0139] Figures 17, 19, and 21 each comprise asset images 1700, 1900, and 2100 that have not been transformed according to embodiments of the present disclosure. The domain expert first looks at the image, identifies it as "center_pump", and looks at the composite image where the visible light image and the infrared red image are blended. Next, the expert draws regions of interest (yellow boxes) for each region (see FIG. 21). Note how the boxes here are of different sizes and locations compared to the previous image. This requires the domain expert to understand the image and the equipment. On the other hand, FIG. 22 eliminates the need for the domain expert after image positioning and can use the same region of interest (red box) (see FIG. 22) for the same pump ID, i.e., "center_pump", to calculate the required temperature. This enables the temperature calculation to be fully automated.

[0140] In one embodiment, FIGS. 23-28 show another example. With respect to FIGS. 23-28, the pump was positioned at the overall pump within the field of view, but the user only took a photo of the right side (pump side) of the pump where the winding area, MDE, PDE, and PNDE regions are present, as shown in FIG. 24 (see FIG. 23). In one embodiment, the pump type was identified as "center_pump" using Model 1 and similar steps were taken to successfully transform the image and calculate the required temperature.

[0141] FIG. 23 shows an asset image 2300 according to an embodiment of the present disclosure. In one embodiment, the asset image 2300 shows an asset image captured in the visible spectrum.

[0142] FIG. 24 shows an asset image 2400 according to an embodiment of the present disclosure. In one embodiment, the asset image 2400 shows an asset image captured in the visible spectrum.

[0143] FIG. 25 shows an asset image 2500 according to an embodiment of the present disclosure. In one embodiment, the asset image 2500 shows an asset image captured in the infrared spectrum.

[0144] FIG. 26 shows an asset image 2600 according to an embodiment of the present disclosure. In one embodiment, the asset image 2600 shows an asset image captured in the infrared spectrum.

[0145] FIG. 27 shows an asset image 2700 according to an embodiment of the present disclosure. In one embodiment, the asset image 2700 shows a combined image having regions provided by a domain expert (e.g., combining asset image 2300 and asset image 2500).

[0146] FIG. 28 shows an asset image 2800 according to an embodiment of the present disclosure. In one embodiment, the asset image 2800 shows a combined image having regions provided by Model 1 (e.g., combining asset image 2400 and asset image 2600).

[0147] Figure 29 shows a collage 2900 comprising asset images 2902, 2904, 2906, and 2908 according to an embodiment of the present disclosure. Collage 2900 shows an asset image 2902 indicating an image positioning template, an asset image 2904 showing a test image that is the same as the positioning image, and two test images (asset image 2906 and asset image 2908) that are different from the positioning image. Note how the visible light image and the infrared red image are automatically converted before the temperature in each region of interest is acquired, and that each region of interest (red box) occupies the same coordinates (x, y) as the original template image. Except for the one-time image positioning (asset image 2902) that requires domain experts to label, subsequent test image processing is fully automated and can handle differences in photography and image viewpoints.

[0148] Figure 30 shows a collage 3000 comprising asset images 3002 and 3004 according to an embodiment of the present disclosure. In model 2 (e.g., model 2 (318)), a set of key points or distinct features is generated from the test image. This may involve using a computer vision deep learning model or conventional computer vision techniques. These points are shown in blue within asset image 3002. From the template image, an existing set of key points, shown in green, is extracted. Each test image key point (in blue) is identified using the template image key points (in green). Other potential key points in yellow and red are detected but are filtered out based on some heuristic methods such as low-probability matches.

[0149] For each pairing of a cyan major point and a green major point, a mathematical matrix called a homography matrix is calculated that maps template image coordinates to test image coordinates using the total coordinate (x,y) shift. An inverse matrix is calculated that maps test image coordinates back to template image coordinates. This matrix transformation is applied to the visible light image, resulting in each cyan major point moving very close to, or exactly to, the paired green point. Asset image 3004 shows the transformed image having the original major points (in green) of the template image. Note that the green major points align very closely with the silhouette of the pump, indicating a successful transformation. Looking at the upper image, it is observed that the right side (pump region) is tilted upward, as if the user carelessly tilted and captured the test image, and the final image is adjusted to be flat. The same transformation is applied to the infrared red image for the information obtained from the visible light image, i.e., to shift the image before calculating the temperature in each region of interest using the matrix transformation.

[0150] Figure 31 shows a collage 3100 comprising an asset image 3102 and an asset image 3104 according to an embodiment of the present disclosure. Asset image 3102 was captured by the user taking the image from too far away compared to the template image. The red points are the major points of the test image identified by model 2, and the green points are the major points of the template image. If possible, a pairing is found between each red major point and each green major point. Although not all major points may be successfully paired, the final transformation can still be successfully calculated. As a result, the transformed image (asset image 3104) appears to be zoomed in here, and the template major points in green closely align with the shape and silhouette of the pump. This illustrates that the present invention can handle different types of image viewpoints before calculating the required temperature values for each region of interest.

[0151] In the foregoing description, for purposes of illustration, the methods were described in a particular order. In alternative embodiments, it should be appreciated that the methods may be performed in an order different from that described, without departing from the scope of the present embodiment. It should also be appreciated that the methods described above may be implemented as algorithms executed by custom hardware components (e.g., circuit configurations) customized to perform one or more algorithms or portions thereof described herein. In another embodiment, the hardware components may include a general-purpose microprocessor (e.g., a CPU, GPU) that is first converted into a dedicated microprocessor. A dedicated microprocessor is then provided, into which is loaded an encoded signal that enables the microprocessor, now dedicated, to read and execute a machine-readable set of instructions derived from the algorithm and / or other instructions described herein. The machine-readable instructions utilized to execute the algorithm or portions thereof are not limited and utilize a finite set of instructions known to the microprocessor. The machine-readable instructions may, in one or more embodiments, be encoded in the microprocessor as signals or values within components that generate signals, by voltages within a memory circuit, the configuration of a switching circuit, and / or the selective use of particular logic gate circuits. Additionally or alternatively, the machine-readable instructions may be accessible to the microprocessor and encoded in a medium or device as magnetic fields, voltage values, charge values, reflective / non-reflective portions, and / or physical indicia.

[0152] In another embodiment, the microprocessor may further comprise one or more of a single microprocessor, a multi-core processor, multiple microprocessors, a distributed processing system (e.g., an array, blade, server farm, "cloud", multi-purpose processor array, cluster, etc.), and / or may be placed together with a microprocessor that executes other processing operations. Any one or more microprocessors may be integrated within a single processing device (e.g., a computer, server, blade, etc.), or may be located wholly or partially within individual components and may be connected via a communication link (e.g., a bus, network, backplane, etc., or multiple ones thereof).

[0153] Examples of general-purpose microprocessors may include a central processing unit (CPU) having data values encoded in an instruction register (or other circuit configuration that maintains instructions), or data values having a memory location that serves as an instruction, which may further comprise a memory location external to the CPU. Such components external to the CPU may be embodied as one or more of a field-programmable gate array (FPGA), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), random access memory (RAM), bus-accessible storage, network-accessible storage, etc.

[0154] These machine-executable instructions may be stored on one or more machine-readable media such as a CD-ROM or other type of optical disk, floppy disk, ROM, RAM, EPROM, EEPROM, magnetic or optical card, flash memory, or other type of machine-readable medium suitable for storing electronic instructions. Alternatively, the method may be executed by a combination of hardware and software.

[0155] In another embodiment, the microprocessor may be a system or collection of processing hardware components such as a microprocessor on a client device and a microprocessor on a server, a set of devices each having their respective microprocessors, or a shared or remote processing service (e.g., a "cloud" - based microprocessor). The system of microprocessors may include task - specific allocation of processing tasks and / or shared or distributed processing tasks. In yet another embodiment, the microprocessor may execute software to provide a service for emulating one or more different microprocessors. As a result, a first microprocessor with a first set of hardware components may virtually provide the service of a second microprocessor, such that the hardware associated with the first microprocessor may operate using the instruction set associated with the second microprocessor.

[0156] Machine - executable instructions may be stored and executed locally on a particular machine (e.g., a personal computer, a mobile computing device, a laptop, etc.), but are typically connected via a connection to a remote data storage and / or processing device or set of devices, referred to as the "cloud", which may include public, private, dedicated, shared, and / or other service bureaus, computing services, and / or "server farms", such that storage of data and / or instructions and / or execution of at least a portion of the instructions may occur.

[0157] Examples of microprocessors as described herein include, but are not limited to, Qualcomm® Snapdragon® 800 and 801, Qualcomm® Snapdragon® 610 and 615 with 4G LTE integration and 64-bit computing, Apple® A7 microprocessor with 64-bit architecture, Apple® M7 motion co-processor, Samsung® Exynos® series, Intel® Core™ microprocessor family, Intel® Xeon® microprocessor family, Intel® Atom™ microprocessor family, Intel Itanium® microprocessor family, Intel® Core™ i5-4670K and i7-4770K 22nm Haswell, Intel® Core™ i5-3570K 22nm Ivy Bridge, AMD® FX™ microprocessor family, AMD® FX-4300, FX-6300, and FX-8350 32nm Vishera, AMD® Kaveri microprocessor, Texas Instruments® Jacinto C6000™ automotive infotainment microprocessor, Texas Instruments® OMAP™ automotive grade mobile microprocessor, ARM® Cortex™-M microprocessor, ARM® Cortex-A and ARM926EJ-S™ microprocessors, and may include at least one of other industry equivalent microprocessors, and may perform computer functions using any standard, instruction set, library, and / or architecture known or to be developed in the future.

[0158] Any of the steps, functions, and operations described herein may be continuously and automatically performed.

[0159] Exemplary systems and methods of the present invention are described with respect to communication systems, components, and methods for monitoring, augmenting, and decorating communications and messages. However, to avoid needlessly obscuring the present invention, the foregoing description omits some well-known structures and devices. This omission should not be construed as a limitation on the scope of the claimed invention. Specific details are set forth to provide an understanding of the present invention. However, it should be understood that the present invention may be practiced in various ways beyond the specific details described herein.

[0160] Furthermore, the exemplary embodiments illustrated herein show various components of the systems placed together, but some components of the systems can be located remotely, in separate parts of a distributed network such as a LAN and / or the Internet, or within a dedicated system. Thus, it should be understood that the components of the system or portions thereof (e.g., microprocessors, memory / storage, interfaces, etc.) can be combined in one or more servers, multiple servers, computers, computing devices, terminals, "clouds" or other distributed processing, etc., or placed together on a particular node of a distributed network such as an analog and / or digital telecommunications network, a packet-switched network, or a circuit-switched network. In another embodiment, the components may be physically or logically distributed across multiple components (e.g., a microprocessor may comprise a first microprocessor on one component and a second microprocessor on another component, each executing a part of a shared task and / or an assigned task). It will be understood from the foregoing description, and for reasons of computational efficiency, that the components of the system can be placed at any location within the distributed network of components without affecting the operation of the system. For example, the various components can be located at a switch, such as a PBX and a media server, a gateway, or some combination thereof, forming one or more communication devices at the facility of one or more users. Similarly, one or more functional parts of the system may be distributed between a computing device associated with a telecommunications device.

[0161] Furthermore, it should be understood that the various links connecting the elements can be wired links or wireless links or any combination thereof, or any other known or later-developed element capable of supplying and / or communicating data between the connected elements. These wired or wireless links can also be secure links and may be capable of communicating encrypted information. The transmission medium used as a link can be any suitable carrier for electrical signals, including, for example, coaxial cables, copper wires, and optical fibers, and may take the form of acoustic or light waves, such as those generated during radio waves and infrared data communication.

[0162] Also, although a flowchart is described and illustrated with respect to a particular sequence of events, it should be understood that changes, additions, and omissions to this sequence can be made without materially affecting the operation of the present invention.

[0163] Some variations and modifications of the present invention can be used. It is possible to provide some features of the present invention without providing other features.

[0164] In yet another embodiment, in cooperation with a dedicated computer, a programmed microprocessor or microcontroller and peripheral integrated circuit elements, an ASIC or other integrated circuit, a digital signal microprocessor, a hardwired electronic circuit or logic circuit such as discrete element circuits, a programmable logic device or gate array such as a PLD, PLA, FPGA, PAL, a dedicated computer, any equivalent means, etc., the systems and methods of the present invention can be implemented. Generally, any device or means capable of implementing the methods described herein can be used to implement various aspects of the present invention. Exemplary hardware that can be used for the present invention includes computers, handheld devices, (e.g., cellular, Internet-enabled, digital, analog, hybrid, and others) telephones, and other hardware known in the art. Some of these devices include a microprocessor (e.g., single or multiple microprocessors), memory, non-volatile storage, input devices, and output devices. Further, without limitation, alternative software implementations, including distributed processing or component / object distributed processing, parallel processing, or virtual machine processing, can also be constructed to implement the methods described herein as provided by one or more processing components.

[0165] In yet another embodiment, the disclosed method may be readily implemented in cooperation with software that uses an object or object-oriented software development environment that provides portable source code that can be used on various computer or workstation platforms. Alternatively, the disclosed system may be implemented in part or in whole in hardware using standard logic circuits or VLSI designs. Whether software or hardware is used to implement the system according to the present invention depends on the speed requirements and / or efficiency requirements of the system, specific functions, and the particular software or hardware system or microprocessor or microcomputer system being utilized.

[0166] In yet another embodiment, the disclosed method may be partially implemented in software stored on a storage medium and executed on a programmed general-purpose computer, a dedicated computer, a microprocessor, etc. that cooperate with a controller and a memory. In these cases, the systems and methods of the present invention may be implemented as programs incorporated on a personal computer, such as an applet, JAVA (registered trademark) or CGI script, as resources resident on a server or computer workstation, as a dedicated measurement system, as routines incorporated in system components, etc. The system may also be implemented by physically incorporating the system and / or method into a software and / or hardware system.

[0167] Embodiments herein that include software are executed by one or more microprocessors or stored for subsequent execution and executed as executable code. Executable code is selected to execute instructions for a particular embodiment. The instructions to be executed are a constrained set of instructions selected from an individual set of native instructions that are understood by the microprocessor and delegated to memory accessible by the microprocessor prior to execution. In another embodiment, human-readable "source code" software is first converted to system software prior to execution by one or more microprocessors to comprise a platform-specific set of instructions selected from the native instruction set of the platform (e.g., a computer, a microprocessor, a database, etc.).

[0168] The present invention describes the components and functions implemented in this embodiment with respect to specific standards and protocols, but the present invention is not limited to such standards and protocols. Other similar standards and protocols not described in this specification exist and are considered to be included in the present invention. Moreover, the standards and protocols described in this specification, as well as other similar standards and protocols not described in this specification, are periodically replaced by faster or more effective equivalents having essentially the same function. Such replacement standards and replacement protocols having the same function are considered to be equivalents included in the present invention.

[0169] The present invention, configurations, and aspects in various embodiments include components, methods, processes, systems, and / or devices substantially as illustrated and described herein, including various embodiments, sub-combinations, and subsets thereof. Those skilled in the art will understand how to make and use the present invention after understanding this disclosure. The present invention, configurations, and aspects in various embodiments include providing devices and processes in various embodiments, configurations, or aspects thereof, including when there are no items not illustrated and / or described herein, or when there are no items, for example, that have been used in previous devices or processes to improve performance, achieve ease, and / or reduce the cost of implementation.

[0170] The foregoing description of the invention has been presented for purposes of illustration and description. It is not intended to limit the invention to the one or more forms disclosed herein. In the forms for carrying out the invention, for example, the various features of the invention are grouped together in one or more embodiments, configurations, or aspects for the purpose of streamlining the disclosure. Features, configurations, or aspects of the embodiments may be combined in alternative embodiments, configurations, or aspects other than those described above. This method of disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected by the following claims, aspects of the invention lie in less than all of the features of the single embodiment, configuration, or aspect disclosed above. Accordingly, the following claims are hereby incorporated herein by reference as preferred separate embodiments of the invention, each claim standing on its own basis.

[0171] Moreover, the description of the invention includes the description of one or more embodiments, configurations, or aspects, but, for example, when within the skill and knowledge of those skilled in the art after understanding the present disclosure, some variations and modifications, other variations, combinations, and modifications are within the scope of the invention. Such alternative, interchangeable, and / or equivalent structures, functions, scopes, or steps are intended to be included in the extent of alternative embodiments, configurations, or aspects to the extent permitted, including alternative, interchangeable, and / or equivalent structures, functions, scopes, or steps to the claimed ones, without any intention of publicly disclosing any patentable subject matter.

Description of the Reference Numerals

[0172] 100 System 102 Asset 104 Conveying Equipment 106 Image Capture Device 108 Human 112 Server 114 Data Storage 200 Asset Image 202 Bounding Box 204 Bounding Box 206 Bounding Box 208 Bounding Box 600 System 602 Device 604 Processor 606 Computer Memory 608 Data Storage 610 Communication Interface 612 Human Input / Output Interface 614 Bus 620 Network 622 Network Type Component 624 Network 626 Computer Memory 628 Data Storage 630 Input / Output Device 700 User Interface 702 Asset Image 800 User Interface 802 Asset Image 900 User Interface 902 Asset Image 1000 User Interface 1002 Asset Image 1100 User Training Image, Equipment / Pump Training Image 1200 User Training Image 1300 Asset Image 1400 Asset Image 1500 Asset Image 1600 Asset Image 1700 Asset Image 1900 Asset Image 2100 Asset Image 2300 Asset Image 2400 Asset Image 2500 Asset Image 2600 Asset Image 2700 Asset Image 2800 Asset Image 2900 Collage 2902 Asset Image 2904 Asset Image 2906 Asset Image 2908 Asset Image 3000 Collage 3002 Asset Image 3004 Asset Image 3100 Collage 3102 Asset Image 3104 Asset Image

Claims

1. 1. A device for monitoring the temperature of an asset in an environment, comprising: a vehicle that is instructed to patrol an environment, the vehicle moving autonomously or semi-autonomously within the environment; a thermal image capture device disposed on the transport equipment to assist in autonomously capturing thermal images of the environment; a processor configured to process the thermal image to identify the asset from a database of assets and to determine at least one area of ​​interest on the asset where a temperature measurement is to be performed; Equipped with A device, wherein the transport equipment is configured to move according to results from the processing of the thermal images.

2. The device of claim 1, wherein the transport equipment is configured to move autonomously in one or more manners such that image capture is performed at an angle suitable for successful processing of the thermal image, to adjust the distance between the asset and the thermal image capture device for successful processing of the thermal image, or to adjust the position of the thermal image capture device for successful processing of the thermal image according to machine learning.

3. 2. The device of claim 1, wherein the transport device being configured to move according to the results from the processing of the thermal image comprises moving an internal portion of the transport device configured to hold the thermal image capture device while remaining stationary in a predetermined position.

4. The device of claim 1 , wherein the thermal image comprises a combined image from a visual image capture device and a thermal image capture device.

5. 1. A method for monitoring a temperature of an asset located in an environment, comprising: receiving a thermal image of the environment including the asset by using a thermal image capture device disposed on a transport equipment commanded to patrol the environment, the transport equipment moving autonomously or semi-autonomously within the environment; identifying said asset from a database of assets; determining at least one area of ​​interest of said asset where temperature measurements will be performed; coordinating the movement of the transport device for successful processing of the thermal images; calculating at least one temperature value of said asset; A method for providing the above.

6. providing the thermal image to a machine learning model; receiving an output from the machine learning model in response to the machine learning model relating to the processed results of the thermal image being based at least in part on a detection model; Further equipped with The method of claim 5 , wherein the output comprises one or more areas of interest for the asset.

7. 7. The method of claim 6, wherein the detection model comprises a region of interest (ROI) detection model trained based at least in part on a set of base thermal images associated with the type of asset that are prepared prior to capturing the thermal images.

8. The method of claim 6 , wherein each of the one or more regions of interest is prepared from one or more pre-determined image locations within the thermal image.

9. The method of claim 5 , wherein the temperature comprises at least one of a maximum temperature value, a minimum temperature value, an average temperature value, a median temperature value, or other statistical measure related to temperature.

10. at least one object located within the environment, the thermal image of which is used for identification purposes of the asset or for identifying the location of the asset; 7. The method of claim 6, wherein the processed result is obtained through a step of mapping the at least one object contained in the portion of the thermal image with a reference image based at least in part on the machine learning model.

11. 11. The method of claim 10, wherein the reference image is obtained based at least in part on the machine learning model, the machine learning model is trained based on a set of images associated with the asset, and the set of images are transformed via at least one of cropping, blurring, rotating, or brightness adjustment.

12. The method of claim 10 , wherein the reference image comprises the one or more areas of interest, each of the one or more areas of interest corresponding to a predetermined portion of the asset.

13. 11. The method of claim 10, wherein the mapping of the at least one object comprises at least one of: mapping image coordinates of the thermal image to image coordinates of the reference image based at least in part on the machine learning model; or a re-mapping process in which the image coordinates of the thermal image are updated to a different set of coordinates resulting in a different region of the thermal image being mapped.

14. 6. The method of claim 5, wherein the transport device is configured to adjust at least one of an angle of image capture according to results of the processed thermal images, a distance of image capture according to results of the processed thermal images, or a location of image capture according to results of the processed thermal images.

15. The method of claim 5 , wherein the step of identifying the asset from the database of assets comprises processing a color image, and wherein the step of determining the at least one area of ​​interest of the asset comprises processing the color image.

16. the thermal image comprises a combined image from a visual image capture device and a thermal image capture device; said identifying said asset from said database of assets comprises processing said thermal image; The method of claim 5 , wherein determining the at least one area of ​​interest of the asset comprises processing the thermal image.

17. 6. The method of claim 5, wherein the identifying the asset from the database of assets comprises mapping the thermal image to an image template generated based on a machine learning model.

18. The method of claim 5 , wherein the calculating at least one temperature value of the asset comprises processing an ambient temperature of the environment in which the asset is located.

19. 6. The method of claim 5, wherein the calculating at least one temperature value of the asset comprises determining an operational state of the asset, the operational state comprising a standby state, a transient state, and a steady state.

20. 1. A computer-implemented method for training a neural network for asset detection, comprising: collecting a set of digital asset images from a database; applying one or more transformations, including mirroring, rotating, smoothing, sharpening, blurring, increasing contrast, increasing saturation, increasing brightness, decreasing contrast, decreasing saturation, decreasing brightness, cropping, zooming in, or zooming out, to each digital asset image to create a modified set of digital asset images; creating a first training set comprising the collected set of digital asset images, the modified set of digital asset images, and a set of digital non-asset images; training the neural network in a first stage using the first training set; creating a second training set for a second stage of training comprising the first training set and digital non-asset images that are erroneously detected as asset images after the first stage of training; training the neural network in the second stage using the second training set; 23. A computer-implemented method comprising:

Images