Asset hierarchy construction system using pre-labeled annotations, multispectral imagery and video data

Abstract

The systems and methods described herein may include generating an asset hierarchy from a plurality of images of a plurality of assets, and may include performing a machine learning process to identify the plurality of assets from the plurality of images, identifying relationships among the identified plurality of assets based on feature extraction and analysis of the extracted features of the identified plurality of assets, and generating the asset hierarchy from one or more asset hierarchy templates derived from the identified relationships among the identified plurality of assets.

Description

[Technical Field]

[0001] The present disclosure relates to asset management systems, and more particularly to asset hierarchy building systems using image and video data. [Background technology]

[0002] Many industries are in the process of digitalization. However, digitalization is a tedious and error-prone task to create and maintain inventory and update the inventory with existing assets and asset components. Digitalization is a common industrial challenge due to the lack of a recognized and organized asset hierarchy and component specific documentation for operational optimization. There is a need to fill this void by providing appropriate automation methods and processes.

[0003] For many industrial applications in the related art, a single infrastructure component (e.g., a transmission line) is spread across multiple physical locations, each with its own specific context. For example, a transmission line carrying voltage from location A to B may require a different maintenance schedule and may have different physical requirements, such as acceptable levels of tension versus weather conditions. Similarly, a conveyor belt may have different maintenance schedules and requirements based on carried weight and volumetric metrics. Therefore, it is important to understand the hierarchy that manages assets, their components, and the assets within their corresponding operational contexts.

[0004] Centralized asset libraries and asset management can be difficult to maintain in many industries due to challenges in collecting, creating, and inserting new assets and deleting obsolete assets. Additionally, difficulties can be encountered due to a lack of domain expertise and asset related meta-information and relationships between hierarchical components of digital content. Centralized asset libraries and asset management can provide all information about assets in a unified manner, such as the location of electrical systems and electrical panels on a factory floor, or the type of insulation used in an electrical system. The combination of various information can assist sustainability engineers in operational maintenance and product improvement. Summary of the Invention [Means for solving the problem]

[0005] Legacy industries can improve their operational performance by creating and maintaining a digital inventory of numerous assets. The additional effort to transition legacy industries to digital inventory methods can save time and effort in the long run.

[0006] There is a wealth of unstructured data available, including asset hierarchy information. However, ingesting, transforming, and categorizing such unstructured data in an automated and economical manner is difficult. Categorizing assets by context is a challenging task, and resolving context-aware information is critical to complex system operation. Manual categorization is prone to human error, which can propagate all the way down the pipeline.

[0007] In related art, adding new assets to a digital library is a manual process and does not include contextual information. To address such issues, example implementations described herein aim for automatic discovery of asset hierarchies and asset relationships along with context. Furthermore, example implementations described herein may facilitate fusion of information about data from multiple sources and multiple sensors to ensure updated and accurate information.

[0008] Example implementations described herein may include systems and methods for generating an asset hierarchy using images and image-related data. In aspects, the systems and methods described herein may use object detection models to identify assets and generate an asset hierarchy from the detected assets and their corresponding bounding boxes.

[0009] Example implementations described herein may include systems and methods for novel data discovery. In one aspect, the systems and methods described herein may discover desired images from irregular buckets via a learning network. In another aspect, the systems and methods described herein may use images to discover accurate context for accurate image asset hierarchy. In another aspect, the systems and methods described herein may use multiple models in concert to reduce errors in asset hierarchy discovery. In another aspect, the systems and methods described herein may use multiple sources to generate relationships between assets as well as identify the characteristics of assets in a physical system.

[0010] Example implementations described herein may include systems and methods for asset hierarchy template generation. In one aspect, a generic asset hierarchy template is discovered based on irregular images coming from many different data sources. In an aspect, the systems and methods described herein may use annotation boxes and constrained frequent pattern mining algorithms and informed pruning methods to identify not only parent-child relationships but also sibling relationships. Example implementations described herein may include using mathematical models to identify frequent relationships and reduce the discovery of inaccurate relationships. Example implementations described herein may enable the systems and methods to provide streamlined feedback from subject matter experts (SMEs), discover new relationships, and understand relationships between assets to categorize assets (e.g., strong vs. weak relationships) as well as identify parent assets in parent-child relationships.

[0011] Example implementations described herein may include systems and methods that facilitate asset hierarchy discovery. In one aspect, the systems and methods described herein may use data coming from a single data source (e.g., data from only a single company) to identify an accurate asset hierarchy. In another aspect, the systems and methods described herein may use asset hierarchy templates to identify potential missing hierarchy assets and add the potential missing hierarchy assets to the hierarchy. Additionally, the systems and methods described herein may store data for easy asset management in JavaScript Object Notation (JSON) format or another format as needed.

[0012] Aspects of the present disclosure may include a method for generating an asset hierarchy from a plurality of images of a plurality of assets, the method including: performing a machine learning process to identify a plurality of assets from the plurality of images; identifying relationships among the identified plurality of assets based on feature extraction and analysis of the extracted features of the identified plurality of assets; and generating the asset hierarchy from one or more asset hierarchy templates derived from the identifying relationships among the identified plurality of assets.

[0013] Aspects of the present disclosure may include a computer program having instructions for generating an asset hierarchy from a plurality of images of a plurality of assets, the computer program including: performing a machine learning process to identify a plurality of assets from the plurality of images; identifying relationships among the identified plurality of assets based on feature extraction and analysis of the extracted features of the identified plurality of assets; and generating the asset hierarchy from one or more asset hierarchy templates derived from the identifying relationships among the identified plurality of assets. The computer program and instructions may be stored on a non-transitory computer-readable medium and may be executed by one or more processors.

[0014] Aspects of the present disclosure may include a system for generating an asset hierarchy from a plurality of images of a plurality of assets, the system including: means for performing a machine learning process to identify a plurality of assets from the plurality of images; means for identifying relationships among the identified plurality of assets based on feature extraction and analysis of the extracted features of the identified plurality of assets; and means for generating the asset hierarchy from one or more asset hierarchy templates derived from the identifying relationships among the identified plurality of assets.

[0015] Aspects of the present disclosure may include an apparatus configured to generate an asset hierarchy from a plurality of images of a plurality of assets, the apparatus including a processor configured to perform a machine learning process to identify a plurality of assets from the plurality of images, identify relationships among the identified plurality of assets based on feature extraction and analysis of the extracted features of the identified plurality of assets, and generate the asset hierarchy from one or more asset hierarchy templates derived from the identifying relationships among the identified plurality of assets. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 illustrates an example of an asset hierarchy discovery process according to an example implementation.

[0017] [Figure 2] FIG. 2 illustrates an example of data discovery according to an example implementation.

[0018] [Figure 3] FIG. 3 illustrates an example of context discovery according to an example implementation.

[0019] [Figure 4] FIG. 4 shows an example of annotation box location according to an example implementation.

[0020] [Figure 5] FIG. 5 illustrates an example of the processing of an asset hierarchy generation pipeline according to an example implementation.

[0021] [Figure 6]FIG. 6 illustrates an example computing environment using an example computer device suitable for use with some example implementations.

[0022] [Figure 7] FIG. 7 illustrates an example output of an asset hierarchy derived from an asset hierarchy template according to an example implementation. DETAILED DESCRIPTION OF THE INVENTION

[0023] The following detailed description provides details of the drawings and implementation examples of the present application. Redundant element reference numbers and descriptions between drawings are omitted for clarity. Terms used throughout the description are provided by way of example and are not intended to be limiting. For example, the use of the term "automatic" may include a fully automatic implementation or a semi-automatic implementation that includes user or administrator control over certain aspects of the implementation, depending on the desired implementation of one skilled in the art practicing the implementation of the present application. Selections may be made by a user via a user interface or other input means, or may be implemented by a desired algorithm. The implementations described herein may be utilized alone or in combination, and the functionality of the implementations may be implemented by any means depending on the desired implementation.

[0024] Obtaining an asset hierarchy for a legacy system can be important to facilitate data analysis because the asset hierarchy directly affects the assets within the scope. The assets within the scope can be linked to other assets in subsystems via the asset hierarchy. From the subsystem linking, failure modes can be analyzed, and corresponding analyses can be performed. Although the example implementations described herein are provided with respect to the field of electric tower transmission and distribution, they may be applied with respect to other fields where asset hierarchies are applicable (e.g., manufacturing, factory floors, Internet of Things (IoT) systems, etc.) according to desired embodiments, and the present disclosure is not limited thereto.

[0025] In instances where there may be new equipment upgrades or equipment retirements, it can be difficult to cohesively manage an asset library or asset hierarchy. With example implementations described herein, the system itself can provide the most accurate, up-to-date, and current information and eliminate irrelevant information. In related art, categorizing these assets can be very time-consuming and manually intensive to maintain.

[0026] Figure 1 illustrates an example of an asset hierarchy discovery process according to an example implementation. As shown in Figure 1, the asset hierarchy discovery process is divided into three stages.

[0027] During the data preparation process, data can be accumulated and filtered from multiple sources. For example, while mining images for transmission lines, one may find ambiguous, poor-quality images and / or images lacking the desired asset. Such images can be stored locally on a device or server from manually captured images, streamed from a camera or drone, or in another manner depending on the desired implementation. The images may or may not be related to the area (transmission line).

[0028] In an example implementation involving high-voltage lines, images may be sourced from a drone, which may then have images of distribution lines and substations that look similar. However, some of the substations may be older and provide lower voltages, while other substations may have images of high-voltage transmission lines that are not related to the substation. Thus, although the transmission lines appear similar to the distribution lines, they may have entirely different underlying assets.

[0029] To address these issues, models are used to extract features from images, analyze the image in context, and determine the appropriate regions and context of the image through a data discovery process as described below. To perform asset hierarchy, the first step is to clean the data and find data relevant to the desired asset hierarchy system to be detected. First, raw images 101 are taken as input. Next, the data is passed through data discovery 102, which identifies data of interest based on a learning model; this stage may filter the desired data from general data buckets. Further, the image is classified into the image's context by context discovery 103, enabling accurate generation of the asset hierarchy. Annotation boxes are generated for the detected assets at annotation box locations 104. The image and associated metadata, along with the annotation boxes and labels, are then ready for asset hierarchy discovery.

[0030] 2 illustrates an example of data discovery 102 according to an example implementation. Data discovery 102 includes an autoencoder trained on sample images that are part of the asset hierarchy system to be discovered. The trained autoencoder learns a trained image representation, receives an image as input, and reproduces the image based on the trained representation. If the input image belongs to the same type of image for which the network was trained, the output image will be similar to the input image. Conversely, if the image does not belong to the same type of image for which the network was trained, the output image will be substantially different from the input.

[0031] In an example involving power distribution lines versus transmission lines, images of different groups (e.g., from a historical database, from labeled data from a data analyst, etc.) are used to train an autoencoder. The autoencoder can generate representations of images for comparison to determine what kinds of images belong to what kinds of regions. The autoencoder can be trained with images known to be of a particular group (e.g., power lines and various components of a power grid, assets in a factory, etc.) depending on the desired implementation.

[0032] At 201, an image received as input (e.g., raw image 101) is resized to match the input requirements of the trained autoencoder (e.g., the size is based on the images the autoencoder was trained on). Such an image is then input to the autoencoder as X at 202. The encoder then outputs image X' as output image 203.

[0033] At 204, the process calculates a Fréchet Inception Distance (FID) score for X and X' to identify how similar X and X' are, and compares the calculated FID to a predetermined threshold set by the desired implementation. If the FID is greater than the threshold, the image is added to a low-similarity cluster 205. If the FID is less than the threshold, the image is added to a high-similarity cluster 206. The high-similarity cluster 206 contains images determined to be similar to the images used to train the autoencoder. The lower the FID score, the higher the image similarity.

[0034] 3 illustrates an example of context discovery 103 according to an example implementation. Once an image has been processed and object detection / data discovery performed, feature vectors can be formed for all components of the image to perform context discovery. For example, in a particular image of a power system, there may be components such as insulators, electrical towers, etc. For each image, the feature vector indicates the group of components that may be included. Such feature vectors can be used to determine commonalities of assets and can be configured according to a desired implementation.

[0035] First, images are passed through a trained object detection model 301 to discover assets. Feature vectors are then automatically generated from the discovered assets at 302, performed without labels. If available, the SME may provide benchmark feature vectors for the desired classes, as shown at 303. At 304, the process uses cosine similarity to associate the generated feature vectors with specific classes.

[0036] Another way that context can be discovered is by having an SME provide a context for each image and generate a feature vector with class labels as shown at 305, which then automatically generates a feature vector with labels as shown at 306. At 307, a multi-classification machine learning model is trained to discover the context.

[0037] From context discovery and predefined correlation rules, relationships can be built and the strength of the relationships can be determined. Relationships can be based on how visible the images are to each other. Furthermore, missing relationships can be detected by implementation-based suggestions.

[0038] FIG. 4 shows an example of annotation box locations 104 according to an example implementation. After running the object detection model 301, bounding boxes of detected objects can be determined. Such bounding boxes represent the detection of components or defects in the object. Furthermore, the model output can be used to find precise context by passing the model output through a trained multiple classification mode. The example in FIG. 4 shows that the bounding box of the insulator completely contains the bounding boxes for the upper hook and disk. Such bounding boxes can be useful in generating itemsets for asset hierarchy detection.

[0039] In the example of Figure 4, there are bounding boxes annotated on the image, including the bounding boxes of the insulator and the disk. The disk's bounding box is enclosed within another bounding box. Such relationships, along with positioning relationships, can be captured in feature vectors that associate each of these subcomponents together. Other relationships can be associated with each other based on box overlap.

[0040] Regarding the asset hierarchy generation process of FIG. 1, once the data has been identified and cleaned, the next step is to identify all possible assets and generate a comprehensive asset hierarchy template using the identified assets and their corresponding locations. The captured assets may be sorted based on regions based on the detected bounding boxes and models. The asset hierarchy template is generated as are the relationships between each asset (e.g., the relationship between conductors and towers, or between conductors and insulators, etc.). There are different aspects of assets that can be decomposed into subcomponents to determine asset relationships. Relationships between components within each image may be constructed, and the strength of each relationship may be examined.

[0041] Annotation boxes are generated for the discovered assets and then passed to Asset Hierarchy Generation. The Asset Hierarchy Generation (AHC) pipeline is used at the start of asset hierarchy discovery and collection, and the first step is to run a relationship discovery process that uses bounding boxes (along with any configuration information or metadata 109) to identify which assets contain the complete bounding boxes of other assets. Then, by using pattern mining, association rules between parent and child elements are discovered by the relationship discovery process 105.

[0042] Relationship pruning 106 then utilizes pruning rules so that the association rules are further filtered and only those that meet the criteria are retained. The SME may continue to fine-tune the rules as long as required by the desired implementation. Additionally, if such an implementation calls for it, the SME may further validate the relationships at 107. In an example of relationship pruning 106, a child asset cannot have more than one parent asset, and a child cannot have more than one parent. Thus, most pruning is performed based not only on the relationships, but also on the structure of the hierarchy.

[0043] Finally, all discovered relationships 108 are added to an asset hierarchy template 110 along with additional metadata details.

[0044] 5 illustrates an example of the processing of an asset hierarchy generation pipeline according to an example implementation. The rules identified for pruning discovered association relationships from image data may include the following, but may be modified depending on the desired implementation:

[0045] 1) The left hand side (LHS) of an association rule should have only one element.

[0046] 2) One-to-one match between assets; there should be only one LHS element to one right-hand side (RHS) element of the association rule.

[0047] 3) The bounding box of the LHS element is larger than the element on the RHS.

[0048] 4) An asset cannot have two parents. If one of the parents is already a child of the other parent, remove it from the parent list.

[0049] In an example of processing by the asset hierarchy generation pipeline, for an image 501 with annotated bounding boxes, the bounding boxes are used to identify itemsets. Each asset is given an asset identifier (asset ID), as shown in FIG. 5, which may be provided from configuration information and metadata 109. In this example, the asset IDs are given simple letters (A, B, C, D, T) for illustrative purposes, but may be set otherwise depending on the desired implementation. If the bounding box for asset A completely contains the bounding box for asset B, the assets are considered to be in the same itemset. In an example of relationship discovery 105, frequent pattern mining or association rules are used to discover the most frequent and high-confidence relationships shown in 502. From the annotated image 501, the relationships relating asset A to B, asset B to A, asset C to assets A, B, and D individually and as a group, and asset A and B to asset C and D are derived, but not exhaustive, as shown in 502.

[0050] Thereafter, to prune relationships at 106, valid relationships are identified by applying pruning rules as described above. In this example, the relationship for asset T is determined to be that it is the parent of asset A and asset C. Other relationships determined are that asset A is the parent of asset B, and asset C is the parent of asset A and asset D, and other multiple relationships (e.g., due to multiple parents) are removed to obtain the pruned relationship shown at 503. The relationships may then be validated to obtain the example validated relationship at 504, which may be used as the asset hierarchy template 110.

[0051] The process illustrated in FIG. 5 is repeated each time a new image is processed, and the generated asset hierarchy template 110 can then be used to generate an asset hierarchy as described herein.

[0052] Regarding asset hierarchy discovery (AHD) in Figure 1, the final step is to use images obtained from a particular source to detect the source-specific asset hierarchy. However, the information may not be complete, so the asset hierarchy template finds the missing pieces and complements the asset hierarchy information.

[0053] Once the asset hierarchy template 110 is discovered, the live data is passed to the Asset Hierarchy Discovery (AHD), which uses metadata to batch the data into batch data 111 containing related image portions. The AHC Pipeline 112 runs on the batch data, and asset hierarchies and relationships are discovered from the batch data, executing the flows shown at 105 to 109.

[0054] Once the asset hierarchy is generated, it is matched to a template for validation by template comparison 113. Some assets may be missing in the batch data asset identification, so the template generated by AHC is used to identify the missing assets. The basic principle is that if a child exists, the parent should exist as well. Therefore, template comparison 113 is used to fill in the gaps when a parent is not found in the batch data, but will skip adding any children that are not found in the data.

[0055] The information contained in the templates and asset hierarchy is converged into converged information 114, which is then added to the inventory control system 115. The converged information 114 is based on information determined from the templates. For example, there may be multiple templates (e.g., a history template and a transmission line derived template) that may be applied and compared against the converged information 114.

[0056] In example implementations, there may be a power system that includes a large number (e.g., over 10,000) of power towers that are imaged using cameras (e.g., from deployed cameras or from manually captured images from physical locations) and by using drones. Such towers may be mapped by physical location, and then, in accordance with example implementations described herein, systems and methods may capture not only the relationships between assets, but also the number of assets, associated components / subcomponents, their relationships, etc. Such results cannot be performed without time-consuming manual analysis, which cannot be performed manually depending on the number of assets involved. Furthermore, for systems such as transmission lines or power systems, processing such captured images, if performed manually, incurs a high risk of error propagation.

[0057] In an example of an iteration involving a power system as described above, such power system is imaged from images captured from towers or from images captured from drones. Such images may be stored and preprocessed to separate the images captured from towers and the images captured from drones (e.g., based on image metadata). Asset hierarchy templates may then be trained via the process described in FIGS. 1-5 to generate templates based on the images captured from towers, and separate asset hierarchy templates are trained for the images captured from drones. An asset hierarchy may then be constructed from a combination of templates for the images captured from towers and the images captured from drones via convergence information 114.

[0058] Thus, through example implementations described herein, legacy industries can be enabled to transition to digital inventory and hierarchy generation methods by alleviating the tedious task of manual conversion. Example implementations described herein can facilitate an automated and efficient way to ingest, convert, and categorize large amounts of available unstructured image data into hierarchical data (e.g., in JSON or YAML). Example implementations described herein reduce the likelihood of human error and error propagation in manual categorization by automatically categorizing asset context and sharing structural behavior and failure mode information with all relevant systems.

[0059] Additionally, example implementations include automated discovery of assets that allow for accurate updates to the system's asset hierarchy. Additionally, example implementations combine discovery of assets, asset hierarchies, and asset relationships to enable joint calculation of asset health.

[0060] Additionally, example implementations may ensure industrial stability and sustainability by enabling continuous monitoring of assets and the effects of cascading failures from other sub-assets, and may also enable the use of irregular buckets to identify images of interest, facilitating easy categorization of unstructured data.

[0061] Additionally, example implementations allow for the ingestion of unstructured data from multiple sources and presenting it in a single, combined format. Example implementations may provide a basis for tracking individual assets and standardization of asset hierarchy templates.

[0062] 6 illustrates an exemplary computing environment having an exemplary computing device suitable for use in some exemplary implementations. The computing device 605 in the computing environment 600 may include one or more processing units, cores, or processors 610, memory 615 (e.g., RAM, ROM, and / or the like), internal storage 620 (e.g., magnetic, optical, solid-state storage, and / or organic), and / or IO interface 625, any of which may be connected over a communication mechanism or bus 630 to communicate information or may be incorporated within the computing device 605. The IO interface 625 may also be configured to receive images from a camera or provide images to a projector or display, depending on the desired implementation.

[0063] The computing device 605 may be communicatively connected to an input / user interface 635 and an output device / interface 640. One or both of the input / user interface 635 and the output device / interface 640 may be wired or wireless interfaces and may be detachable. The input / user interface 635 may include any physical or virtual device, component, sensor, or interface that can be used to provide input (e.g., buttons, a touchscreen interface, a keyboard, pointing / cursor control, a microphone, a camera, Braille, a motion sensor, an accelerometer, an optical reader, and / or the like). The output device / interface 640 may include a display, television, monitor, printer, speakers, Braille, or the like. In some example implementations, the input / user interface 635 and the output device / interface 640 may be incorporated with or physically connected to the computing device 605. In other example implementations, other computing devices may function as or provide the functionality of input / user interface 635 and output device / interface 640 for computing device 605 .

[0064] Examples of computing devices 605 may include, but are not limited to, highly mobile devices (e.g., smartphones, devices in vehicles or other machines, devices carried by humans and animals, and the like), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, and the like), and devices not designed for mobility (e.g., desktop computers, other computers, information kiosks, televisions, radios, and the like having one or more processors embedded therein and / or connected thereto).

[0065] Computing device 605 may be communicatively connected (e.g., via IO interface 625) to external storage 645 and a network 650 for communicating with any number of networked components, devices, and systems, including one or more computing devices of the same or different configurations. Computing device 605 or any connected computing device may function as, provide functionality for, or be referred to as a server, client, thin server, general-purpose machine, special-purpose machine, or other label.

[0066] IO interface 625 may include, but is not limited to, wired and / or wireless interfaces using any communication or IO protocol or standard (e.g., Ethernet, 802.11x, Universal Serial Bus, WiMax, modem, cellular network protocols, and the like) to communicate information to and / or from at least all connected components, devices, and networks within computing environment 600. Network 650 may be any network or combination of networks (e.g., the Internet, a local area network, a wide area network, a telephone network, a cellular network, a satellite network, and the like).

[0067] The computing device 605 may use and / or communicate using computer-usable or computer-readable media, including transitory media and non-transitory media. Transitory media include transmission media (e.g., metallic cables, optical fibers), signals, carrier waves, and the like. Non-transitory media include magnetic media (e.g., disks and tapes), optical media (e.g., CD-ROMs, digital video disks, Blu-ray disks), solid-state media (e.g., RAM, ROM, flash memory, solid-state storage), and other non-volatile storage or memory.

[0068] The computing device 605 can be used to implement techniques, methods, applications, processes, or computer-executable instructions in some exemplary computing environments. The computer-executable instructions can be obtained from transitory media and stored on and retrieved from non-transitory media. The executable instructions can be from one or more of any programming, scripting, and machine language (e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, and others).

[0069] The processor 610 may run under any operating system (OS) (not shown) in a native or virtual environment. Along with the OS and other applications (not shown), one or more applications may be deployed, including a logic unit 660, an application programming interface (API) unit 665, an input unit 670, an output unit 675, and an inter-unit communication mechanism 695 for different units to communicate with each other. The described units and elements may be modified in design, function, configuration, or implementation and are not limited to the provided description. The processor 610 may be in the form of a hardware processor, such as a central processing unit (CPU), or a combination of hardware and software units.

[0070] In some example implementations, information or instructions to execute, once received by API unit 665, may be communicated to one or more other units (e.g., logic unit 660, input unit 670, output unit 675). In some cases, logic unit 660 may be configured to control information flow between units and to control services provided by API unit 665, input unit 670, and output unit 675 in some example implementations described above. For example, the flow of one or more processes or implementations may be controlled by logic unit 660 alone or in cooperation with API unit 665. Input unit 670 may be configured to obtain inputs for computations described in example implementations, and output unit 675 may be configured to provide outputs based on computations described in example implementations.

[0071] The processor 610 may be configured to execute a method or instructions for generating an asset hierarchy from multiple images of multiple assets (as shown from raw images 101 to data discovery 102), which may include performing a machine learning process (e.g., autoencoder 202 shown in FIG. 3) to identify multiple assets from the multiple images, identifying relationships between the identified multiple assets based on feature extraction of the identified multiple assets and analysis of the extracted features as described with respect to relationship discovery 105, and generating an asset hierarchy (e.g., as shown in FIG. 7) from one or more asset hierarchy templates derived from the identified relationships between the identified multiple assets (e.g., as shown with respect to asset hierarchy template 110 and exemplary validation relationships 504).

[0072] The processor 610 may be configured to execute the above-described methods or instructions, further including: running an autoencoder on the raw image to generate an output image, as shown in FIG. 3, where the autoencoder is configured to be trained on an asset image of a particular region 201 and generate an output image 203 that replicates the raw image in a new region; and filtering the raw image to form a plurality of images based on the similarity between the output image and raw images that meet a threshold, as shown at 204 to 206.

[0073] The processor 610 may be configured to execute the above-described methods or instructions, wherein the machine learning process is configured to generate bounding boxes on the identified assets, as shown at 103 and 104 in FIG. 1 and in the example annotated image 501 in FIGS. 4 and 5.

[0074] The processor 610 may be configured to execute the above-described method or instructions, wherein identifying relationships among the identified assets based on feature extraction of the identified assets and analysis of the extracted features may include generating annotation boxes that identify one of the bounding boxes of the identified assets as encapsulated within one of the bounding boxes of the identified assets, performing a relationship discovery process to identify potential relationships, identifying relationships from the potential relationships through execution of pruning rules, and receiving validation regarding the identified relationships, as shown in FIG. 5. In example embodiments, the relationship discovery process may include processing of predefined association rules or frequent pattern mining, as described herein.

[0075] The processor 610 may be configured to execute the above-described method or instructions, wherein generating an asset hierarchy from one or more asset hierarchy templates derived from identified relationships between the identified assets includes generating a first asset hierarchy template derived from identified relationships between the identified assets associated with a first set of images from the plurality of images captured by the drone, generating a second asset hierarchy template derived from identified relationships between the identified assets associated with a second set of images from the plurality of images captured from the tower, and generating the asset hierarchy from a combination of the first asset hierarchy template and the second asset hierarchy template.

[0076] The processor 610 may be configured to execute the above-described method or instructions, wherein each of the plurality of images is associated with an asset ID, and the method or instructions further include updating the asset hierarchy and asset metadata in response to the additional images, as shown in Figures 5 and 7.

[0077] 7 illustrates an example output of an asset hierarchy derived from an asset hierarchy template, according to an example implementation. According to the example implementations described herein, raw images of a legacy system may be processed to determine an asset hierarchy for the target system, including asset parent-child relationships, asset components (Comp), and asset sub-components (Sub). Each of these assets, components, and sub-components is given an asset identifier (e.g., Asset 11, Asset 23, etc.), which may be updated with each iteration of images received.

[0078] Some portions of the detailed description are presented in terms of algorithms and symbolic representations of operations within a computer. These algorithmic descriptions and symbolic representations are the means used by those skilled in the data processing arts to convey the essence of their innovations to others skilled in the art. An algorithm is a sequence of defined steps leading to a desired end state or result. In one implementation, the steps performed require physical manipulations of tangible quantities to achieve a tangible result.

[0079] Unless otherwise specified, as will be apparent from the description, the description utilizing terms such as "processing," "computing," "calculating," "determining," "displaying," and the like throughout the description will be understood to include the actions and processes of a computer system or other information processing device that manipulates and converts data represented as physical (electronic) quantities in the registers and memory of the computer system into other data similarly represented as physical quantities in the memory or registers of the computer system, or other information storage, transmission, or display device.

[0080] Implementations may also relate to apparatuses for performing the operations described herein. This apparatus may be specially constructed for the required purposes, or it may comprise one or more general-purpose computers selectively activated or reconfigured by one or more computer programs. Such computer programs may be stored in a computer-readable medium, such as a computer-readable storage medium or a computer-readable signal medium. Computer-readable storage media may include tangible media, such as, but not limited to, optical disks, magnetic disks, read-only memory, random-access memory, solid-state devices and drives, or any other type of tangible or non-transitory medium suitable for storing electronic information. Computer-readable signal media may include media such as carrier waves. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. A computer program may include a pure software implementation containing instructions for performing the operations of a desired implementation.

[0081] Various general-purpose systems may be used with the programs and modules according to the examples herein, or it may prove convenient to construct more specialized apparatus to perform the desired method steps. Additionally, the implementations are not described with reference to any particular programming language. It will be understood that a variety of programming languages ​​may be used to implement the techniques of the implementations described herein. Instructions in the programming language may be executed by one or more processing units, such as a central processing unit (CPU), processor, or controller.

[0082] As is known in the art, the operations described above may be performed by hardware, software, or some combination of software and hardware. Various aspects of the implementations may be implemented using circuits and logic (hardware), while other aspects may be implemented using instructions stored on a machine-readable medium (software), which, when executed by a processor, cause the processor to perform methods for carrying out the implementations of the present application. Furthermore, some implementations of the present application may be performed exclusively in hardware, while other implementations may be performed exclusively in software. Furthermore, the various functions described may be performed in a single unit or distributed among several components in any number of ways. When performed by software, the methods may be executed by a processor, such as a general-purpose computer, based on instructions stored on a computer-readable medium. If desired, the instructions may be stored on the medium in compressed and / or encrypted format.

[0083] Additionally, other implementations of the present application will be apparent to those skilled in the art upon consideration of this specification and practice of the present techniques. Various aspects and / or components of the described implementations may be used alone or in any combination. It is intended that the specification and implementations be considered as examples only, with the true scope and spirit of the present application being indicated by the following claims.

Claims

1. 1. A method for generating an asset hierarchy from a plurality of images of a plurality of assets, comprising: performing a machine learning process to identify the plurality of assets from the plurality of images; identifying relationships among the identified assets based on feature extraction and analysis of the extracted features of the identified assets; generating the asset hierarchy from one or more asset hierarchy templates derived from the identified relationships among the identified assets; A method comprising:

2. running an autoencoder on the raw image to generate an output image, the autoencoder being trained on an asset image of a particular domain and configured to generate an output image that is a reproduction of the raw image in a new domain; filtering the raw images to form the plurality of images based on similarities between the output image and the raw images that satisfy a threshold; The method of claim 1 further comprising:

3. The method of claim 1 , wherein the machine learning process is configured to generate a bounding box with the identified assets.

4. Identifying the relationships among the identified assets based on feature extraction and analysis of the extracted features of the identified assets includes: generating an annotation box that identifies one of the identified asset bounding boxes enclosed within another of the identified asset bounding boxes; performing a relationship discovery process to identify potential relationships; identifying said relationships from said potential relationships through execution of pruning rules; receiving verification of the identified relationship; The method of claim 1 , comprising:

5. generating the asset hierarchy from the one or more asset hierarchy templates derived from the identifying relationships among the identified assets, generating a first asset hierarchy template derived from the identified relationships among the identified assets associated with a first set of images from the plurality of images captured by the drone; generating a second asset hierarchy template derived from the identified relationships between the identified assets associated with a second set of images from the plurality of images captured from a tower; generating the asset hierarchy from a combination of the first asset hierarchy template and the second asset hierarchy template; The method of claim 1 , comprising:

6. each of the plurality of images is associated with an asset ID; The method of claim 1 , further comprising updating the asset hierarchy and asset metadata in response to additional images.

7. 1. A non-transitory computer-readable medium storing instructions for generating an asset hierarchy from a plurality of images of a plurality of assets, the instructions comprising: performing a machine learning process to identify the plurality of assets from the plurality of images; identifying relationships among the identified assets based on feature extraction and analysis of the extracted features of the identified assets; generating the asset hierarchy from one or more asset hierarchy templates derived from the identified relationships among the identified assets; 1. A non-transitory computer-readable medium comprising:

8. The instruction: running an autoencoder on the raw image to generate an output image, the autoencoder being trained on an asset image of a particular domain and configured to generate an output image that is a reproduction of the raw image in a new domain; filtering the raw images to form the plurality of images based on similarities between the output image and the raw images that satisfy a threshold; The non-transitory computer-readable medium of claim 7 further comprising:

9. The non-transitory computer-readable medium of claim 7 , wherein the machine learning process is configured to generate a bounding box with the identified assets.

10. identifying the relationships among the identified assets based on feature extraction and analysis of the extracted features of the identified assets, generating an annotation box that identifies one of the identified asset bounding boxes enclosed within another of the identified asset bounding boxes; performing a relationship discovery process to identify potential relationships; identifying said relationships from said potential relationships through execution of pruning rules; receiving verification of the identified relationship; 8. The non-transitory computer-readable medium of claim 7, comprising:

11. generating the asset hierarchy from the one or more asset hierarchy templates derived from the identified relationships among the identified assets, generating a first asset hierarchy template derived from the identified relationships among the identified assets associated with a first set of images from the plurality of images captured by the drone; generating a second asset hierarchy template derived from the identified relationships between the identified assets associated with a second set of images from the plurality of images captured from a tower; generating the asset hierarchy from a combination of the first asset hierarchy template and the second asset hierarchy template; 8. The non-transitory computer-readable medium of claim 7, comprising:

12. each of the plurality of images is associated with an asset identifier; The non-transitory computer-readable medium of claim 7 , wherein the instructions further comprise updating the asset hierarchy and asset metadata in response to additional images.

13. 1. An apparatus configured to generate an asset hierarchy from a plurality of images of a plurality of assets, the apparatus comprising: performing a machine learning process to identify the plurality of assets from the plurality of images; identifying relationships among the identified assets based on feature extraction and analysis of the extracted features of the identified assets; generating the asset hierarchy from one or more asset hierarchy templates derived from the identified relationships among the identified assets; Processors configured to An apparatus comprising:

Images