Detecting power grid assets

Abstract

Methods, systems, and apparatus, including computer programs encoded on a storage device, for mapping an electrical grid are disclosed. In one embodiment, a method includes sampling a plurality of locations within a geographic area, performing a detection process for each location, the detection process including applying a set of images of the locations as input to a machine learning (ML) model trained to identify a grid asset depicted within images taken from a combination of different perspectives, and obtaining an output from the machine learning model, the output indicating whether a same power grid asset is identified in each of the images of the locations. In response to a positive identification ML output indicating that the same power grid asset is depicted in a particular set of images at a particular location, the method further includes selecting a plurality of sub-locations within the region and performing a detection process for each sub-location.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. Application No. 18 / 226,716, filed July 26, 2023. The disclosure of the earlier application is considered part of the disclosure of this application and is incorporated herein by reference. Background Technology

[0003] Utility companies often lack up-to-date data on how many electrical grid assets (such as power poles) are in their grid, or where those assets are located, which can make modeling, maintaining, and responding to disturbances in the grid more difficult. Summary of the Invention

[0004] This manual pertains to the inspection of power grid assets in the power grid, such as power poles.

[0005] Generally, this disclosure relates to processes and systems for detecting and locating power poles using images. For example, power poles, also known as electric poles, can be detected based on overhead images, ground-level images, and oblique images. However, different types of imagery (such as images taken from different perspectives) include different information and metadata. For example, overhead images typically include accurate latitude and longitude information but have low resolution for the pole. Ground-level images typically have high resolution for the pole but often lack latitude and longitude information. Oblique images typically have higher resolution than overhead images and can provide multiple views of the same pole, but often lack information about the pole's latitude and longitude. Therefore, image-based power pole detection can be improved by using more than one type of image.

[0006] Typically, the innovative aspects of the subject matter described in this specification can be embodied in a method comprising the following actions: sampling multiple locations within a geographic region by obtaining an image set for each location, each image set including at least one image from a first-viewpoint location, at least one image from a second-viewpoint location, and at least one image from a third-viewpoint location; performing a detection process for each location, the detection process including applying the image set of the location as input to a machine learning (ML) model trained to identify power grid assets depicted within a combination of images taken from different viewpoints, and obtaining an output from the machine learning model indicating whether the same power grid asset is identified in each image of the location; in response to the ML output indicating positive identification of the same power grid asset depicted in a specific image set for a specific location: selecting multiple sub-locations within the region and performing the detection process for each sub-location. Each sub-location may cover different areas of a region near the specific location.

[0007] Other embodiments of this aspect include corresponding systems, apparatuses, and computer programs encoded on computer storage devices and configured to perform actions of methods.

[0008] These and other implementations may each optionally include one or more of the following features.

[0009] In some implementations, power grid assets include any of the following: utility poles, power towers, or components on utility poles or power towers.

[0010] In some implementations, the method further includes determining a plurality of sub-locations along a selected search vector. In some implementations, determining the search vector includes: selecting first sub-locations within a predefined radius of a particular location; performing a detection process for each of the first sub-locations and obtaining a set of ML outputs corresponding to the first sub-locations; and determining the search vector along a direction extending from the particular location toward at least one of the first sub-locations whose corresponding ML outputs indicate positive detection and extending through at least one of the first sub-locations.

[0011] In some implementations, the method further includes performing a detection process along a second sub-position of the search vector to obtain a second set of ML outputs corresponding to the second sub-position.

[0012] In some implementations, the method further includes: selecting a third sub-position within a predefined radius of the other of the second sub-positions, wherein the ML output of the other of the second sub-positions indicates a positive detection result; performing a detection process for each of the third sub-positions and obtaining a third set of ML outputs corresponding to the third sub-position; and determining a search vector along a direction extending from the other of the second sub-positions toward at least one of the third sub-positions whose corresponding ML output indicates a positive detection and extending through at least one of the third sub-positions.

[0013] On the other hand, this can be embodied in a method comprising the following actions: obtaining labeled training data comprising a first set of images, each first set of images comprising at least one image from a first viewpoint, at least one image from a second viewpoint, and at least one image from a third viewpoint, wherein each image in each first set of images comprises a label indicating the presence of power grid assets in the image, and wherein the images in each first set of images are grouped based on representations of public geographic areas; obtaining unlabeled training data comprising a second set of images, each second set of images comprising at least one image from a first viewpoint, at least one image from a second viewpoint, and at least one image from a third viewpoint, and wherein the images in each second set of images are grouped based on representations of public geographic areas; and training a machine learning model to associate power grid assets in a combination of images representing public geographic locations taken from different viewpoints, wherein training comprises applying unlabeled training data and labeled training data as training inputs to the machine learning model.

[0014] Other embodiments of this aspect include corresponding systems, apparatuses, and computer programs configured to perform actions of methods encoded on computer storage devices.

[0015] These and other implementations may each optionally include one or more of the following features. In some implementations, the first viewpoint is a top-down viewpoint, the second viewpoint is a tilted viewpoint, and the third viewpoint is a ground viewpoint.

[0016] In some implementations, at least two of the first or second image sets each include images representing the same geographic area but taken in different seasons or under different weather conditions.

[0017] In some implementations, labels indicate the location of the same power grid asset within each image of a corresponding first image set. Some images in the labeled and unlabeled training data include more than one power grid asset, and the method also includes training a machine learning model to distinguish between multiple power grid assets in each image.

[0018] In some implementations, at least one image from a first viewpoint, a second viewpoint, or a third viewpoint is omitted from at least one image set in the first image set.

[0019] In some implementations, at least one image from a first viewpoint, a second viewpoint, or a third viewpoint is omitted from at least one image set in the second image set.

[0020] In some implementations, the labeled training data and the unlabeled training data each comprise a matrix of corresponding training images, which includes multiple rows of training data for power grid assets, each row comprising different combinations of images from a first, second, or third perspective.

[0021] In some implementations, power grid assets include any of the following: utility poles, power towers, or components on utility poles or power towers.

[0022] Other embodiments of these and other aspects include corresponding systems, apparatuses, and computer programs encoded on a computer storage device and configured to perform actions of a method. A system of one or more computers or other processing devices may be configured by software, firmware, hardware, or combinations thereof installed on the system, which, in operation, causes the system to perform actions. One or more computer programs may be configured by having instructions that, when executed by a data processing device, cause the device to perform actions.

[0023] Specific implementations of the subject matter described in this specification can be carried out to achieve one or more of the following advantages. Compared to methods using only one type of image, the implementations can provide the advantage of detecting and locating power poles with greater accuracy. For example, the system learns patterns based on different viewpoints of the same location and uses these patterns to detect and locate power poles more accurately. This is advantageous because accurate location of the power poles allows for a more accurate model of the power grid. This specification presents a technique for building a machine learning model that is trained to detect and locate power poles. This technique can also be generalized to detect and locate power towers and assets.

[0024] The system employs machine learning models, such as binary classifiers, to predict the presence of a pole at a specified latitude-longitude using images of different types or from different perspectives. The system leverages the overlap of visual cues from different types of images to identify the same object in images of different types.

[0025] Machine learning models can be trained using images of known poles and images without poles. By omitting certain types of data during training, machine learning models can be made more robust. For example, each known pole in the training data can be expanded into multiple rows of training data by including different combinations of image types in each row of training data. Machine learning models can also be made more robust by training with training datasets from different times of day and different seasons.

[0026] The system can also be extended to detect and locate other objects. For example, it can detect and locate telecommunication poles and assets, streetlights, power towers, and assets on utility poles such as transformers and switches. To detect and locate assets, the system can train a multi-class classifier to predict which asset is on the pole at a given latitude-longitude.

[0027] In some instances, the system can also be extended to incorporate other types of data into the machine learning model, such as other types of image data, sensor data, utility data, etc., which can provide more accurate predictions.

[0028] Details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0029] Figure 1 This is a block diagram of an example system used for modeling power grid assets.

[0030] Figures 2A-2C Example top-view images, tilted images, and ground images of utility poles are shown.

[0031] Figures 3A-3B This is a graphical representation of an example image search process used for geolocating power lines.

[0032] Figure 4 This is a flowchart of an example process for modeling power grid assets.

[0033] Figure 5This is a diagram of an example system used to train a machine learning model to identify power grid assets.

[0034] Figure 6 Example top-view images, tilted images, and ground images of utility poles are shown.

[0035] Figure 7 This is a flowchart of an example process used to train a machine learning model to identify power grid assets.

[0036] Figure 8 A schematic diagram of a computer system is depicted that can be applied to any computer implementation of the methods and other techniques described herein.

[0037] In the various figures, the same reference numerals and names indicate the same elements. Detailed Implementation

[0038] Figure 1 This is a block diagram of an example system 100 used for modeling power grid assets. System 100 includes a server system 102. The server system 102 may be hosted within a data center 104, which may be a distributed computing system with multiple computers in one or more locations.

[0039] Server system 102 includes image data storage 120, image selector 122, machine learning model 130, power grid model 190, and sampler 140. The components and modules of server system 102 may be provided as one or more computer-executable software modules or hardware modules. That is, some or all of the functionality of server system 102 may be provided as computer code blocks that, when executed by a processor, cause the processor to perform the functions described below. Some or all of the functionality of server system 102 may be implemented in electronic circuitry, for example, by a standalone computer system (e.g., a server), processor, microcontroller, field-programmable gate array (FPGA), or application-specific integrated circuit (ASIC).

[0040] The power grid model 190 is a computer model of a power grid. The power grid model 190 can be used to provide users with a graphical display of the characteristics of a region. The power grid model 190 may include, for example, the locations of utility poles, power towers, or components on utility poles or power towers. Components may include, for example, medium voltage distribution lines, distribution switches and reclosers, fixed capacitors and switched capacitors, voltage regulation schemes such as tapped magnets or switched capacitors, etc. For example, the power grid model 190 may provide a geographical overlay showing where utility poles are located relative to other power grid components, vegetation, roads, or other geographic features. In some implementations, the power grid model 190 may include information about updates to the power grid model 190. For example, when a power grid asset is added to the power grid model 190, the power grid model 190 may include the date and time when an image was taken to identify the power grid asset.

[0041] Server system 102 acquires image data 112 from camera 110. For example, the image data may come from an overhead perspective, an oblique perspective, or a ground perspective, as shown in the following reference. Figures 2A-2C As described above, power grid asset detection can be improved by using more than one type of image or images from different perspectives. For example, images from a top-down view typically have accurate latitude and longitude information, but have low resolution for power grid assets such as utility poles. Images from a ground-level view typically have high resolution for power grid assets, but often lack latitude and longitude information. Images from an oblique perspective typically have higher resolution than images from a top-down view and can provide multiple views of the same power grid asset, but often lack information about the latitude and longitude of the power grid asset.

[0042] In some examples, camera 110 is mounted on an aerial vehicle. Multiple cameras can be mounted on aerial vehicles to capture images from various perspectives (such as tilted or overhead views) as the vehicle passes over a geographical area. In some examples, camera 110 is mounted on a ground vehicle. Multiple cameras can be mounted on ground vehicles to capture images from different locations on the ground. For example, the camera could be mounted on a utility company vehicle.

[0043] Image data 112 may include images at different resolutions. Image data 112 may include a large number of images, such as hundreds, of images of the same location. Image data 112 may include images captured during multiple flights or flights performed by the same air or ground vehicle or different air or ground vehicles. For example, the location may be defined by coordinate positions. In some examples, images of the same location may include images of a series of coordinate positions surrounding the coordinate position.

[0044] Camera 110 may include a visible light camera, an infrared sensor, a radar (RADAR) sensor, and a lidar (LIDAR) sensor. Image data 112 may include visible light data (e.g., red-green-blue (RGB) data), hyperspectral data, multispectral data, infrared data, radar data, and lidar data collected by the camera.

[0045] Image data 112 includes images representing features of a geographic region; for example, a collection of images in image data 112 represents a geographic region. A geographic region may include, for example, areas spanning hundreds of square meters, several kilometers, hundreds of kilometers, or thousands of kilometers. A geographic region may correspond to the location of an electrical distribution feeder or multiple feeders. In some cases, a geographic region may correspond to the location of, for example, a large power system within or across a state, county, province, or country. Server system 102 may store image data 112 in image data storage 120.

[0046] Image data 112 may include metadata associated with each image in image data 112. Image metadata may include, for example, data indicating the location of the camera that captured the image, the orientation of the camera that captured the image, and / or the date and time the image was captured. The camera location may be stored as, for example, GPS coordinates, such as latitude and longitude coordinates. Metadata may be captured by camera 110, the computing system of the vehicle to which the camera is mounted, or both. For ground imagery, metadata may also include vehicle sense data.

[0047] The orientation of the camera 110 capturing image data 112 can include, for example, camera yaw, pitch, and roll. Yaw can represent rotation about a vertical axis relative to the ground plane. For example, zero yaw can mean the camera is pointing north, while 180 yaw can mean the camera is pointing south. Pitch can represent rotation about a horizontal axis relative to the ground plane. For example, zero pitch can mean the camera is pointing downwards at the ground, while 100 pitch can mean the camera is pointing upwards at the sky. Roll can represent rotation about a horizontal axis perpendicular to the pitch axis. In some examples, the camera orientation includes the camera's elevation relative to the ground.

[0048] In some examples, camera 110 is mounted on a vehicle, and the camera's orientation can be determined based on the vehicle's direction of travel when the image is captured. For example, the camera may have a fixed position and orientation relative to the vehicle, such as a yaw of ninety degrees relative to the vehicle's direction of travel. Therefore, the camera's yaw can be calculated based on the vehicle's direction of travel.

[0049] System 102 uses sampler 140 to select locations for performing a detection process on power grid assets. For example, sampler 140 may receive a top-view image from image data storage 120. The sampler may determine one or more sets of geographic coordinates of potential power grid assets based on the top-view image. The sampler may output the set of geographic coordinates as the selected location 142 to image selector 122.

[0050] System 102 can perform the detection process at each iteration in a series of iterations. In some implementations, the number of iterations can be predefined. In some implementations, the system can perform the detection process until a threshold is reached. For example, the threshold could be a threshold number of detected power grid assets.

[0051] In some implementations, sampler 140 may output a randomly selected location 142. In other implementations, sampler 140 may receive input from a user, for example, specifying the selected location 142. In some implementations, sampler 140 may output the selected location 142 in a predefined pattern, for example, at a fixed distance and orientation relative to each other (e.g., a grid pattern).

[0052] Sampler 140 can also select location 142 based on received identified asset data 132, as described below. For example, sampler 140 can receive identified asset data 132 that indicates affirmative identification of power grid assets at certain locations within an area. Sampler 140 can select multiple sub-locations within the area to output as selected location 142. In some embodiments, sampler 140 can select random sub-locations. Reference will be made below. Figures 3A-3B A more detailed description is the search process for selecting sub-locations.

[0053] Image selector 122 receives selected locations 142 and selects a set of images at those locations from image data storage 120 for power grid modeling. For each selected location, image selector 122 selects a set of two or more images at that location (e.g., selected image 124) to determine whether the same power grid asset is depicted in each image in the image set. Selected image 124 also includes metadata about the selected image. For example, image selector 122 can select multiple images of the same location taken from different perspectives (such as overhead, oblique, or ground view) from image data storage 120. For example, image selector 122 can select images located within a specified geographic range from each other and the selected location; that is, image selector 122 searches image data storage 120 for images with metadata indicating that the image was taken at or near that location. For example, image selector 122 can select images captured from locations within a threshold geographic range of a quarter mile from each other. In some examples, image selector 122 can select images that include the same geographic point in their field of view. For example, image selector 122 can perform bundle adjustment to select images that include the same geographic points.

[0054] Image selector 122 outputs selected image 124 to machine learning model 130. Machine learning model 130 receives selected image 124 as input and is trained to identify power grid assets depicted within selected image 124. For each set of images in selected image 124 corresponding to a selected location, system 102 can run the machine learning model and identify whether the same power grid asset is depicted in the image. (Refer to...) Figures 4-7 A more detailed description of the machine learning model 130.

[0055] Machine learning model 130 outputs identified asset data 132, which can indicate a positive identification of the power grid assets depicted for each image set in the selected images 124. The identified asset data 132 may include the GPS coordinates of each power grid asset. Machine learning model 130 can output the identified asset data 132 to power grid model 190 and sampler 140.

[0056] System 102 can add the identified asset data 132 to an existing model of the power grid, such as power grid model 190. For example, the system can determine whether power grid model 190 includes data representing the power grid assets in the identified asset data 132. If power grid model 190 does not include data representing the power grid assets in the identified asset data 132, system 102 can update power grid model 190 to include the power grid asset. For example, system 102 can update power grid model 190 to include a graphical or geographic representation of the power grid asset. For example, power grid assets may include utility poles or electrical components attached to utility poles (e.g., transformers, reclosers, switches). In some embodiments, when updating power grid model 190 to include power grid assets, system 102 may also include data indicating when the power grid asset was added to power grid model 190 or when the image 124 used to identify the power grid asset was captured.

[0057] Adding or updating the location of one or more grid assets to Grid Model 190 can improve the accuracy of monitoring and / or simulating power grid operations using Grid Model 190.

[0058] Sampler 140 can receive identified asset data 132 and select multiple sub-locations within the area as selected locations 142 for output, as shown in the reference. Figures 3A-3B As stated above.

[0059] In some implementations, system 102 can perform sensor data verification. For example, sampler 140 can receive data from a sensor connected to the power grid. The data may include the sensor's geolocation. To verify whether the sensor is attached to a power grid asset at that geolocation, sampler 140 can output the sensor's location as a selected location 142 to image selector 122. Image selector 122 can select an image at the selected location 142 and provide the selected image 124 to machine learning model 130. Machine learning model 130 can identify whether the same power grid asset is depicted in the selected image 124. If machine learning model 130 makes a positive identification of a power grid asset in the selected image 124, system 102 can verify that the sensor is attached to a power grid asset at the sensor's geolocation. If machine learning model 130 does not make a positive identification of a power grid asset in the selected image 124, system 102 can determine, for example, that there is an error in the sensor data, or that the sensor is not attached to a power grid asset. System 102 can thus verify whether the sensor is attached to a power grid asset.

[0060] In some implementations, system 102 can track changes in the power grid. For example, sampler 140 can obtain the location of existing power grid assets based on power grid model 190. To verify whether a power grid asset is currently located at the location represented in power grid model 190, sampler 140 can output that location as selected location 142 to image selector 122. Image selector 122 can select an image at the selected location 142. For example, image selector 122 can select an image taken after an existing power grid asset was added to power grid model 190. Image selector 122 can provide the selected image 124 to machine learning model 130. Machine learning model 130 can identify whether the same power grid asset is depicted in the selected image 124. If machine learning model 130 makes a positive identification of the power grid asset in the selected image 124, system 102 can verify that the power grid asset exists at the location represented in power grid model 190. If the machine learning model 130 fails to make a positive identification of a power grid asset in the selected image 124, the system 102 can determine that, as of the date and time the selected image 124 was captured, the power grid asset that was once located at that location has been moved or no longer exists. The system 102 can then update the power grid model 190 by removing the representation of that power grid asset at that location.

[0061] For example, a user might want to verify that a utility pole, as represented in the power grid model 190, exists at the street corner. Sampler 140 can receive input from the user including the location of the utility pole. For example, system 102 can verify that the utility pole exists at that location and provide the user with an output indicating that the utility pole exists at the location represented in the power grid model 190. If system 102 determines that the utility pole has been moved or no longer exists, system 102 can update the power grid model 190 and provide the user with an update indicating that the utility pole has been moved or no longer exists.

[0062] In some implementations, system 102 can update sections of the power grid model 190. For example, sampler 140 can select multiple power grid assets located within a certain geographic area from the power grid model 190. System 102 can determine which of the power grid assets have been moved or are no longer present at a specified location in the power grid model 190. For example, system 102 can update the power grid model 190 section by section. In some implementations, system 102 can update sections of the power grid model 190 at set time intervals. In some implementations, system 102 can update sections when image data storage 120 receives new image data 112.

[0063] Figures 2A-2CExample top-view images, oblique images, and ground images of utility poles are shown. Top-view image 200 depicts utility poles 202 and 204 from a top-down perspective, for example, taken from a satellite or aerial vehicle. Oblique image 210 depicts utility poles 202 and 204 from an oblique perspective, for example, taken from an aerial vehicle. Ground image 220 depicts utility poles 202 and 204 from a ground perspective, for example, taken from a ground vehicle or a ground-based camera.

[0064] Machine learning models (such as) Figure 1 A machine learning model 130 can be trained to receive images (such as images 200, 210, and 220) and identify the same power grid assets in images 200, 210, and 220. Power grid assets can be, for example, utility poles or components on utility poles. For example, machine learning model 130 can identify that utility pole 202 is the same pole depicted in images 200, 210, and 220. Machine learning model 130 can also identify that utility pole 204 is the same pole depicted in images 200, 210, and 220. Machine learning model 130 can also identify that component 206 is the same component depicted in images 200, 210, and 220. For example, machine learning model 130 can be trained to identify the same utility poles in images 200, 210, and 220 using visual cues such as characteristics of the utility pole, the area around the utility pole, and other visual features depicted in the images (such as vegetation). Characteristics of the utility pole can include, for example, visible features of the pole or electrical components attached to the pole. (See reference...) Figures 5-7 The training of machine learning model 130 is described in more detail. Similar types of visual cues can be used to identify other power grid assets, such as electrical components.

[0065] Figures 3A-3B This is a graphical representation of an example image search process used for geolocating power lines. Samplers (such as...) Figure 1 The sampler 140 can perform the image search process. Systems (such as...) Figure 1 The system 102) can perform the detection process at different locations and sub-locations determined by the image search process. Figures 3A-3B Each circle in the image represents a power grid asset, such as a utility pole. Compared to a completely random search, the image search process can make power grid modeling more efficient and maximize the efficiency of computational resources.

[0066] For example, samplers can choose initial locations that are relatively far apart from each other. Samplers can also choose to be randomly located within a region of 300, where these locations are at least a certain distance apart, or at a fixed distance apart. For example, as... Figure 3A As shown, the sampler can select location A and location B within region 300. Figure 3AIn the example, the system can perform a detection process and identify one utility pole at location A and zero utility poles at location B.

[0067] The sampler can then select sub-locations within area 300. For each initial location, the sampler can select a sub-location for the system to perform the detection process on. For example, the sampler can receive data indicating a positive identification of a utility pole at the initial location. Figures 3A-3B In the example, the sampler receives data indicating a positive identification of the utility pole at location A. For example... Figure 3B As shown, the sampler can select one or more sub-locations near location A, such as sub-locations C, D, and E.

[0068] In some implementations, the sampler may randomly select a sub-location near location A. In other implementations, the sampler may randomly select a sub-location near location A until it receives data indicating a positive identification of the utility pole. The sampler may then select a sub-location in or near the positively identified direction.

[0069] In some implementations, the sampler can select sub-locations along a search vector. For example, for an initial location where the sampler receives data indicating a positive identification, the system can determine the search vector by selecting a first sub-location within a predefined radius of the initial location, performing a detection process for each of the first sub-locations and obtaining a set of outputs corresponding to the first sub-location, and determining the search vector along a direction extending from the initial location toward and through at least one of the first sub-locations with positive identification. The system can also perform a detection process for a second sub-location along the search vector to obtain data indicating a positive or negative identification of the pole. In response to data indicating a negative identification at one of the second sub-locations, the system can select a third sub-location within a predefined radius of another of the second locations where the data indicates a positive detection result. The system can perform a detection process for each of the third sub-locations and obtain a third set of data indicating a positive or negative identification corresponding to the third sub-location. The system can then determine the search vector along a direction extending from the other of the second locations toward and through at least one of the third sub-locations with positive identification.

[0070] In some implementations, the sampler can select sub-locations that are positioned vertically or horizontally relative to location A, for example, in a grid pattern. Many utility poles in a power grid are arranged in a grid pattern, and searching vertically and horizontally from identified poles can make the search process more efficient. In some examples, the poles may be arranged at an angle. In some implementations, the sampler can select sub-locations that extend the search in multiple directions relative to location A, for example, in a circle. The sampler can then select sub-locations along directions with positive identification.

[0071] exist Figure 3B In the example, the sampler selects sub-location C, which is horizontally located relative to location A, and sub-locations D and E, which are vertically located relative to location A. The sampler can receive data indicating a positive identification of the utility pole at location C. Therefore, the sampler can select sub-locations horizontally located relative to locations A and C for location C. The sampler can select sub-locations further away from locations D and E for locations D and E, similar to the sub-locations selected for location B as described below.

[0072] exist Figure 3A In the example, the sampler did not receive data indicating a positive identification at location B, so the sampler could choose to be located at one or more sub-locations further away from location B. For example... Figure 3B As shown, the sampler can select a sub-position F that is farther from position B than sub-positions C, D, and E are from position A. In some embodiments, the sampler can select sub-position F based on position B with random distance and direction. In other embodiments, the sampler can select sub-position F based on position B with fixed distance and direction.

[0073] The number of sub-locations selected by the sampler for each location can be based on whether a power grid asset is identified at that location. The sampler can select a larger number of sub-locations near locations with identified power grid assets, allowing the system to focus the detection process on areas with more or higher density power grid assets. For example, if the system does not identify a power grid asset at location B, the sampler can either not select a sub-location or select one to perform the detection process there. Since the system identifies a power grid asset at location A, the sampler can select sub-locations C, D, and E to perform the detection process there.

[0074] In implementations where the sampler selects a sub-location geographically close to the location, the distance of the sub-location from the location can be based on whether the power grid asset is identified at that location. For example, the sampler can select a sub-location closer to the identified power grid asset to allow the system to focus the detection process on areas with a higher density of power grid assets. For instance, the sampler can select a sub-location F for location B that is farther from location B than selected sub-locations C, D, and E are from location A.

[0075] In some implementations, the system can select sub-locations based on the proportion of locations where power grid assets are identified. For example, if the system samples five nearby locations and four of those locations have identified power grid assets, the system can select sub-locations that are close to those five locations. If the system samples five locations and two of those locations have identified power grid assets, the system can select sub-locations that are far from those five locations.

[0076] Figure 4 This is a flowchart of an example process 400 for mapping power grid assets. Process 400 can be executed by one or more computing systems (including, but not limited to, system 100 described above) to identify and map power grid assets.

[0077] The system samples multiple locations within a geographic area (410). The system may randomly select one or more locations, or obtain one or more locations through user input. In some implementations, the system may select locations positioned at fixed distances relative to each other. The system may then obtain an image set for each location. Each image set may include at least one image from a first-view location, at least one image from a second-view location, and at least one image from a third-view location. For example, each set may include at least one image from locations viewed from above, from an oblique angle, and from a ground-level perspective.

[0078] The system performs a detection process (420) for each location. The detection process includes applying a set of images of that location as input to a machine learning (ML) model trained to identify power grid assets depicted within a combination of images taken from different perspectives. The detection process also includes obtaining an output from the ML model indicating whether the same power grid asset is identified in each image of the location. For example, a power grid asset could be a utility pole, a power tower, or a component on a utility pole or power tower.

[0079] In response to a positive identification (430) ML output indicating that the same power grid asset is depicted in a specific set of images at a specific location, the system selects multiple sub-locations (440) within the region. Each sub-location may cover a different area of ​​the region near the specific location. See reference... Figures 3A-3BAs described, the number of sub-locations within a region can be based on whether a power grid asset is identified at that location. For example, the system can select a larger number of sub-locations for locations with identified power grid assets. The system can select a smaller number of sub-locations for locations without identified power grid assets. The distance of a sub-location from that location can be based on whether a power grid asset is identified at that location. For example, the system can select sub-locations that are closer to locations with identified power grid assets than locations that are closer to locations without identified power grid assets. The system can select sub-locations that are farther away from locations with identified power grid assets than locations that are farther away from locations with identified power grid assets.

[0080] The system performs a detection process (450) for each sub-location. For example, the system returns to step 410 and applies each sub-location as one or more locations to be sampled. In some implementations, the system may determine a search vector and select multiple sub-locations along that search vector. For example, the system may form a search vector based on locations with positive detection or positive identification. The system may select sub-locations located within a predefined radius of a specific location where a positive detection is made. The system may perform a detection process for each of the sub-locations and obtain a set of ML outputs corresponding to the sub-location. When a second detection is within the radius, the system may form a search vector between the two locations with positive detection and perform further searches in the sub-locations along the search vector. In some implementations, the system performs a detection process for the sub-locations along the search vector to obtain another set of ML outputs corresponding to that sub-location. The system may continue searching for sub-locations along the search vector, i.e., selecting sub-locations along the search vector within a predefined radius of the sub-location whose ML output indicates a positive detection, and wherein the ML output of the newly selected sub-location indicates a positive detection result. The system may perform the sampling and detection process as long as the power grid asset is detected in the sub-location. Therefore, the system can determine a search vector along a direction that extends from the location where a positive detection was made toward at least one of the sub-locations where a positive detection was made, located within a predefined radius of that location, and through at least one of those sub-locations. Once a sub-location along the search vector results in a negative result (e.g., no power grid asset was detected), the system can perform a radius search along the sub-locations surrounding the last or most recent positive detection along the search vector. For example, a negative result could indicate a corner or turn in a set of power lines. The new radius search should result in a new positive direction, and a new search vector indicating a change in direction can be created. The search can then continue along the new search vector.

[0081] In some implementations, ML outputs indicating a positive identification of the same grid asset can be used to create or update the grid model. ML outputs that do not indicate a positive identification of the same grid asset can also be used to update the grid model. For example, if the grid model includes a grid asset at a specific geographic location, but the ML output does not indicate the presence of the same grid asset at that specific geographic location, the grid model can be updated to exclude that grid asset.

[0082] In some implementations, the detection process can apply different types of images or different types of sensor data. For example, sensor data may include time-series data, information about customers of the power grid, or information about components of the power grid. The system can obtain sensor data from sensors connected to the power grid. For example, the system can receive data from sensors located at specific locations or send requests for data to sensors located at specific locations.

[0083] Figure 5 This is a diagram of an example system 500 used to train a machine learning model to identify power grid assets. System 500 includes a machine learning model 130, an evaluator 560, and an adjuster 570. The machine learning model 130, evaluator 560, and adjuster 570 can each be provided as one or more computer-executable software modules or hardware modules. Some or all of the functionality of the machine learning model 130, evaluator 560, and adjuster 570 can be implemented in electronic circuitry, for example, via a stand-alone computer system (e.g., a server), processor, microcontroller, FPGA, or ASIC.

[0084] System 500 can acquire training data. Training data can include images from different viewpoints, such as top-down images, oblique images, and ground images from the same location. Figures 2A-2C The images shown are example images from different perspectives. Some images depict known power grid assets. For example, information about known power grid assets, such as their geographic coordinates, can be obtained from a database maintained by the utility company. Images may include labels indicating the presence of power grid assets within the image. For example, an image may include bounding boxes that mark up the boundaries of each power grid asset present in the image. For example, each bounding box may be associated with a text class indicating the type of power grid asset. Labels may also indicate the location of power grid assets within the image. For example, an image may be labeled with the geographic coordinates of each visible power grid asset in the image. Some images do not have any visible power grid assets.

[0085] In some implementations, system 500 can generate training data by obtaining a list of locations of known power grid assets, obtaining top-view images, oblique images, and ground images of each location, and labeling the visible power grid assets in each image. System 500 can also obtain images of locations without power grid assets.

[0086] In some implementations, the training data may include multiple images of the location from each viewpoint. For example, the training data may include multiple top-view images, oblique images, and ground images of the location. Each of the multiple top-view images, oblique images, and ground images may depict the location under different seasonal, weather, or time conditions. The following... Figure 6 Examples of different types of images (e.g., images from different perspectives) are shown at different times of the year.

[0087] In some implementations, the training system 500 may apply different types of images or different types of sensor data as training data. For example, the training system 500 may obtain sensor data from sensors located on electrical grid components such as utility poles. If the sensor data indicates the geographic location of the sensor, the training system 500 may cascade the sensor data to one or more sets of images corresponding to the same geographic location.

[0088] System 500 can divide the training data into labeled training data 502 and unlabeled training data 504. For example, labeled training data 502 may include labeled images from the training data at certain locations. Unlabeled training data 504 may include unlabeled versions of subsets of images from labeled training data 502 at certain locations.

[0089] In some implementations, system 500 can omit at least one image from a different viewpoint from at least one set of labeled and unlabeled training data. For example, system 500 can omit a ground image from a set of labeled and unlabeled training data for a specific location. For example, a utility pole or power tower may be located in a hillside or backyard where ground images may be difficult to obtain. Therefore, machine learning model 130 can be trained on training inputs that more closely resemble machine learning model 130 after training (e.g., when running a detection process, such as...). Figure 4 (As described in the text) possible inputs. Training the machine learning model 130 on inputs that are more closely similar to the inputs during inference can improve the accuracy and performance of the machine learning model 130.

[0090] In some implementations, labeled training data 502 and unlabeled training data 504 may each comprise a matrix of training images, including multiple rows of training data for each power grid asset. For example, each row in the matrix may include different combinations of images from different viewpoints. For instance, one row may include top-down images, oblique images, and ground images. Another row may include top-down images and oblique images, another row may include oblique images and ground images, and yet another row may include top-down images and ground images. Additional rows may include additional combinations of images from different viewpoints and other types of training data (such as sensor data).

[0091] Machine learning model 130 is trained to associate power grid assets in a combination of images representing public geographic locations taken from different perspectives. For example, machine learning model 130 could be a classifier that uses images of an area around a specified location from different perspectives to predict whether a power grid asset exists at that location. For example, machine learning model 130 could output the geographic coordinates of the power grid asset.

[0092] System 500 trains machine learning model 130 by applying labeled training data 502 and unlabeled training data 504. For example, in an implementation where the labeled training data 502 is labeled with bounding boxes and class labels, machine learning model 130 can learn based on pixels within the bounding boxes. For example, machine learning model 130 can use visual cues in each image to match features across images. Visual cues may include characteristics of the power grid asset visible from different viewpoints, or background characteristics of the power grid asset. For example, utility poles may have features such as color or marking. Utility poles may also include electrical components that can be used to identify the utility poles, such as reclosers and transformers. For example, electrical components may be located at different positions and orientations on different utility poles. Background characteristics may include roads, vegetation, or geographical features. Figure 6 Trees located near utility poles 602 and 604 are shown, which can be used as visual cues to match features across images taken from different perspectives.

[0093] Machine learning model 130 can be trained to identify power grid assets, such as utility poles or power towers, depending on the power grid assets depicted in the training data. For example, to train machine learning model 130 to identify utility poles, the training data may include images of utility poles. To train machine learning model 130 to identify power towers, the training data may include images of power towers.

[0094] In some implementations, the machine learning model 130 can be trained to identify what type of power grid asset is present at the location. For example, the machine learning model 130 can be a multi-class classifier that predicts what type of power grid asset is on a utility pole at a given location. For example, the machine learning model 130 can be trained on training data that includes images of different types of power grid assets from different viewpoints. For example, if the labeled training data 502 is labeled with bounding boxes and class labels, the machine learning model 130 can learn based on the pixels within the bounding boxes and associate each bounding box with a corresponding class label.

[0095] In some implementations, the machine learning model 130 can be trained to predict whether a power grid asset exists at a specified location and whether the specified location is an accurate location. For example, if the machine learning model 130 determines that a power grid asset exists at a specified location, it can also determine that the specified location is an accurate location. If the machine learning model 130 determines that a power grid asset does not exist at a specified location, it can also determine that the specified location is not an accurate location.

[0096] System 500 provides unlabeled training data 504 as input to machine learning model 130. Machine learning model 130 receives a set of images as input. Machine learning model 130 generates training output 552, which identifies whether the same power grid asset is identified in each image of the input image set. For example, training output 552 may include input images labeled with predicted geographic coordinates of the same power grid assets visible in the input images. If machine learning model 130 does not identify the same power grid asset in the image set, training output 552 may also include input images without any labels.

[0097] In an implementation where machine learning model 130 is a multi-class classifier, training output 552 may include the predicted location of the power grid asset among the candidate power grid assets with the highest confidence score. For example, machine learning model 130 may propose multiple candidate bounding boxes for the input image and associate each candidate bounding box with a class label for the type of power grid asset. Machine learning model 130 may select the candidate bounding box with the highest confidence score to include in training output 552.

[0098] System 500 can adjust the parameters of machine learning model 130 based on the output from machine learning model 130. Evaluator 560 can compare training output 552 with labeled training data 502. Evaluator 560 can determine an error 562 between training output 552 and labeled training data 502. For example, error 562 could be an error in the predicted geographic coordinates of the power grid assets in training output 552. Error 562 could also be an identified power grid asset that is not present in the labeled training data 502 in training output 552. Error 562 could also be a missing identified power grid asset present in the labeled training data 502 in training output 552.

[0099] System 500 can use adjuster 570 to adjust the parameters of machine learning model 130 based on error 562. For example, adjuster 570 can provide adjusted model parameters 580 for machine learning model 130. Model parameters may include, for example, configuration variables of the machine learning model, neural network weights, support vectors, and coefficients. By adjusting the model parameters based on error 562, machine learning model 130 can be trained to more accurately detect and identify power grid assets in images.

[0100] Figure 6 Examples of top-down, oblique, and ground-level images of utility poles are shown. Images 600, 610, and 620 depict utility poles 602 and 604 at a certain time of year, for example, when the trees between utility poles 602 and 604 have leaves (e.g., in summer). Images 650, 660, and 670 depict utility poles 602 and 604 at a certain time of year, for example, when the trees have no leaves (e.g., in winter). These two sets of images are fed as training input to a machine learning model 130 (as shown above). Figure 5The methods described herein can make machine learning model 130 more robust to images taken under different conditions. Providing machine learning model 130 with a larger amount of training input can also make machine learning model 130 more accurate. For example, by providing training input with images of the same location at different times of the year, machine learning model 130 can use a wider variety of visual cues (such as background features of the area around the utility pole at different times of the year) and visual features (such as the characteristics of the utility pole at different times of the year). As described above, when machine learning model 130 is used to identify power grid assets, such as during detection, machine learning model 130 can receive a set of images of the same location. The set of images can depict the location at any time of the year. In some instances, the images within the set can depict the location at different times of the year. Because machine learning model 130 has already been trained on training input depicting the same location at different times of the year, machine learning model 130 can more accurately identify and use the visual cues in the set of images to identify any power grid asset depicted in the set of images.

[0101] Figure 7 This is a flowchart of an example process 700 for training a machine learning model to identify power grid assets. Process 700 may be executed by one or more computing systems (including, but not limited to, system 500 described above) to train a machine learning model to identify power grid assets. Power grid assets may include, for example, utility poles, power towers, or components on utility poles or power towers.

[0102] The system obtains labeled training data (710). The labeled training data may include sets of images, and each set may include at least one image from a first viewpoint, at least one image from a second viewpoint, and at least one image from a third viewpoint. For example, the first viewpoint may be a top-down viewpoint, the second viewpoint may be an oblique viewpoint, and the third viewpoint may be a ground viewpoint. Figures 2A-2C The image shows example images from top-down, oblique, and ground-level perspectives. Each image in each image set can include a label indicating the presence of power grid assets in the image. For example, the label could indicate the geographic coordinates of a power grid asset. Each image in each image set can be grouped based on representations of common geographic areas.

[0103] The system obtains unlabeled training data (720). The unlabeled training data may include sets of images, and each set may include at least one image from a first viewpoint, at least one image from a second viewpoint, and at least one image from a third viewpoint. For example, the first viewpoint may be a top-down viewpoint, the second viewpoint may be an oblique viewpoint, and the third viewpoint may be a ground viewpoint. Figures 2A-2CThe image shows example images from top-down, oblique, and ground-level perspectives. Each image in each image set can be grouped based on representations of common geographic areas. Unlabeled training data can include unlabeled versions of image subsets based on labeled training data.

[0104] In some implementations, at least two image sets in the labeled or unlabeled training data each include images representing the same geographical area but taken in different seasons or under different weather conditions. For example, Figure 6 This example shows two sets of images of the same utility pole taken at different times of the year.

[0105] In some implementations, at least one image may be omitted from at least one set of images in the labeled training data set, for example, an image from a first viewpoint, an image from a second viewpoint, or an image from a third viewpoint. That is, in some implementations, at least one image from a top-down viewpoint, an oblique viewpoint, or a ground viewpoint is omitted from at least one set of images in the training data set. In some implementations, at least one image from a first viewpoint, a second viewpoint, or a third viewpoint is omitted from at least one set of images in the unlabeled training data set. Omitting images from different viewpoints can make the machine learning model more robust to missing data. For example, the system may remove one or more specific types of images from the training data set. For example, the system may remove images from a second viewpoint from the training data set. Therefore, in some implementations, the labeled training data and the unlabeled training data may each include matrices of training images. Each matrix of training images may include multiple rows of training data for each power grid asset. Each row may include different combinations of images from a first viewpoint, a second viewpoint, or a third viewpoint.

[0106] In some implementations, each image may be labeled with an indication that distinguishes at least one power grid asset and links that at least one power grid asset among the images in each image set. For example, each image may be labeled with the geographic coordinates of a different power grid asset. In some examples, different power grid assets may have the same geographic coordinates, such as components located on a utility pole and the utility pole itself. Therefore, each image may be labeled with the geographic coordinates and / or a unique identifier of a different power grid asset. As another example, each image may be labeled with a bounding box and / or text class for each power grid asset. The bounding box may be a rectangle marking the boundary of the asset. The text class may indicate the type of power grid asset, such as a utility pole, transformer, crossarm, etc.

[0107] The system trains a machine learning model to associate power grid assets in a combination of images representing public geographic locations taken from different perspectives (730). For example, different perspectives may include overhead views, oblique views, or ground views. Training (as referenced above) Figure 5 The described method may include applying both unlabeled and labeled training data as training inputs to a machine learning model. For example, the system may train a machine learning model to identify the same power grid assets using visual cues depicted in an image.

[0108] In some implementations, machine learning models can be trained to identify that the power grid assets are the same power grid assets, for example, where multiple power grid assets are visible in each image.

[0109] In some implementations, the labeled and unlabeled training data may also include sensor data for the area, and the machine learning model may be trained to identify grid assets using images and sensor data.

[0110] Figure 8 This is a schematic diagram of computer system 800. According to some embodiments, system 800 can be used to perform operations described in association with any previously described computer-implemented method. In some embodiments, the computing systems and devices described herein, as well as their functional operations, can be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware (including structures disclosed herein (e.g., system 800) and their structural equivalents), or in combinations of one or more of these. System 800 is intended to include various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers, including vehicles mounted on base units or pod units of modular vehicles. System 800 may also include mobile devices, such as personal digital assistants, cellular phones, smartphones, and other similar computing devices. Additionally, the system may include portable storage media, such as Universal Serial Bus (USB) flash drives. For example, a USB flash drive can store an operating system and other applications. USB flash drives may include input / output components, such as a USB connector that can be plugged into the USB port of another computing device or a wireless transducer.

[0111] System 800 includes a processor 810, a memory 820, a storage device 830, and an input / output device 840. Each of components 810, 820, 830, and 840 is interconnected using a system bus 850. Processor 810 is capable of processing instructions for execution within system 800. The processor can be designed using any of a variety of architectures. For example, processor 810 can be a CISC (Complex Instruction Set Computer) processor, a RISC (Reduced Instruction Set Computer) processor, or a MISC (Minimum Instruction Set Computer) processor.

[0112] In one embodiment, processor 810 is a single-threaded processor. In another embodiment, processor 810 is a multi-threaded processor. Processor 810 is capable of processing instructions stored in memory 820 or storage device 830 to display graphical information of a user interface on input / output device 840.

[0113] The memory 820 stores information within the system 800. In one embodiment, the memory 820 is a computer-readable medium. In one embodiment, the memory 820 is a volatile memory cell. In another embodiment, the memory 820 is a non-volatile memory cell.

[0114] Storage device 830 provides large-capacity storage for system 800. In one embodiment, storage device 830 is a computer-readable medium. In various other embodiments, storage device 830 may be a floppy disk device, a hard disk device, an optical disk device, or a magnetic tape device.

[0115] Input / output device 840 provides input / output operations for system 800. In one embodiment, input / output device 840 includes a keyboard and / or a pointing device. In another embodiment, input / output device 840 includes a display unit for displaying a graphical user interface.

[0116] The embodiments of the subject matter and functional operations described in this specification can be implemented in digital electronic circuits, in tangibly implemented computer software or firmware, in computer hardware (including the structures disclosed in this specification and their equivalents), or in a combination of one or more of these. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, that is, one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier, for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these.

[0117] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuitry, such as a central processing unit (CPU), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). In some embodiments, the data processing apparatus and / or special-purpose logic circuitry may be hardware-based and / or software-based. The apparatus may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these. This disclosure contemplates the use of data processing apparatuses with or without conventional operating systems (e.g., Linux, UNIX, Windows, Mac OS, Android, iOS, or any other suitable conventional operating system).

[0118] A computer program, also referred to or described as a program, software, software application, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but does not necessarily, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinating files (e.g., a file storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected via a communication network. While portions of a program shown in the various figures are depicted as separate modules implementing various features and functions through various objects, methods, or other processes, a program may alternatively and appropriately include multiple submodules, third-party services, components, libraries, etc. However, the features and functions of various components may be appropriately combined into a single component.

[0119] The processes and logic flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by performing operations on input data and generating output. The processes and logic flows can also be executed by dedicated logic circuitry, and the device can be implemented as dedicated logic circuitry, such as a central processing unit (CPU), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC).

[0120] Computers suitable for executing computer programs include, for example, central processing units (CPUs) that may be based on general-purpose or special-purpose microprocessors or both, or any other type. Generally, the CPU receives instructions and data from read-only memory or random access memory or both. The basic components of a computer are the CPU for executing or running instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to or both. However, a computer does not need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0121] Computer-readable media (temporary or non-temporary, as appropriate) suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Memory can store a variety of objects or data, including caches, classes, frames, applications, backup data, jobs, web pages, web page templates, database tables, repositories storing business and / or dynamic information, and any other suitable information including any parameters, variables, algorithms, instructions, rules, constraints, or references to them. Additionally, memory may include any other suitable data, such as logs, policies, security or access data, report files, and others. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0122] To provide user interaction, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube), LCD (liquid crystal display), or plasma monitor) and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide user interaction; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0123] The term "graphical user interface" or GUI can be used in the singular or plural to describe one or more graphical user interfaces and each display of a particular graphical user interface. Therefore, a GUI can represent any graphical user interface, including but not limited to a web browser, touchscreen, or command-line interface (CLI) that processes information and effectively presents the results to a user. Typically, a GUI may include some or all of several user interface (UI) elements associated with a web browser, such as interactive fields, dropdown lists, and buttons that can be operated by the business suite user. These and other UI elements may be related to or represent the functionality of the web browser.

[0124] The embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer having a graphical user interface or web browser through which a user can interact with embodiments of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs) (e.g., the Internet), and wireless local area networks (WLANs).

[0125] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is established through computer programs running on the respective computers and having a client-server relationship with each other.

[0126] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Some features described in the context of individual embodiments in this specification may also be implemented in combination in a single embodiment. However, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may be for sub-combinations or variations thereof.

[0127] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or to perform all the shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be helpful. Furthermore, the separation of the various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0128] Specific embodiments of the subject matter have been described. Other embodiments, modifications, and substitutions of the described embodiments are within the scope of the appended claims and will be apparent to those skilled in the art. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result.

[0129] Therefore, the above description of the exemplary embodiments does not limit or restrict this disclosure. Other variations, substitutions, and modifications are possible without departing from the spirit and scope of this disclosure.

Claims

1. A method for training a machine learning model to identify power grid assets, the method comprising: Obtain labeled training data comprising a first set of images, each first set of images comprising at least one image from a first viewpoint, at least one image from a second viewpoint and at least one image from a third viewpoint, wherein each image in each first set of images comprises a label indicating the presence of power grid assets in the image, and wherein the images in each first set of images are grouped based on representations of public geographic areas; Obtain unlabeled training data comprising second image sets, each second image set including at least one image from a first viewpoint, at least one image from a second viewpoint, and at least one image from a third viewpoint, wherein the images in each second image set are grouped based on representations of common geographic regions; and Training a machine learning model to associate power grid assets in a combination of images representing public geographic locations taken from different angles involves applying unlabeled and labeled training data as training inputs to the machine learning model.

2. The method according to claim 1, wherein, The first-person perspective is a top-down view, the second-person perspective is a tilted view, and the third-person perspective is a ground view.

3. The method according to any one of claims 1 to 2, wherein, At least two of the first or second image sets each include images representing the same geographic area but taken in different seasons or under different weather conditions.

4. The method according to any one of claims 1 to 3, wherein, The labels indicate the location of the same power grid asset within each image of the corresponding first image set. Some images in the labeled and unlabeled training data include more than one power grid asset, and The method also includes training a machine learning model to distinguish between multiple power grid assets in each image.

5. The method according to any one of claims 1 to 4, wherein, At least one image from a first-view, second-view, or third-view perspective is omitted from at least one image set in the first image set.

6. The method according to any one of claims 1 to 5, wherein, At least one image from a first-view, second-view, or third-view perspective is omitted from at least one image set in the second image set.

7. The method according to any one of claims 1 to 6, wherein, The labeled training data and the unlabeled training data each comprise a matrix of corresponding training images, the matrix of training images comprising multiple rows of training data for power grid assets, each row comprising different combinations of images from a first viewpoint, a second viewpoint, or a third viewpoint.

8. The method according to any one of claims 1 to 7, wherein, Electric grid assets include any of the following: power poles, power towers, or components on power poles or power towers.

9. A method for power grid mapping, comprising: Multiple locations are sampled within a geographic area by obtaining an image set for each location, each image set including at least one image from a location from a first viewpoint, at least one image from a location from a second viewpoint, and at least one image from a location from a third viewpoint; A detection process is performed for each location, which includes applying a set of images of the location as input to a machine learning (ML) model trained to identify power grid assets depicted in a combination of images taken from different perspectives, and obtaining an output from the machine learning model indicating whether the same power grid asset is identified in each of the images of the location. In response to a positive identification of the same power grid asset depicted in a specific set of images at a particular location: Multiple sub-locations are selected within the region, wherein each sub-location covers a different area of ​​the region near the specific location; and The detection process is performed for each sub-location.

10. The method according to claim 9, wherein, Electric grid assets include any of the following: power poles, power towers, or components on power poles or power towers.

11. The method of any one of claims 9 to 10, further comprising determining the plurality of sub-positions along a search vector in which they are selected.

12. The method according to claim 11, wherein, Determining the search vector includes: Select a first sub-position within a predefined radius of the specific position; Perform a detection process for each of the first sub-positions, and obtain a set of ML outputs corresponding to the first sub-position; and A search vector is determined along a direction that extends from the specific location toward at least one of the first sub-locations indicated by its corresponding ML output and extends through at least one of the first sub-locations.

13. The method of claim 12, further comprising: A detection process is performed along the second sub-position of the search vector to obtain a second set of ML outputs corresponding to the second sub-position.

14. The method of claim 13, further comprising an ML output indicating a negative detection result in response to one of the second sub-positions: Choose a third sub-position within a predefined radius of another of the second sub-positions, where, The ML output of the other of the second sub-positions indicates a positive detection result; For each of the third sub-positions, a detection process is performed, and a third set of ML outputs corresponding to the third sub-position is obtained; and A search vector is determined along a direction that extends from another of the second sub-positions toward at least one of the third sub-positions whose corresponding ML output indicates a positive detection and extends through at least one of the third sub-positions.

15. A system comprising: At least one processor; A data storage device is coupled to the at least one processor, the data storage device having instructions stored thereon, the instructions causing the at least one processor to perform operations when executed by the at least one processor, the operations including: Multiple locations are sampled within a geographic area by obtaining an image set for each location, each image set including at least one image from a location from a first viewpoint, at least one image from a location from a second viewpoint, and at least one image from a location from a third viewpoint; A detection process is performed for each location, which includes applying a set of images of the location as input to a machine learning (ML) model trained to identify power grid assets depicted in a combination of images taken from different perspectives, and obtaining an output from the machine learning model indicating whether the same power grid asset is identified in each of the images of the location. In response to a positive identification of the same power grid asset depicted in a specific set of images at a particular location: Select multiple sub-locations within the region, where each sub-location covers a different area of ​​the region near the specific location; and The detection process is performed for each sub-location.

16. The system according to claim 15, wherein, Electric grid assets include any of the following: power poles, power towers, or components on power poles or power towers.

17. The system of any one of claims 15 to 16, further comprising determining the plurality of sub-locations along a search vector in which they are selected.

18. The system according to claim 17, wherein, Determining the search vector includes: Select a first sub-position within a predefined radius of the specific position; Perform a detection process for each of the first sub-positions, and obtain a set of ML outputs corresponding to the first sub-position; and A search vector is determined along a direction that extends from the specific location toward at least one of the first sub-locations indicated by its corresponding ML output and extends through at least one of the first sub-locations.

19. The system of claim 18, further comprising performing a detection process along a second sub-position of the search vector to obtain a second set of ML outputs corresponding to the second sub-position.

20. The system of claim 19, further comprising an ML output indicating a negative detection result in response to one of the second sub-positions: Choose a third sub-position within a predefined radius of another of the second sub-positions, where, The ML output of the other of the second sub-positions indicates a positive detection result; For each of the third sub-positions, a detection process is performed, and a third set of ML outputs corresponding to the third sub-position is obtained; and A search vector is determined along a direction that extends from another of the second sub-positions toward at least one of the third sub-positions whose corresponding ML output indicates a positive detection and extends through at least one of the third sub-positions.

21. A system comprising one or more computers and one or more storage devices storing instructions, the instructions, when executed by the one or more computers, causing the one or more computers to perform operations including the method according to any one of claims 1 to 14.

22. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations including the method according to any one of claims 1 to 14.

Images