Autonomous solar power generation installation using artificial intelligence

Abstract

A system and method for installing solar panels is provided. The method acquires an image of a solar panel during ongoing solar power installation and estimates characteristics of the solar panel based on the first image using distance simulation, geometric correction, and / or angle adjustment. The method further generates a control signal for operating a robotic controller to pick the solar panel based on the estimated characteristics. The method further acquires a second image of the solar panel when the solar panel is in a perspective view and detects the orientation of the solar panel based on the image by determining whether the solar panel is coplanar with the fixed solar panel and at a predetermined offset from the fixed solar panel. A control signal for operating a second robotic controller to align the solar panel with the fixed solar panel based on the detected orientation is generated.

Description

[Technical Field]

[0001] Related application data This application is based on and claims priority under U.S. Provisional Patent Application No. 63 / 397,125, filed August 11, 2022, to that provisional patent under 35 U.S.C. § 119, the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates generally to solar panel handling systems, and more particularly to systems and methods for installing solar panels on a mounting structure. [Background technology]

[0003] In the following description, reference is made to certain structures and / or methods. However, the following reference should not be construed as an admission that these structures and / or methods constitute prior art. Applicant expressly reserves the right to demonstrate that such structures and / or methods do not qualify as prior art against the present invention.

[0004] Solar array installation typically involves securing solar panels to a mounting structure. This base support not only provides mounting points for the individual solar panels but also assists in routing the electrical system and, if applicable, any mechanical components. Due to the fragility and large size of solar panels, the process of securing solar panels to a mounting structure presents unique challenges. For example, in many instances, solar panels in a solar array are mounted on a rotatable structure. This structure rotates the solar panels around an axis, allowing the array to track the sun. In such instances, it is difficult to ensure that all solar panels in the array are flush and level with the axis of the rotatable structure. Furthermore, the installation cost of a solar array can account for a significant portion of the overall cost of building a solar array. Therefore, there is a need for a more efficient and reliable solar panel handling system for installing solar panels within a solar array. In ideal environments, traditional computer vision techniques can be utilized. However, glare, overexposure, or underexposure can adversely affect object detection algorithms.

[0005] The use of solar panels is particularly suited to installation in tropical and / or equatorial regions, which, while ideal for solar utilization, present harsh working environments. Thus, a need exists for an autonomous robotic solution for installing solar panels in such environments. However, implementation requires further improvements in robotic installation, such as improvements related to one or more of: guidance and navigation for automated picking of panels from shipping or storage containers, e.g., crates; precise placement onto installation hardware, e.g., torque tubes, and in alignment with previously placed panels; and detection of (and avoidance of) potentially present mechanical structures, jigs, and fixtures, e.g., clamps and fan gears. Summary of the Invention

[0006] Accordingly, the present invention is directed to a solar panel handling system that substantially obviates one or more of the problems resulting from limitations and drawbacks of the related art.

[0007] The solar panel handling system disclosed herein facilitates the installation of solar panels of a solar array onto an existing installation structure, such as a torque tube. By combining tools for handling solar panels with components that allow for coupling of the solar panels to the solar panel support structure, solar panel installation can be made more efficient and reliable. Some embodiments utilize machine learning techniques to overcome environmental inconsistencies. The system can learn from examples with glare and lighting issues and have the ability to generalize to new data during inference.

[0008] Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the invention. The objectives and other advantages of the invention may be realized and attained by the structure particularly pointed out in the detailed description and claims hereof, as well as the appended drawings. [Means for solving the problem]

[0009] To achieve these and other advantages and in accordance with the purposes of the present invention, as embodied and broadly described, a system for installing solar panels may include an arm assembly tool end including a frame and a suction cup coupled to the frame, a linear guide assembly coupled to the arm assembly tool end, the linear guide assembly including a linearly movable clamping tool including an engagement member configured to engage a clamping assembly slidably coupled to the installation structure, a force-torque transducer configured to move the clamping tool along the installation structure, and a junction box coupled to the frame including a controller and power source configured to control the force-torque transducer and the suction cup.

[0010] In another aspect, a method of installing a solar panel may include engaging an arm assembly tool end with a solar panel, the arm assembly tool end comprising a frame and a suction cup coupled to the frame; positioning the solar panel relative to a mounting structure to which a clamping assembly is slidably coupled; engaging a linear guide assembly coupled to the arm assembly tool end with a clamping assembly, the linear guide assembly comprising a linearly movable clamping tool comprising an engagement member configured to engage the clamping assembly and a force-torque converter configured to move the clamping tool along the mounting structure; and actuating the force-torque converter to move the clamping assembly along the mounting structure to engage a side of the solar panel, thereby securing the solar panel relative to the mounting structure.

[0011] In another aspect, a method for installing solar panels may include directing a robot to accurately pick and place solar panels by utilizing machine learning algorithms to automatically detect centers and corners of solar panels (of various sizes). In some embodiments, the method includes isolating the centers and corners of the solar panels and locating the solar panels relative to mounts, such as torque tubes, and previously placed panels. In some embodiments, the method includes detecting auxiliary tools or structures, such as clamps and / or fan gears. In some embodiments, the technology described herein is a simulation-based approach that does not use synthetic images and therefore utilizes a predictor-corrector scheme that is emulatable.

[0012] In another aspect, a method for installing a solar panel may include acquiring a first image of the solar panel during ongoing solar power installation. The method further includes estimating a plurality of characteristics of the solar panel based on the first image using distance simulation, geometric correction, and angle adjustment, and generating a first set of control signals for operating a first robotic controller to pick the solar panel based on the estimated plurality of characteristics. The method further includes acquiring a second image of the solar panel when the solar panel is in a perspective view, and detecting the orientation of the solar panel based on the second image by determining whether the solar panel is flush with and at a predetermined offset from the fixed solar panel. The method further includes generating a second set of control signals for operating a second robotic controller to align the solar panel with the fixed solar panel based on the detected orientation.

[0013] In another aspect, a method for installing a solar panel may include acquiring a first image of the solar panel in a staging area using a viewpoint camera. The method further includes estimating multiple regions / features of the solar panel based on the first image using at least one of distance simulation, geometric correction, and angle adjustment. The method further includes generating a first set of control signals for operating a first robotic controller to pick the solar panel, the first set of control signals being based on one or more of the estimated multiple regions / features. The method further includes acquiring a second image of the solar panel when the solar panel is picked and in a perspective orientation with respect to the viewpoint camera, and detecting an orientation in space of the picked solar panel based on the second image. The method further includes generating a second set of control signals for moving the picked solar panel to an installation position based on the detected orientation. At the installation position, the picked solar panel is aligned with a previously installed solar panel. The floating / picked panel is moved into position to align with a fixed panel on a torque tube. By "moved into position" it is meant that the floating panel is aligned directionally, distance-wise, and orientation-wise with a fixed (reference) panel.

[0014] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed.

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate the invention and, together with the description, further serve to explain the principles of the invention and to enable one skilled in the relevant art to make and use the invention. The illustrative embodiments can be best understood from the following detailed description when read in conjunction with the accompanying drawings. It is emphasized that, according to common practice, the various features of the drawings are not to scale. To the contrary, dimensions of the various features have been arbitrarily expanded or reduced for clarity. The drawings include the following figures: [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a perspective view of a solar panel handling system with a container of solar panels according to one embodiment of the present disclosure. [Figure 2A] FIG. 2 is a plan view of the solar panel handling system and solar panel container of FIG. 1. [Figure 2B] FIG. 2 is a front view of the solar panel handling system and solar panel container of FIG. 1. [Figure 2C] FIG. 2 is a side view of the solar panel handling system and solar panel container of FIG. 1. [Figure 3A] FIG. 1 is a plan view of a solar panel handling system coupled to a single solar panel, according to one embodiment of the present disclosure. [Figure 3B] FIG. 1 illustrates a front view of a solar panel handling system coupled to a single solar panel, according to one embodiment of the present disclosure. [Figure 3C] FIG. 1 is a side view of a solar panel handling system coupled to a single solar panel according to one embodiment of the present disclosure. [Figure 4A] FIG. 1 is a perspective view of a solar panel handling system according to one embodiment of the present disclosure. [Figure 4B] FIG. 1 is a perspective view of a solar panel handling system according to one embodiment of the present disclosure. [Figure 5A]FIG. 1 is a plan view of a solar panel handling system according to one embodiment of the present disclosure. [Figure 5B] FIG. 1 is a front view of a solar panel handling system according to one embodiment of the present disclosure. [Figure 5C] FIG. 1 illustrates a side view of a clamping tool of a solar panel handling system in a stowed position according to one embodiment of the present disclosure. [Figure 5D] FIG. 10 is a side view of a clamping tool in an extended or advanced position according to one embodiment of the present disclosure. [Figure 6A] FIG. 1 is a perspective view of a clamping tool of a solar panel handling system in engagement with a clamping assembly coupled to a mounting structure according to one embodiment of the present disclosure. [Figure 6B] FIG. 1 is a perspective view of a clamping tool of a solar panel handling system in engagement with a clamping assembly coupled to a mounting structure according to one embodiment of the present disclosure. [Figure 7A] FIG. 10 is a plan view of a clamping tool of a solar panel handling system in engagement with a clamping assembly coupled to a mounting structure according to one embodiment of the present disclosure. [Figure 7B] FIG. 1 is a front view of a clamping tool of a solar panel handling system engaged with a clamping assembly coupled to a mounting structure according to one embodiment of the present disclosure. [Figure 7C] FIG. 1 is a side view of a clamping tool of a solar panel handling system in engagement with a clamping assembly coupled to a mounting structure according to one embodiment of the present disclosure. [Figure 7D] FIG. 10 is a rear view of a clamping tool of a solar panel handling system engaged with a clamping assembly coupled to a mounting structure according to one embodiment of the present disclosure. [Figure 8] FIG. 1 is a schematic diagram of an overhead view of a solar panel handling system during the process of installing a solar panel, according to one embodiment of the present disclosure. [Figure 9]FIG. 1 illustrates a solar panel handling system including an assembly tool coupled to an assembly mobile robot using a robotic arm. [Figure 10] FIG. 1 illustrates a solar panel handling system with two robotic arms, with two assembly tools coupled to an assembly mobile robot using their respective robotic arms. [Figure 11A] FIG. 1 illustrates a process for installing solar panels. [Figure 11B] FIG. 1 illustrates a process for installing solar panels. [Figure 11C] FIG. 1 illustrates a process for installing solar panels. [Figure 12A] FIG. 1 is a diagram showing the configuration of a mobile robot system including two modular vehicles and a land vehicle with two robotic arms. [Figure 12B] FIG. 1 is a diagram showing the configuration of a mobile robot system including two modular vehicles and a land vehicle with two robotic arms. [Figure 13] FIG. 1 is a schematic diagram of placement achieved using computer vision registration. [Figure 14] FIG. 1 is a schematic diagram of an arrangement in which a modular vehicle is replaced with a new modular vehicle having additional supplemental solar panels. [Figure 15] FIG. 1 is a detailed diagram of an example configuration of a system for installing solar panels according to an embodiment of the present disclosure. [Figure 16] FIG. 1 is a detailed diagram of an example configuration of a system for installing solar panels according to an embodiment of the present disclosure. [Figure 17] FIG. 1 is a detailed diagram of an example configuration of a system for installing solar panels according to an embodiment of the present disclosure. [Figure 18] FIG. 1 is a detailed diagram of an example configuration of a system for installing solar panels according to an embodiment of the present disclosure. [Figure 19] FIG. 1 is a detailed diagram of an example configuration of a system for installing solar panels according to an embodiment of the present disclosure. [Figure 20] FIG. 1 is a detailed diagram of an example configuration of a system for installing solar panels according to an embodiment of the present disclosure. [Figure 21A] FIG. 1 is a detailed diagram of an example configuration of a system for installing solar panels according to an embodiment of the present disclosure. [Figure 21B] FIG. 1 is a detailed diagram of an example configuration of a system for installing solar panels according to an embodiment of the present disclosure. [Figure 22] FIG. 1 is a detailed diagram of an example configuration of a system for installing solar panels according to an embodiment of the present disclosure. [Figure 23] FIG. 1 is a detailed diagram of an example configuration of a system for installing solar panels according to an embodiment of the present disclosure. [Figure 24] FIG. 1 is a detailed diagram of an example configuration of a system for installing solar panels according to an embodiment of the present disclosure. [Figure 25] FIG. 1 is a detailed diagram of an example configuration of a system for installing solar panels according to an embodiment of the present disclosure. [Figure 26] FIG. 1 is a detailed diagram of an example configuration of a system for installing solar panels according to an embodiment of the present disclosure. [Figure 27] FIG. 1 is a detailed diagram of an example configuration of a system for installing solar panels according to an embodiment of the present disclosure. [Figure 28] FIG. 1 is a detailed diagram of an example configuration of a system for installing solar panels according to an embodiment of the present disclosure. [Figure 29] FIG. 1 is a detailed diagram of an example configuration of a system for installing solar panels according to an embodiment of the present disclosure. [Figure 30] FIG. 1 is a detailed diagram of an example configuration of a system for installing solar panels according to an embodiment of the present disclosure. [Figure 31] FIG. 1 is a detailed diagram of an example configuration of a system for installing solar panels according to an embodiment of the present disclosure. [Figure 32] FIG. 1 is a detailed diagram of an example configuration of a system for installing solar panels according to an embodiment of the present disclosure. [Figure 33]FIG. 1 is a detailed diagram of an example configuration of a system for installing solar panels according to an embodiment of the present disclosure. [Figure 34] FIG. 1 is a detailed diagram of an example configuration of a system for installing solar panels according to an embodiment of the present disclosure. [Figure 35A] FIG. 1 is a block diagram of an example image processing pipeline in accordance with some embodiments. [Figure 35B] FIG. 1 illustrates an example captured corrected image according to some embodiments. [Figure 35C] FIG. 35C illustrates the output of an example of neural network image segmentation of the acquired corrected image shown in FIG. 35B, according to some embodiments. [Figure 35D] FIG. 1 illustrates an example panel corner detection according to some embodiments. [Figure 36] 1A-1C illustrate examples of images of roads under different lighting conditions and segmentation masks for those images, according to some embodiments. [Figure 37A] FIG. 1 illustrates an example of a captured image including a solar panel and a torque tube, according to some embodiments. [Figure 37B] FIG. 37B illustrates an example of an annotated image of the captured image shown in FIG. 37A, according to some embodiments. [Figure 38A] FIG. 10 is a diagram illustrating an example of image classification. [Figure 38B] FIG. 38B shows an example of object localization for the image shown in FIG. 38A. [Figure 38C] FIG. 1 illustrates an example of semantic segmentation according to some embodiments. [Figure 39] FIG. 1 illustrates an example of instance segmentation of a solar panel according to some embodiments. [Figure 40] FIG. 1 illustrates an example image handling system according to some embodiments. [Figure 41] FIG. 1 illustrates a trailer system according to some embodiments. [Figure 42A]FIG. 10 illustrates a histogram of the norm of the pose error of a neural network when using coarse position, according to some implementations. [Figure 42B] FIG. 10 illustrates a histogram of the norm of the pose error of a neural network without coarse position, according to some implementations. [Figure 43A] FIG. 1 illustrates an example of the approximate location of solar panels using the B Mask R-CNN model, according to some embodiments. [Figure 43B] FIG. 1 illustrates an example of the approximate location of solar panels using the B Mask R-CNN model, according to some embodiments. [Figure 44] FIG. 1 illustrates a system for solar panel installation according to some embodiments. [Figure 45A] FIG. 1 illustrates a vision system for tracking the position of a trailer. [Figure 45B] FIG. 1 illustrates an expanded view of a vision system in accordance with some embodiments. [Figure 46A] FIG. 1 illustrates a vision system for module picking according to some embodiments. [Figure 46B] FIG. 1 illustrates an expanded view of a vision system in accordance with some embodiments. [Figure 47A] FIG. 1 illustrates a system for distance measurement at a modular angle according to some embodiments. [Figure 47B] FIG. 1 illustrates a system for distance measurement at a modular angle according to some embodiments. [Figure 47C] FIG. 1 illustrates a system for distance measurement at a modular angle according to some embodiments. [Figure 48A] FIG. 1 illustrates a system for generating laser lines to detect the position of the tube and clamp, according to some embodiments. [Figure 48B] FIG. 48B is an enlarged view of the laser line generating system shown in FIG. 48A. [Figure 48C]FIG. 10 illustrates laser line generation according to some embodiments (horizontal lines detect clamps, vertical lines detect tubes). [Figure 49A] FIG. 49 illustrates a vision system 4900 for estimating the position of the tube and clamp. [Figure 49B] FIG. 1 illustrates an expanded view of a vision system in accordance with some embodiments. [Figure 50A] 1 is a flow diagram of an autonomous solar power plant installation according to some embodiments. [Figure 50B] 1 is a flow diagram of a method for training a neural network for autonomous solar power installations according to some embodiments. [Figure 51A] FIG. 1 is a schematic diagram of an example method for estimating the centers and corners of a solar panel according to some embodiments. [Figure 51B] FIG. 1 is a schematic diagram of an example method for Hough line estimation in the presence of glare (or other optical phenomena) according to some embodiments. [Figure 52] FIG. 1 illustrates an example process for solar panel placement, according to some embodiments. [Figure 53] FIG. 1 is a schematic diagram of an example solar panel arrangement according to some embodiments. [Figure 54] FIG. 54 illustrates an example solar panel installation infrastructure with fan gear attached to a torque tube 5404, according to some embodiments. [Figure 55A] FIG. 1 illustrates a top view of an example solar panel installation infrastructure with torque tubes and clamps or fan gear, according to some embodiments. [Figure 55B] FIG. 1 illustrates an example process for performing clamp and fan gear detection according to some embodiments. [Figure 56] FIG. 1 illustrates an example application of the centroid method to estimate the center of a solar panel according to some embodiments. [Figure 57]1 is a flow diagram of an example method for autonomous solar power installation according to some embodiments. [Figure 58] 1 is a flow diagram of another example method for autonomous solar power installation according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0017] The features and advantages of the present invention will become more apparent from the following detailed description when taken in conjunction with the drawings, in which like reference numbers identify corresponding elements throughout. Generally, in the drawings, like reference numbers indicate identical, functionally similar, and / or structurally similar elements.

[0018] Reference will now be made in detail to the embodiments of the present invention, examples of which are illustrated in the accompanying drawings.

[0019] 1 is a perspective view of a solar panel handling system with a solar panel box according to one embodiment of the present disclosure. The solar panel handling system may include an end of arm assembly tool 100 that can couple to individual solar panels 120 from the solar panel box and move the panels to an installation position relative to an installation structure.

[0020] The end of the arm assembly tool 100 may include a frame 102 and one or more mounting devices 104 coupled to the frame 102. Some example mounting devices 104 include suction cups or other structures capable of releasably attaching to the surface of the solar panel 120 and maintaining the attachment, at least collectively, during manipulation of the solar panel 120 by the end of the arm assembly tool 100. The frame 102 may be comprised of multiple trusses 102-A to provide structural strength and stability to the frame 102. Additionally, the frame 102 also serves as a base for the end of the arm assembly tool 100 and other associated components of the solar panel handling system disclosed herein.

[0021] Other relevant components of the solar panel handling system disclosed herein may be coupled to frame 102 to fix the relative positions of these components on the end of arm assembly tool 100. One or more of the various components of the solar panel handling system may be coupled to one or more of trusses 102-A to fix the relative positions of the components on the end of arm assembly tool 100.

[0022] The attachment device 104 is configured to securely attach to a flat surface, such as the surface of a solar panel, for example, by using a vacuum. In one suction cup embodiment, the suction cup can be activated by pressing the suction cup against the flat surface, which forces air out of the suction cup and creates a vacuum seal against the flat surface. As a result, the flat surface adheres to the suction cup with an adhesive strength that depends on the size of the suction cup and the integrity of the seal to the flat surface. In some embodiments, the suction cup engages the solar panel to form an airtight seal, and then a vacuum pump draws air out of the suction cup, thereby creating the vacuum required for proper attachment to the solar panel. In some embodiments, when the flat surface is sealed to the suction cup, an air inlet (not shown) supplies air to the flat surface, thereby releasing the vacuum and releasing the flat surface from the suction cup.

[0023] The system may further include a linear guide assembly 106 coupled to the end of the arm assembly tool 100. The linear guide assembly 106 includes a linearly movable clamping tool 108 having an engagement member 108-A configured to engage a clamping assembly coupled to the installation structure. The linear guide assembly 106 may be actuated to move the clamping tool 108 along an axis, such as between an extended position and a retracted position. The axis of movement of the clamping tool 108 may be parallel to the axis of the installation structure. Thus, the linear guide assembly 106 is capable of moving the clamping tool 108 and the engagement member 108-A along the installation structure.

[0024] In some embodiments, the engaging member 108-A may comprise an electromagnet that can be actuated to grip the clamping assembly 602 (see FIGS. 6A, 6B). Alternatively or additionally, the engaging member 108-A may comprise a gripper to prevent disengagement between the clamping assembly 602 and the engaging member 108-A when the linear guide assembly 106 is actuated to move the clamping tool relative to the installation structure, as described in more detail elsewhere herein.

[0025] The linear guide assembly 106 is actuated using a force-torque transducer 110. In some embodiments, the linear guide assembly 106 and the force-torque transducer 110 may form a rack-and-pinion arrangement, in which case rotation of the force-torque transducer 110 results in the clamping tool 108 moving forward or backward. In some embodiments, the linear guide assembly 106 may be a hydraulic assembly including a telescoping shaft coupled to the clamping tool 108. In such embodiments, the force-torque transducer 110 may be in the form of a pump for pumping hydraulic fluid. In other embodiments, the force-torque transducer 110 may be in the form of or coupled to a linear drive motor that engages a surface of a telescoping shaft coupled to the clamping tool 108.

[0026] In some embodiments, the linear guide assembly 106 may include an electric rod actuator for moving the clamping tool 108 parallel to the axis of the installation structure.

[0027] In some embodiments, the guide assembly 106 may include rollers 606 to facilitate movement of the clamping tool 108 along the mounting structure 604. The rollers may include, for example, bearings or other components designed to reduce friction as the clamping tool 108 moves relative to the mounting structure. The rollers may be coupled to sensors, such as force or rotation sensors, to provide feedback to the controller.

[0028] In some embodiments, the guide assembly may include a spring mechanism 608 that allows for a small amount of tilt (up to 15 degrees) of the clamp tool 108 relative to the installation structure 604. Such tilt may occur when the orienting assembly 804 tilts the end of the arm assembly tool 100 relative to the installation structure 604 to properly level the solar panel.

[0029] The system may further include a junction box 112 coupled to the frame 102. The junction box 112 may include a controller configured to control the force-torque transducer 110 and the mounting device 104. In some embodiments, the junction box 112 may further include a power supply or power controller for controlling power to the various components.

[0030] In some embodiments, the controller 112 may include a processor operatively coupled to memory. The controller 112 may receive inputs from sensors associated with the solar panel handling system (such as, for example, a light sensor or proximity sensor 108-B described elsewhere herein). The controller 112 may then process the received signals and output control commands to control one or more components (such as, for example, the linear guide assembly 106, the clamping tool 108, or the mounting device 104). For example, in some embodiments, the controller 112 may receive a signal from a proximity sensor that determines that the clamping assembly is approaching the trailing edge of the solar panel being installed, and in response, reduce the speed of the linear guide assembly 106 to reduce excessive force and impact on the solar panel.

[0031] Referring to FIG. 8 , in some embodiments, the solar panel handling system may further include an optical sensor 802, such as a camera, a photodetector, or any other optical imaging or light-sensing device. The optical sensor may be suitably positioned on the frame 102, such as on the outer or lower surface of an edge member, as shown by position 802-A in FIG. 8 , or at an interior position of the frame 102 having a field of view that includes the front edge of the solar panel, as shown by position 802-B in FIG. 8 . The optical sensor may be configured to sense the orientation of the solar panel relative to the installation structure during operation of the arm assembly tool end. In some embodiments, the optical sensor may be configured in the form of one or more optical guidance levels (not shown). In such embodiments, one or more light beams (e.g., laser beams) may be projected from one end of the arm assembly tool 100 end, such as a first position on the frame 102, along or parallel to the axis of the installation structure 604. One or more photodetectors may be positioned at another end of the arm assembly tool 100 end, such as a second position on the frame 102, to detect the one or more laser beams. Thus, if the solar panel 120 being installed is not properly oriented or properly leveled relative to the installation structure 604, the solar panel 120 may block some or all of the one or more laser beams, resulting in a change in the signal from one or more photodetectors, which may indicate that the solar panel 120 is not properly oriented or properly leveled relative to the installation structure 604.

[0032] In some embodiments, one or more sensors, such as optical sensor 802, may be used to detect and recognize objects for improved accuracy in positioning and control of installation. These sensors may be implemented with a neural network, such as an artificial intelligence (AI) system. For example, the neural network may include acquiring and modifying images related to the solar panel handling system, the solar panels (both existing and to-be-installed), and the installation environment (both the natural environment, such as the terrain, and existing equipment, such as structures associated with the solar panel array). Further, for example, the neural network may include acquiring and modifying position or proximity information. The modified images and / or modified position or proximity information are input into the neural network and processed to estimate the movement and position of equipment in the solar panel handling system, such as those associated with autonomous vehicles, storage vehicles, robotic equipment, and installation equipment. The estimated movement and position are published to control systems associated with individual pieces of equipment in the solar panel handling system or to a master controller for the entire solar panel handling system.

[0033] In some embodiments, a signal from the optical sensor may be input to a controller. In some embodiments, the solar panel handling system may further include an orienting assembly 804 (see FIG. 8 ) configured to tilt the end of the arm assembly tool 100 relative to the installation structure 604. In such an embodiment, the controller 112 may control the orientation in response to input from the optical signal that indicates that the solar panel being installed is not properly oriented or properly leveled relative to the installation structure, such as the torque tube 604. Although the orienting assembly 804 is illustrated as being coupled to the force-torque transducer 110, other means of implementing the orienting assembly 804 will be readily apparent to those skilled in the art.

[0034] In some embodiments, the controller 112 may be configured to control the attachment device 104 to activate or deactivate the attachment / detachment of the attachment device 104. In embodiments where the attachment device 104 is a suction cup, a vacuum may enable coupling or detachment of the solar panel 120 to the end of the arm assembly tool 100.

[0035] In some embodiments, the mounting structure 604 may have an octagonal cross-section, such as shown in Figures 6A, 6B, and 7A-7D, which may form a torque tube that prevents unintentional sliding of the clamp assembly 602. However, other cross-sectional shapes may be used, such as, for example, a square, oval, or other shape. Additionally, the mounting structure 604 may use a circular cross-sectional shape.

[0036] In some embodiments, the assembly tool 100 may be configured to couple to an assembly mobile robot 903 (an example of which is shown in FIGS. 9 and 10 ). The assembly mobile robot 903 may be configured to position the end of the arm assembly tool 100 relative to a stack or storage container 905 of solar panels, move a selected solar panel, and position the selected solar panel relative to the installation structure 604. In some embodiments, the assembly mobile robot 903 may be operatively coupled to the end of the arm assembly tool 100 via a force transducer 110 (or an orienting assembly 804, if applicable). In some embodiments, the assembly mobile robot may further be operatively coupled to a controller, allowing an operator of the assembly mobile robot to control various functions of the end of the arm assembly tool 100, such as, for example, actuating and / or deactuating the attachment device 104, advancing and / or retracting the clamping tool, and / or actuating and / or deactuating an engagement member relative to the clamping assembly.

[0037] 1 , 6A, 6B, 7A-7D, 9, and 10, in operation, a solar panel 120 is grasped and positioned on the mounting structure 604. The solar panel is then tilted relative to the mounting structure 604 so that the leading edge of the solar panel (i.e., the edge that will be adjacent to the edge of a previously installed solar panel, or in the case of a first solar panel, the edge that will be adjacent a stop secured to the mounting structure 604) is oriented closer to the mounting structure 604 than the opposite trailing edge. The leading edge is then placed into a receiving channel (either a receiving channel positioned along the edge of a previously installed solar panel, either as part of a clamp assembly or in a stop) and the tilt of the solar panel is reduced to an installed position on the mounting structure. The tilt angle is reduced so that, while the solar panel is urged into the receiving channel, the edge region of the top flat surface of the solar panel (i.e., the photovoltaic active surface oriented toward the sun) is captured within the receiving channel in the installed position. An example of a receiving channel 610 on the clamp assembly 602 is shown in Figures 6A and 6B.

[0038] Once the solar panel is in position on the installation structure, the force-torque actuator 110 actuates the guide assembly 106 on the end of the arm assembly tool 100 to contact the engaging member 108-A of the clamping tool 108 with the clamping assembly 602. This clamping assembly was originally positioned outside the area on the installation structure that the solar panel would occupy, but close enough for the associated components on the end of the arm assembly tool 100 to reach. The surfaces and features of the engaging member 108-A may be positioned and sized to match complementary features on the clamping assembly 602. After this contact, the force-torque actuator 110 is actuated (either by continuing to actuate or by actuating in a second mode) to slide the clamping assembly 602 axially over a portion of the length of the installation structure 604. The axial sliding of the clamping assembly 602 engages the receiving channel of the clamping assembly 602 with the rear end of the most recently installed solar panel. A sensor located within the force-torque actuator 110 or clamping tool 108, etc., can provide feedback to the controller indicating when the receiving channel of the clamping assembly 602 has fully engaged the rear end of the solar panel. Once the clamping assembly 602 is positioned, the guide assembly 106 retracts, allowing the next solar panel installation to occur.

[0039] In some embodiments, the linear guide assembly 106 may include a proximity sensor 108-B configured to sense the distance between the engagement member 108 and the rear edge of the solar panel 120 during installation of the solar panel 120. The output from this proximity sensor 108-B may be used to avoid excessive force and impact on the solar panel 120 by appropriately controlling the speed of the clamping tool 108 during operation of the linear guide assembly 106. In some embodiments, the proximity sensor 108-B may be, for example, an optical sensor or an audio sensor (e.g., sonar) that detects the distance between the front edge of the solar panel 120 and the engagement member 108. In other embodiments, the proximity sensor 108-B may be a limit switch that is retracted upon contact.

[0040] 9 and 10 , the assembly mobile robot 903 may be implemented using a land vehicle 907. For example, the land vehicle 907 may be implemented as an electric vehicle (EV). The land vehicle 907 may move autonomously in the vicinity of the installation structure 604. Although not shown, the land vehicle 907 may move along a track or rail attached to or spaced apart from the installation structure. In some embodiments, the land vehicle 907 may be controlled using sensors or based on input or feedback from sensors. These sensors may be, for example, optical sensors or proximity sensors. In further embodiments, a neural network using artificial intelligence may be used to control the movement of the land vehicle 907, for example, by analyzing the operating environment and developing instructions for the movement of the land vehicle.

[0041] FIG. 10 shows an embodiment of a solar panel handling system with two robotic arms, where two assembly tools are coupled to an assembly mobile robot using their respective robotic arms.

[0042] As shown in FIG. 9 , a storage container 905 containing the solar panels to be installed may be disposed on a land vehicle. Here, FIG. 9 illustrates a solar panel handling system including an arm assembly tool 100 coupled to an assembly mobile robot using a robotic arm. Alternatively, as shown in FIG. 10 , one or more storage containers 905 may be disposed on each one or more of the modular vehicles 1005 adjacent to the land vehicle 907. Thus, FIG. 10 illustrates a solar panel handling system with two robotic arms, in which case two assembly tools are coupled to the assembly mobile robot using respective robotic arms. In embodiments of the present disclosure, these robotic arms may be articulated arms having two or more sections connected by joints, or alternatively, may be truss arms. The drawings herein are intended to disclose the use of any type of arm according to the present disclosure.

[0043] Referring to FIG. 9 , a robotic arm, such as arm assembly tool 100 having upper and lower sections 908 and 909, can increase freedom of movement while maintaining light weight and operational simplicity. As further shown in FIG. 9 , a second robotic arm 911 may include arm assembly tool 100 having a nut runner or nut driver at its end for fastening the solar panel to installation structure 604. While any type of robotic arm may be used for second robotic arm 911, FIG. 9 illustrates the use of an articulated arm having a nut runner or nut driver at its end. In this case, robotic arms 100 and 911 may operate autonomously using computer vision with neural networks and artificial intelligence control. Alternatively, robotic arms 100 and 911 may be manually or remotely operated.

[0044] In some embodiments, the land vehicle 907 may be an autonomous vehicle, where neural networks and artificial intelligence control movement and operation, and the modular vehicles 1005 are towed or coupled to the land vehicle 907. In other embodiments, the modular vehicles 1005 may be autonomous vehicles, where neural networks and artificial intelligence control movement and operation, and the land vehicle 907 is towed or coupled to the modular vehicles 1005. Additionally, in some embodiments, the assembly mobile robot 903 is mounted on one of the land vehicle 907 and the modular vehicles 1005. In other embodiments, the assembly mobile robot 903 may be mounted on a dedicated robotic vehicle.

[0045] A process for installing solar panels is shown in FIGS. 11A-11C. As shown in FIG. 11A, pallets of solar panels may be delivered by truck. In some embodiments, the pallets may constitute storage containers 905 for the solar panels. The pallets may include machine-readable symbols, such as barcodes, QR codes, or other manufacturing references, that can be read to provide information about the solar panels, installation procedures, or other information used in the installation process, particularly information used by neural networks and artificial intelligence control. Such information may include, for example, the number of solar panels, the type of solar panel, physical characteristics such as the size of the solar panels, installation-related characteristics such as the type and location of hardware, installation procedures, or other characteristics of the solar panels, storage of the solar panels on the pallets, and information related to installation. Furthermore, by using the machine-readable symbols, the system can control the supply or replenishment of panel boxes in the correct order and / or ensure that panels with similar impedance from the factory are used.

[0046] As shown in FIG. 11B, a mechanized device such as a forklift may be used to move and position the pallet on the land vehicle. In this case, the forklift may be manually operated, remotely operated, or autonomous. In FIG. 11B, the pallet is positioned on the land vehicle. Alternatively, the pallet may be positioned on a modular vehicle. Then, as shown in FIG. 11C, a robotic arm is used to install the solar panels. In the illustrated example, two arms are used to handle each solar panel to be installed on each installation structure. In this case, the land vehicle moves between the two installation structures. Additionally, one modular vehicle is provided, but this may be separate from the land vehicle.

[0047] As will be appreciated by those skilled in the art, modifications and variations of the implementation may be made. For example, as shown in Figures 12A and 12B, two modular vehicles may be provided for each robotic arm. In a further alternative, the modular vehicles may be coupled to the land vehicle without being spaced apart. Thus, as shown in Figure 12A, the robotic arm may engage with each solar panel to be installed, as shown in Figure 12B.

[0048] In some embodiments, placement may be achieved using computer vision registration, as shown in Figure 13. For example, as described above, optical sensors and the like may be used in conjunction with neural networks for artificial intelligence.

[0049] 14, in some embodiments, when modular vehicles are used with land vehicles, the modular vehicles may be replaced with replenished modular vehicles when all of the modular vehicle's solar panels have been installed. In this case, computer vision processes may be used to communicate with and control an autonomous stand-alone vehicle, such as a forklift, to deliver additional solar panel boxes. In this manner, the supply of solar panels may be replenished.

[0050] In a replenishment operation using the forklift example, a forklift (whether autonomous, remotely controlled, or manually operated) can be used to return empty boxes or containers of solar panels to a disposal area, remove straps from boxes being delivered, open lids or cut off box faces, pick up boxes to correct the rotation / orientation of the solar panels, or perform other tasks. Additionally, the forklift may remain near the land vehicle to wait for the system to empty the next box of solar panels. Thus, the forklift can manually or autonomously discard the emptied box, position the next box on the land vehicle or modular vehicle, unpack the box (including removing straps, opening lids, or cutting off box faces), and back away from the land vehicle / modular vehicle. As noted above, this replenishment may be, for example, autonomous, remotely controlled, or manually operated.

[0051] 15 to 34 show detailed diagrams of an example of the configuration of a system for installing solar panels according to an embodiment of the present disclosure.

[0052] 35A shows a block diagram of an example image processing pipeline 3500 according to some embodiments. Pipeline 3500 includes a module 3502 for acquiring an image, a module 3504 for rectifying the image, a module 3506 for neural network image segmentation of the rectified image, a module 3508 for post-processing the output of module 3506 using computer vision techniques, a module 3510 for performing a Hough transform on the output of module 3508, a module 3512 for filtering and segmenting the Hough lines output by module 3510, a module 3514 for identifying intersections of the horizontal and vertical Hough lines output by module 3512, a module 3516 for estimating the pose of the panel based on the intersections of the horizontal and vertical Hough lines, and a module 3518 for publishing the pose estimate (e.g., using the 3D panel shape and the locations of corners in the image, etc.). Figure 35B shows an example corrected acquired image 3520 (output of modules 3502 and 3504) including an image of a solar panel 3522 and other objects 3524-2 (e.g., tape, etc.) and 3524-4 (e.g., wire, etc.). Figure 35C shows an example output 3526 (output of module 3506) for neural network image segmentation of the acquired corrected image shown in Figure 35B, according to some embodiments. Figure 35D shows an example panel corner detection 3528 (output of module 3514), according to some embodiments. In this example, corners 3530-2 and 3530-4 are detected based on horizontal lines 3532-4 and 3532-8 and vertical lines 3532-2 and 3532-6.

[0053] 36 shows example 3600 images 3602, 3606, and 3610 of a road under various lighting conditions and image segmentation masks 3604, 3608, and 3612, according to some embodiments. Traditional computer vision techniques are useful when the environment is ideal. However, glare and overexposure / underexposure can have a negative impact on object detection algorithms. Machine learning techniques can overcome environmental inconsistencies and learn from examples with glare and lighting issues, and may have the ability to generalize to new data during inference.

[0054] Solar panel segmentation example Some embodiments perform solar panel segmentation by capturing images of the solar panel and torque tube under various lighting conditions. FIG. 37A shows an example of a captured image 3700 including a solar panel 3702 and a torque tube 3704, according to some embodiments. Some embodiments annotate the captured image of the solar panel. FIG. 37B shows an example of an annotated image 3706 (sometimes referred to as an annotated ground truth mask) of the captured image 3700, according to some embodiments. The annotated image includes a black background 3708, an outline of the torque tube 3712 shown in dark gray, and an outline of the solar panel 3710 shown in light gray. Some embodiments generate a dataset based on the annotated images, use the dataset to train an image segmentation model, and use the trained model to detect the solar panel and torque tube in poor lighting conditions. FIG. 37C shows an example prediction 3714 from the trained model, according to some embodiments. This trained model predicts the background 3708, the solar panel 3710, the torque tube 3712, and an object 3716 in the background (not shown in Figures 37A and 37B).

[0055] In some embodiments, images are continuously collected (and a dataset is built) and used to improve the accuracy of the model. Some embodiments improve the accuracy of the model by using human annotations. In some embodiments, the user can adjust the parameters of the segmentation model.

[0056] Some embodiments provide separate models for semantic segmentation and instance segmentation. Figure 38A illustrates an example of image classification. In this example, image classification detects the presence of a bottle 3802, a cup 3806, and a cube 3804. Figure 38B illustrates an example of object localization 3816 for the image shown in Figure 38A. In this example, rectangle 3808 locates bottle 3802, rectangle 3810 locates a first cube, rectangle 3812 locates cup 3806, and rectangles 3814-2 and 3814-4 locate cube 3804. Figure 38C illustrates an example of semantic segmentation 3818 according to some embodiments. Semantic segmentation helps identify label 3820 for bottle 3802, label 3822 for cube 3804, and label 3824 for cup 3806. FIG. 38D shows an example of instance segmentation 3826 according to some embodiments. Apart from identifying labels 3828 and 3832 for bottles 3802 and 3806, respectively, instance segmentation can determine labels 3830, 3834, and 3836 for cube 3804 to distinguish between instances of cube 3804. Instance segmentation can distinguish between multiple solar panel instances within a single image. Instance segmentation generates a mask for each class instance within a camera frame, enabling identification and localization of individual panels, using the same data collected for semantic segmentation and supporting lighting invariance. FIG. 39 shows an example of solar panel instance segmentation 3900 according to some embodiments. In this example, panel instances 3902, 3904, 3906, 3908, and 3910 are identified. This example shows instances of a panel (such as instances 3902 and 3904) each with a different orientation.

[0057] FIG. 40 illustrates an example image handling system 4000 according to some embodiments. The system 4000 includes multiple cameras: a camera 4002 for coarse positioning, a camera 4004 for capturing images as panels are picked, and a camera 4006 for capturing images as panels are placed. Camera 4002 includes a narrow field of view lens, while cameras 4004 and 4006 each include a wide field of view lens. Camera 4002 may be used to identify the trailer position and the robot initial position. In some embodiments, cameras 4004 and 4006 may be the same camera. Additionally, in some embodiments, camera 4002 may be used to locate clamps and a central structure during solar panel installation. Cameras 4002, 4004, and 4006 are coupled to image sensors 4008, 4010, and 4012 (e.g., AR0820 sensors), respectively. In some embodiments, these image sensors are optimized for both low light and / or high dynamic range performance. In some embodiments, system 4000 includes a high-speed digital video interface (e.g., FPD-Link, etc.) and Ethernet to connect the camera to one or more GPUs (e.g., a GPU 4014 suitable for edge AI processing such as Nvidia XT, a GPU suitable for image processing applications such as Nvidia AGX Xavier®, etc.). GPU 4016 implements the example image processing pipeline 3500 described above and is connected to robot controller 4018 using Ethernet. In some systems, GPU 4014 may be removed, and according to some embodiments, output from sensors may be connected directly to GPU 4016.

[0058] Some embodiments continue capturing training images while installing solar panels. Figure 41 shows a trailer system 4100 with a coarse-resolution camera 4102 that can be used to capture training images, according to some embodiments. The AI / neural network system calculates the position of each of the panel's four corners by considering internal parameters (e.g., camera / lens distortion) and external parameters (e.g., the position and angle of the camera on the robot arm and the pose of the robot arm at the moment of image capture).

[0059] 42A and 42B show histograms 4200 and 4202 of the norm of the pose error of a neural network with and without a coarse position, according to some implementations. As shown, using a coarse position significantly reduces the error of the neural network (the difference between the actual position of the solar panel corners and the estimate from the neural network) (in some cases, down to about 5 inches to 0.7 inches).

[0060] 43A and 43B show examples 4300 and 4302 of approximate solar panel locations (e.g., locations 4304, 4306, 4308, and 4310) using the B Mask R-CNN model in accordance with some embodiments. Mask R-CNN is a convolutional neural network (CNN) used for image segmentation and instance segmentation. This deep neural network detects objects in an image and generates high-quality segmentation masks for each instance. Mask R-CNN is based on a region-based convolutional neural network. Image segmentation is the process of partitioning a digital image into multiple segments or sets of pixels that correspond to image objects. This segmentation is used to locate objects and boundaries (lines, curves, etc.). Mask R-CNN can be used for semantic segmentation and instance segmentation. Semantic segmentation classifies each pixel into a fixed set of categories without distinguishing between object instances. In other words, semantic segmentation corresponds to identifying / classifying similar objects as a single class at the pixel level. All objects are classified as a single entity (solar panel). Semantic segmentation is sometimes called background segmentation because it separates the subject matter of an image (e.g., solar panel, wires, etc.) from the background. On the other hand, instance segmentation (sometimes called instance recognition) addresses the accurate detection of all objects in an image and further provides accurate segmentation of each instance. In that sense, instance segmentation combines object detection, object localization, and object classification and helps identify instances of each object in an image. In Mask R-CNN, in addition to the two outputs for each candidate object, including the class label and bounding box offset, a third branch outputs the object mask.This mask output helps extract a finer spatial layout of the object. In addition to being easier to train, performing well, and being more efficient than other models, Mask R-CNN is particularly well-suited for solar panel identification because the neural network has the ability to perform both semantic and instance segmentation. Furthermore, the mask branch adds only a small amount of computational overhead, allowing for faster solar panel detection and more rapid experimentation. Mask R-CNN can be used for image segmentation, identifying objects in an image and generating masks within the object's boundaries.

[0061] 44 illustrates a system 4400 for solar panel installation, according to some embodiments. According to some embodiments, the system 4400 includes a main enclosure 4404, a battery enclosure 4402, an upper robotic end effector (EOAT) 4406, a lower robotic EOAT 4408, and a cradle 4410 for holding a solar panel 4414 on a trailer 4412.

[0062] Figure 45A shows a vision system 4502 mounted on a trailer and used to estimate the pose of the structure 4500, and Figure 45B shows a close-up view of the vision system 4502 according to some embodiments. Various embodiments may have the vision system mounted on different parts of the land vehicle, on a robotic arm, or at the end of an end effector.

[0063] FIG. 46A shows a vision system 4602 for module picking 4600, and FIG. 46B shows a close-up view of the vision system 4602 with a high-resolution camera with laser line generation according to some embodiments.

[0064] FIG. 47A shows a system 4700 for performing distance measurements at a module angle (i.e., when facing the module) between position 4702 (shown in an enlarged view in FIG. 47B) and position 4704 (shown in an enlarged view in FIG. 47C), according to some embodiments.

[0065] Figure 48A shows a laser line generation system 4800 for detecting the position of the tube and clamp according to some embodiments, Figure 48B shows a close-up view of the laser line generation system 4802, and Figure 48C shows a diagram 4804 of the laser line generation according to some embodiments (the horizontal line detects the clamp and the vertical line detects the tube).

[0066] Figure 49A shows a vision system 4900 for estimating the position of the tube and clamp and locating the nut on the clamp, according to some embodiments. Figure 49A also shows a socket wrench 4902 used to tighten the nut. Figure 49B shows a close-up of this vision system. As shown in Figure 49B, a camera uses the laser lines mentioned above to locate the tube and clamp, and a flash ring light to locate the nut on the clamp. These lasers allow for accurate estimation of the position of the tube and clamp. The flash ring light is used to locate the nut on the clamp, as shown in Figure 49C. When tightened, this nut compresses the clamp to hold the panel in place.

[0067] Example of solar panel installation using AI FIG. 50A shows a flow diagram of a method 5000 for autonomous solar power installation, according to some embodiments. The method includes acquiring 5002 images of the solar power installation in progress. The images include images of one or more solar panels and one or more torque tubes. In some embodiments, acquiring the images includes using one or more filters to avoid direct sunlight glare to detect the end effector (EOAT). In some embodiments, acquiring the images includes using a high-resolution camera with laser line generation to identify the location of one or more torque tubes and / or clamps. In some embodiments, the images include images of the clamps and / or central structure for the solar power installation in progress. In some embodiments, multiple images are acquired using a wide-angle fisheye lens to generate a composite HDR (high dynamic range) image within the camera hardware. These images are sent through the Robot Operating System (ROS), a high-level software framework for integrating robots and servos, and the images are corrected (e.g., from fisheye distortion to a flat image) using OpenCV (an image processing framework) modules. The region and bit depth are then selected and used to reduce the HDR image to a standard 8-bit image, which effectively crops the region and bit depth to adjust as input for the trained neural network. At the time of acquisition, the robot pose may be saved (using ROS) to generate a translation camera result relative to the trailer (trailer system used for solar panel installation). This may include the robot position and camera position to identify the location of the image in 3D space.

[0068] The method further includes detecting solar panel segments by inputting the images into a trained neural network (5004) that is trained to detect solar panels in poor lighting conditions. The neural network may be implemented using software and / or hardware (sometimes referred to as neural network hardware) using conventional CPUs, GPUs, ASICs, and / or FPGAs. In some embodiments, the trained neural network is composed of (i) a model for semantic segmentation to identify solar panel segments and (ii) a model for instance segmentation to identify multiple solar panels. In some embodiments, the trained neural network utilizes the Mask R-CNN framework for instance segmentation. These trained neural networks detect solar panel segments based on features extracted from images of ongoing solar installations. In some embodiments, the acquired images are input into the neural network via ROS (e.g., the input images go from an OpenCV module to a neural network module (Detectron)). An example technique for training a neural network according to some embodiments is described below with reference to FIG. 50B. In some embodiments, the neural network performs image segmentation to identify the panel (or panels) without identifying the panel's location. In some embodiments, there is one model that performs both functions (semantic segmentation and instance segmentation). Some embodiments use two instances of the same model to optimize throughput. In such an example, the camera takes two images, one image passing through each instance. By running two models, twice as many images can be processed simultaneously.

[0069] The method further includes utilizing a computer vision pipeline to estimate 5006 a panel pose of one or more solar panels based on the solar panel segments. In some embodiments, the computer vision pipeline includes one or more computer vision algorithms for post-processing, Hough transform, filtering and segmenting Hough lines, finding horizontal and / or vertical Hough line intersections, and panel pose estimation using a given 3D panel shape and corner locations. In some embodiments, the computer vision pipeline estimates the panel pose by locating clamps and / or central structures. In some embodiments, the computer vision pipeline estimates the panel pose by locating one or more torque tubes and / or clamps. In some embodiments, the computer vision pipeline locates nuts. After locating the nuts, a socket wrench mounted on a smaller robotic arm engages and tightens the nuts to secure the panel in place. Prior to this step, the clamps are loose and the panel may fall due to wind.

[0070] In some embodiments, panel pose estimation is performed using conventional machine vision hardware to identify the panel's location in 3D space. In some embodiments, this is a rough identification of rounded edges and is not intended to be very precise. A Hough transform is then used to determine the exact location of the edges, followed by extrapolation of the panel's edge lines, determination of where the panels intersect, and identification of panel corners. The panel corners are published to identify the panel's location relative to the robot. For example, based on the panel's shape in 3D, the panel's pose is calculated based on the location of the panel corners in the image.

[0071] In some embodiments, to estimate the pose of the panel, the computer vision pipeline utilizes a PnP (Perspective n-Point) solver with the camera's intrinsic parameters (aware of the camera's inherent distortion and parallax). The extrinsic parameters then capture the camera's position relative to the robot using the pose of the robot arm and EOAT at the moment of image capture. The robot's pose may be continuously captured with a timestamp, which can then be used to match the robot's pose to the camera acquisition timestamp. In some embodiments, the computer vision pipeline utilizes the known pose of the robot arm and end effector (where the camera is mounted) at the time of image capture to calculate the position of one or more corners of the panel.

[0072] The method further includes generating 5008 control signals to operate a robot controller to install one or more solar panels based on the estimated panel pose. In some embodiments, after a panel is found, its location is projected along the tube, which locates clamp pixels to identify clamp locations (e.g., how far apart the clamp is, how close the clamp is to a clamp puller, etc.). In some embodiments, the clamp locations are used to verify that the clamps are located within a tolerance window required by the clamp puller on the EOAT. Some embodiments use a central structure to determine the sequence for placing one panel or two panels to avoid collisions with the fan gear. Some embodiments use the panel locations to verify that the trailer is in a valid position relative to the tube so that the robot is within reach of the tasks it needs to perform. Some embodiments use the pose of the leading panel to guide the lower robot's precise tube information acquisition, which drives the positions of the upper and lower robots for panel placement and nut driving. In some embodiments, the above-described precise tube information acquisition forms a profilometer system that uses horizontal and vertical lasers to locate the tube and clamp positions. This profilometer system refines the working position, which has a 10-20 mm error based on the coarse tube information, and reduces that error to less than plus or minus 5 mm. In the first panel, the error from the coarse tube is within 5 mm, but this becomes larger as it is projected, and is limited to less than plus or minus 5 mm by using the precise tube information.

[0073] FIG. 50B shows a flow diagram of a method 5010 for training a neural network for autonomous solar power installations, according to some embodiments. The method includes acquiring multiple images of a solar panel installation under various lighting conditions (5012), annotating the multiple images to identify solar panel images (5014) (human-annotated images may be used instead of or in addition to automatically annotated images), and detecting solar panels in poor lighting conditions by training one or more image segmentation models using the solar panel images (5016). In some embodiments, the neural network is manually trained using a variety of images, such as multiple images of panels with or without clamps and cardboard corners, on various backgrounds (e.g., grass, dirt, etc.), with various panel quantities, and under various weather conditions (e.g., sunny conditions, rainy conditions, etc.). Masks (lines) are drawn within the images to indicate which pixels represent panels, clamps, tubes, and central structures. Using these images and their masks, a series of pseudo-images are generated, which are then used by the neural network in the training process. These pseudo-images are input images with angular distortions to allow for multiple training sessions using the same input image. For example, 300-1000 real (input) images may be used for training, and 10-20 pseudo-images may be generated for each real image.

[0074] Synthetic images can be used to train machine learning algorithms for solar panel installation. However, synthetic images or other synthetic training data can be prohibitively expensive and, in some cases, impossible to generate. Producing realistic images can require weeks of training, even on expensive hardware, and can require intensive supervision. Generating useful synthetic data is typically a trial-and-error process. Furthermore, there are risks associated with overtraining using synthetic data. Artificial data is often used in areas where real-world data is scarce, so the generated data may not accurately reflect real-world scenarios. Given these constraints, there is a need for systems and methodologies that do not rely on synthetic data. The techniques described herein are not site-limited because no images are used for training. These techniques are panel-centric, and background profiles do not affect accuracy. Some embodiments utilize a predictor-corrector mechanism to predict the center, test for faults, and use the faults as feedback to improve subsequent estimates of this center.

[0075] Some embodiments extract the relative position of the sun in the sky by using solar physics, solar elevation angle, and / or azimuth angle. Some embodiments estimate glare based on the estimate of the sun's position. Some embodiments apply pseudocolor to the glare by utilizing high-fidelity noise reduction and image correction algorithms. Some embodiments identify panel centers and corners, detect wear on torque tubes, apply shaded pseudocolor to wear lines / scratches (e.g., blue shading), position panels, and / or detect structures such as clamps and fan gears.

[0076] Some embodiments pick solar panels by detecting the corners and centers of the panels to help the upper robot (sometimes called the upper robot arm) accurately pick the solar panels. Panels can be of various sizes. Some embodiments detect an already placed panel and help the lower robot accurately align its panel relative to the already placed panel. Some embodiments detect the position of the clamps and assist the operator in moving the clamps. The panels are protected from impacts. For example, some embodiments detect the position of the fan gear and assist the operator in safely placing the solar panel relative to the fan gear. Additionally, the panels are protected against collisions. These aspects are described in more detail below.

[0077] According to some embodiments, the first step in picking a solar panel is to estimate the location of the panel's center. An upper robotic arm (such as the upper robotic end effector (EOAT) 4406) can use this center estimate to pick the panel. These solar panels may be of various sizes but have the same overall shape, primarily rectangular, even if the solar panels are supplied by different manufacturers. The algorithmic techniques described herein use the rectangular shape of the solar panel to estimate the center. Sizes may vary, i.e., the width and length of the solar panel may change, which affects the location of the panel's center and, consequently, the locations of the panel's corners. While described herein with respect to rectangular solar panels, these methods and techniques may also be applied to, or appropriately modified for use with, other polygonal-shaped solar panels.

[0078] After identifying the center and corners, the upper arm can pick a solar panel and enter a placement mode. This placement mode requires the picked panel to be aligned with the torque tube and with the existing (or previously placed) panel to avoid collision with the existing (or previously placed) panel. Additionally, some embodiments avoid collisions with other structures, such as clamps, located in the installation environment. Because the placed panel must not collide with the existing panel and must be properly aligned with it, once this alignment process is complete, in some embodiments, the lower robotic arm (e.g., lower robot EOAT 4408) rises and tightens the clamp screws. Thus, the upper robotic arm picks and places the solar panel, and the lower robotic arm optionally uses a laser to ensure alignment. In some embodiments, image recognition software detects other structures, such as fan gears, located in the installation environment, and the operator is notified so that the panel placement can be offset to avoid collision or impact with the structure.

[0079] The movement of the sun can have an impact during picking, placement, and / or alignment. The movement of the sun can depend on the time of day and / or the season. Some embodiments utilize the solar elevation angle and / or the solar azimuth angle. The azimuth angle is the angle at which the sun strikes the ground. The solar elevation angle is the position of the sun in the sky on a particular day and can be particularly important for solar panel installations. Typically, solar panels are topped with an inherently reflective coating. The physical properties of the sun must be taken into consideration. High transmittance is desirable to allow more photons to be converted into electrons, but excessive conversion results in reduced performance of the solar panel. Therefore, to protect the solar panel from this unwanted radiation, some embodiments apply an inherently reflective thin film coating to the solar panel. Due to the movement of the sun on a particular day and the orientation of these solar panels during picking, many reflections and glare are seen on the panels. As a result, when a camera captures the image, dark white spots appear in the image, which can affect and confuse computer vision algorithms.

[0080] Another failure mode is when clamps must be manually adjusted, so they must be moved along the torque tube. This is a physical movement of the clamps, which involves metal-to-metal contact. Moving the clamps on the torque tube can scratch or scuff the torque tube. When viewed by the lower robotic arm, this scratch will reflect white against a white background. The laser line, which was intended to be an alignment indicator, will be focused on the metal object, thereby reflecting white. From an imaging perspective, the background is also white, so the two colors blend together, making it impossible to discern the location of the laser line. In some embodiments, this is avoided by using colored (e.g., blue) tape so that a contrasting background is present. These issues necessitate a non-synthetic database system. The technology described herein provides similar functionality to that described above for picking, placing, alignment, and shape-based collision and / or impact avoidance.

[0081] Example of a method for estimating the geometric center and corners of a panel FIG. 51A is a schematic diagram of an example method 5100 for estimating the center and corners of a solar panel, according to some embodiments. The center P and corners of the panel 5104 can be estimated by distance simulation, geometric correction, and / or angle adjustment. Straight lines within the panel can be identified, for example, by using Hough lines or a Hough transform. Some embodiments identify and draw the longest visible horizontal and vertical Hough lines, which indicate the edges of the panel. The longest visible lines are useful because the image 5102 may only be a partial image or may only cover a portion of the panel 5104. Some embodiments randomly select a point (e.g., an interior point A or an exterior point E) from the image (any point within the boundary of the image 5102) and determine whether the point is located inside or outside the panel. The image is a combination of foreground and background, where the background corresponds to the solar panel's environment, such as the support on which the solar panel sits. The image can be a top view, a ground profile, or a moving object, and any part of the image can be the background. If a ray emanating from a point intersects the longest visible line at an odd point, the point is located inside the panel. If the ray intersects the longest visible line at an even point, the point is located outside the panel. If the point is located inside the panel, some embodiments calculate the horizontal and vertical distances to the Hough lines.

[0082] Referring to FIG. 51A , an assumption is made that point A is determined to be an interior point. Two normals AB and AC are drawn to the longest visible horizontal line 5106 and the longest visible vertical line 5108, respectively. These lines are not perpendicular to the panel; that is, the angles α and β where these normals intersect with the longest visible horizontal and vertical lines are examples where the angles formed by the light rays are not 90 degrees. This is because these angles are 90 degrees only when the panel is aligned with the image. Such 90-degree angles can occur when the longest visible horizontal line and the image boundary are parallel to each other. This typically does not occur, so α and β will be values ​​either less than or greater than 90 degrees. Some embodiments create the matrix by repeating these steps for all interior points.

[0083] Some embodiments may use the following characteristics: (a) horizontal distance (d horizontal ) is greater than or equal to 45% (or the maximum) of the panel width, which may be user-entered or previously calculated; and (b) the vertical distance (d vertical ) is greater than or equal to 45% (or a maximum value) of the panel height, which may be a user-entered or previously calculated height. Utilizing this tolerance level is important because solar panels may be supplied by different vendors and panel width is not a fixed value. While 45% is used for illustrative purposes, other user-defined values ​​may be used. Various embodiments may use different percentage values ​​depending on the need for error tolerance or precision.

[0084] Some embodiments calculate both the horizontal intersection angle (α) and the vertical intersection angle (β) with the longest visible Hough line from each eligible point. Some embodiments use the calculated angles to modify the estimated distance. The angles α and β may require normalization, i.e., if these distances are evaluated as vertical or horizontal distances, they may need to be corrected based on 90 minus α or 90 plus α and / or 90 minus β and 90 plus β. β equals 0, which means that α equals 90 degrees. Therefore, these angles α and β are used as a guide.

[0085] Some embodiments isolate a subset of points that satisfy the following characteristics: (a) horizontal distance exceeds a predetermined percentage (e.g., 48%) or a maximum panel width; and (b) vertical distance exceeds a predetermined percentage (e.g., 48%) or a maximum panel height. Point A is d vertical is about 50% of the panel height, d horizontalThe solar panel center P is asymptotically approached when α is 50% of the panel width and / or α and β are each approximately 90 degrees. These points are sometimes called representative points. These points are located closer to the center of the solar panel and correspond to bubbles depicted on a circle (circle of equal probability, or CEP). These points are distributed in a circular spread. Points of interest for estimating the geometric center are (i) located inside the solar panel, (ii) located near the longest visible horizontal and vertical Hough lines, and (iii) oriented approximately normal to the longest visible horizontal and vertical Hough lines. Since 50% means center, points that satisfy at least 48% therefore have a CEP error probability or circular error probability of less than 2%. As long as the points being evaluated satisfy the 48% distance condition, they are located within 2% of the geometric center, but they are still a set of points. They are not a single point. Some embodiments identify the center of the panel by determining the centroid of a set of points. Some embodiments utilize the Pythagorean theorem to estimate the corners of the panel in both directions. Some embodiments determine the corners of the panel by correcting the distance using the angle. The value 48% is used for illustrative purposes, but other user-defined values ​​may be used. Various embodiments may use different percentage values ​​depending on the error tolerance or accuracy required.

[0086] The center of the panel can be used to control the attachment device of the suction cups (e.g., eight suction cups) on the upper robotic arm. These suction cups may be symmetrically distributed on the robotic arm. By centering the robotic arm relative to the solar panel, misalignment during picking of the solar panel is avoided. If the solar panel is not picked with the correct orientation, i.e., if tilting occurs during picking, it will take much longer to place the panel and the panel will need to be aligned next to the previously placed panel.

[0087] One goal may be to complete the pick and place process in less than 60 seconds.

[0088] Some embodiments utilize CEP techniques to estimate the center based on proximity and perpendicularity and may not include all of the steps described above. The CEP is a circle, but it is not a perfect shape. The points that form this circle are determined by simulation, so these points form a theoretical circle, which can be visualized based on the best fit. The estimated panel center is then the centroid of these points. In some embodiments, the CEP suggests that a predetermined percentage (e.g., 2%) of deviation from the actual center is acceptable. Perpendicularity is a condition that ensures that all candidate points used to form the CEP are approximately perpendicular to the longest visible Hough line in situations where glare is corrected and / or symmetry is used, or pixel-based energy lines are created.

[0089] Some embodiments utilize the Shi-Tomasi and / or Harris methods to estimate the centers and / or corners of the solar panel.

[0090] The above steps for center and / or corner estimation may be utilized for fixed panels, picked panels, and / or previously placed panels.

[0091] The next step is to place the picked panel. Panel placement energy and proximity result in a balance between the picked panel and the previously placed panel. In some embodiments, images are captured in perspective. In some embodiments, a dynamic comparison is made between the fixed panel and the moving panel (e.g., when the picked panel is moving through the air). Height differences between the picked panel and the placed panel (reference) create potential pitfalls. In some embodiments, placement is complete when six degrees of freedom (three rotations and three translations) are aligned for the two panels.

[0092] FIG. 52 illustrates an example process 5200 for solar panel placement, according to some embodiments. Some embodiments calculate the centers of the fixed and floating panels (5202). Some embodiments draw a line between the two centers as a function of time (5204). Some embodiments establish Euler angles and radius vectors at different times (5206). Some embodiments move the floating panel (5208) (or have a robotic arm move the floating panel) so that the values ​​of the second through sixth degrees of freedom approach zero. Some embodiments use computer vision to verify the coplanarity of the points or Hough lines of both panels (5210). Some embodiments slide the floating panel (5212) (or have a robotic arm slide the floating panel) so that the distance between the two centers is approximately equal to the sum of a safety offset distance (which may be user-entered or predetermined) and the width of each panel (which may be pre-calculated or user-entered).

[0093] FIG. 53 is a schematic diagram of an example solar panel placement 5300, according to some embodiments. When a panel 5302 is picked, it is suspended in the air with six degrees of freedom (three rotations and three translations). The panel can be tilted, angled, and / or curved relative to the placed panel 5306 (sometimes referred to as the fixed panel). Regardless of the position and / or orientation of the picked panel relative to the placed panel, the picked panel has six degrees of separation, meaning that it can vary by three distances and / or three angles. One goal is to align the panels, so the upper robotic arm must place the picked panel so that the longest edge of the picked panel and the longest edge of the placed panel are aligned or coplanar with each other. Based on the placement orientation, the upper robotic arm may place the picked panel so that the shortest edge of the picked panel and the shortest edge of the placed panel are aligned or coplanar with each other. Thus, the robotic arm may initially position the floating panel at an angle (e.g., at an angle relative to the plane on which the panel will ultimately be placed). After the edges are aligned, the robotic arm may tilt the floating panel to make it parallel to the ground and / or the placed panel. Some embodiments may utilize the algorithm described above with reference to FIG. 52.

[0094] Referring to FIG. 53 , a panel center 5308 is estimated for the picked panel 5302. A panel center 5310 of the placed panel 5306 may be predetermined or similarly estimated (e.g., similar to how the center of a floating panel is estimated in a pick-and-place scenario). FIG. 53 corresponds to a top view of these two panels. The dynamic positional degrees of freedom (DOF) of the picked panel relative to the placed panel are denoted by [X, Y, Z, θ, φ, ψ]. θ and φ are shown relative to the reference axes of the fixed panel 5306. Because this is a top view, ψ is not shown. The “distance dapproach,1” corresponds to the distance from the top right corner 5312 of the picked panel 5302 to the top left corner 5314 of the fixed panel 5306. The “distance dapproach,2” corresponds to the distance from the bottom right corner 5316 of the picked panel 5302 to the bottom left corner 5318 of the fixed panel 5306. One goal is to move the floating panels to dashed position 5304 (called the final position with DOF [X,0,0,0,0,0]). The panels are ready for alignment when the DOF vectors take the form [Xsafe,0,0,0,0,0,0]. This occurs when dapproach,1 and dapproach,2 reach Xsafe. The Xsafe distance may take into account the thickness of the clamps. The other degrees of freedom are zero. The projection and reference axes are aligned. This example is provided for illustrative purposes. Other initial configurations and procedures for placement and / or alignment are also possible. For example, vectors other than the Y or Z axis may approach zero and / or the solar panel t may be moved in other directions, with the goal of aligning all but one of the degrees of freedom when the panels are safe to align.

[0095] Some embodiments assist operators with moving clamps and / or safely placing solar panels by detecting structures such as, for example, clamp position and / or fan gear position. Figure 54 shows an example solar panel installation infrastructure 5400 having a fan gear 5402 attached to a torque tube 5404, according to some embodiments. Figure 55A is a plan view showing an example solar panel installation infrastructure 5524 having a torque tube 5526 and clamps or fan gear 5528, according to some embodiments.

[0096] FIG. 55B illustrates an example process 5500 for clamp and fan gear detection according to some embodiments. Some embodiments acquire an image (an example of which is shown in FIG. 55A) in a perspective or top view (5502), convert the image to grayscale (5504), denoise the grayscale image and apply a bilateral filter (5506), erode and threshold the image (5508), apply blurring and Canny edge detection (5510), dilate edges (5512), detect external contours exceeding a certain length (e.g., a length based on the shape of the panel relative to at least 50% of the width and length) (5514), and / or draw a convex hull (5516). Some embodiments test for convex hulls in orthogonal directions (5518). Multiple convex hulls are generated, and if orthogonality is observed, some embodiments mark a feature detection (5520). Some embodiments suggest the detection of an object, such as a clamp or fan gear (5522).

[0097] FIG. 56 illustrates an example application 5600 of the centroid method for estimating the center of a solar panel, according to some embodiments. An image of the panel is acquired and converted to pixel coordinates, and the pixel density is calculated from the X and Y perspectives. Given these pixel densities, the center or relative center can be estimated. Because the centroid always points to the center of the image, this method selects a point that is apparently the center of the image but may not be the center of the panel. This is because the image capture may not be symmetrical, for example, if the panel is bent. In some cases, the pose may be different, and the system may obtain a perspective view as shown in FIG. 56. That is, the near edge of the panel appears longer and the far edge of the panel appears shorter. This suggests a perspective view (and therefore the image may be bent on a torque tube). In such an example, the estimated center (coordinates (1999.5, 1125.5) in this example) has a location offset from the actual geometric center. While this results in an inaccurate estimate, the generated offset is known. When the system detects perspective effects, it knows that the angle is curved, meaning the center estimate cannot be the actual center. Or, the panel's geometric center and image center are not aligned. However, for accurate picking and placement, the system needs to know the panel's geometric center. Therefore, if the panel's angle is curved, distortion correction and pose correction are applied. Additionally, the background is suppressed. These images and poses can be oriented in multiple directions in six degrees of freedom (three rotations and three translations), so either the background must be suppressed to prevent it from contributing to the center of gravity, or the system must compensate for the background and perform adjustments independent of the background.

[0098] The above-described algorithms for picking, placing, and detecting structures, such as clamps and fan gears, are independent of each other. Various embodiments use one or more algorithms or a combination thereof for solar panel installation. Some embodiments combine one or more of these methods with traditional methods for installing solar panels.

[0099] The techniques described herein have several advantages over conventional techniques. For example, systems and methods according to the techniques described herein utilize a non-synthetic image processing scheme to obtain pick-and-place recommendations, thus providing scalability to any terrain and condition. Closed-loop feedback aids in automatic correction and improvement of the estimation process. The simulation procedure is not condition-dependent. Some embodiments consider optical and geometric characteristics to accommodate changes in the environment and manufacturer specifications. Some embodiments utilize pixel energy and pixel color density components during the estimation process.

[0100] Examples of methods for installing autonomous photovoltaic power plants FIG. 57 is a flow diagram of an example method 5700 for autonomous solar power plant installation, according to some embodiments. The method includes acquiring 5702 a first image of a solar panel (e.g., solar panel 4414 held in cradle 4410) during ongoing solar power plant installation. Examples of systems and methods for image acquisition and / or image processing, according to some embodiments, are described above with reference to FIG. 40. In some embodiments, the solar panel includes a reflective material that causes sunlight to be reflected by the solar panel, and the image of the solar panel may include a dark white spot due to such reflection. In some embodiments, the first image includes a view of the solar panel and a background, an example of which is described above with reference to FIG. 51A. In such cases, the centroid method described above with reference to FIG. 56 may not be usable to calculate the center of the solar panel because the centroid method assigns equal importance to all pixels, thus introducing distortions. A simulation approach, such as method 5700, may also be used.

[0101] The method further includes estimating 5704 a plurality of features of the solar panel based on the first image by utilizing distance simulation, geometric correction, and angle adjustment. In some embodiments, the plurality of features include interior points, centers, corners, and / or edges of the solar panel. In some embodiments, an initial set of features is estimated, and other features may be calculated based on the estimation.

[0102] In some embodiments, estimating the plurality of characteristics of the solar panel further includes, in accordance with determining that a panel identifier, such as a barcode label, is visible in the first image, detecting a position of the panel identifier and estimating the plurality of characteristics of the solar panel using the position. For example, the panel identifier may be located on a rim of the solar panel at one of the edges of the solar panel. In some examples, the barcode-like panel identifier may be detected based on a cluster of Hough lines. A high density of vertical Hough lines in a certain area indicates the presence of a barcode.

[0103] In some embodiments, estimating the plurality of characteristics of the solar panel includes identifying and drawing, based on the first image, a longest visible horizontal line and a longest visible vertical line that indicate an edge of the solar panel (e.g., drawing a Hough line on the detected edge boundary); calculating, for each point in the first image that is located inside a solar panel in the first image, a horizontal distance and a vertical distance from each point to the longest visible horizontal line and the longest visible vertical line, respectively; identifying a subset of points based on the calculated horizontal distances and vertical distances, where for the subset of points, (i) the horizontal distance is greater than a first predetermined percentage (e.g., 45% or a maximum) of the width of the solar panel, and (ii) the vertical distance is greater than a second predetermined percentage (e.g., 45% or a maximum) of the height of the solar panel; and calculating a horizontal intersection angle and a vertical intersection angle, respectively, from each point in the subset of points to the longest visible horizontal line and the longest visible vertical line. correcting the calculated horizontal and vertical distances based on the calculated horizontal and vertical intersection angles to obtain corrected horizontal and vertical distances (the range is the difference between the orthogonality and the calculated angle); identifying a set of candidate points based on the corrected horizontal and vertical distances, where (i) the horizontal distance is greater than a first predetermined percentage (e.g., 48% or a maximum value) of the width of the solar panel and (ii) the vertical distance is greater than a second predetermined percentage (e.g., 48% or a maximum value) of the height of the solar panel; calculating a centroid of the set of candidate points to obtain the center of the solar panel; estimating candidate corners of the solar panel based on the centroid and the set of candidate points using Pythagoras' theorem; and correcting the distances of the candidate corners based on the calculated horizontal and vertical intersection angles to obtain the final corners of the solar panel. Examples of these steps are described above with reference to FIG. 51A.

[0104] In some embodiments, the identification and rendering are performed using a Hough transform. The longest visible horizontal line and the longest visible vertical line are Hough lines. Hough lines are good indicators for geometric feature detection. Both standard and probabilistic Hough methods may be used for estimation. The parameters of these methods can be modified for specific applications. Hough lines are used to detect straight lines in an image. A line can be mathematically defined by (i) a slope and offset, (ii) an angle and radius, and (iii) endpoints (e.g., the start and end coordinates of a line segment). In the standard formulation of the Hough transform, the angle and radius vectors are estimated as parameters in a polar coordinate system, which generates two degrees of freedom that can be modified. In the probabilistic formulation of the Hough transform, the coordinates of the endpoints are estimated as parameters in a Cartesian coordinate system, which generates four degrees of freedom that can be modified. Other techniques for detecting straight lines can also be used as alternatives to the Hough transform technique.

[0105] In some embodiments, the method further includes determining whether a point is located inside the solar panel by determining whether a ray of light emanating from the point intersects the longest visible horizontal line and the longest visible vertical line at an odd point.

[0106] In some embodiments, the method further includes, prior to estimating the multiple solar panel regions / characteristics, extracting the relative position of the sun in the sky using, for example, the sun's physical properties, the sun's elevation angle and / or azimuth angle, estimating glare based on the estimated sun position, and applying pseudocolor to the glare by utilizing noise reduction and image correction algorithms. Glare is characterized by a high density of white pixels in the image, which constitutes noise in this particular application. Therefore, noise reduction means removing these clusters of white pixels using the panel color. Some embodiments utilize filtering techniques for this purpose, such as Gaussian filtering, wavelet filtering, Kalman filtering, or pseudocolor filtering. Figure 51B is a schematic diagram of an example method 5110 for Hough line estimation with glare, according to some embodiments. The image boundary is indicated by label 5112, and the panel boundary is indicated by label 5114. The edge between corner A and corner D (line AD) indicates glare 5116. If Hough line detection is impossible or inaccurate, as in this example, an energy balance line BC is drawn as an equivalent to the Hough line. The energy balance line is the locus of all pixels that share approximately the same energy value in a particular direction. These steps are then performed if the solar elevation and azimuth angles on a particular day suggest glare formation. This energy balance line (sometimes called an energy line) is the locus of points along the Hough line or panel width / length direction, where pixels share approximately equal energy values. Some embodiments monitor for two conditions: (i) the pixel energy values ​​must be relatively close to each other (e.g., within 2%), and (ii) the pixel centers must be directionally collinear. The start and end points of the energy line are crucial. Some embodiments monitor for a drop in energy level between adjacent pixels (e.g., a gradual change, such as a change of more than 10%).If such a drop occurs and the pixels are collinear, it is safe to conclude that glare is no longer present. When supplementing visible Hough lines with energy lines, the intensity of pixels captured through energy, such energy drop, and / or collinearity may be used to establish the lines.

[0107] In some embodiments, the method further includes removing solar glare on the solar panel by utilizing image masking and segmentation, or filtering or image correction techniques, prior to estimating the plurality of regions / features of the solar panel. Examples of methods for performing masking and / or segmentation are described above with reference to Figures 36, 38, 43, and 50.

[0108] 57, the method further includes generating 5706 a first set of control signals for operating the first robotic controller to pick the solar panel based on the estimated characteristics. In some embodiments, the first set of control signals further causes the first robotic controller to move the picked solar panel toward the fixed solar panel.

[0109] Additionally, the method further includes acquiring 5708 a second image of the solar panel when the solar panel is in a perspective view, an example of which is described above with reference to FIG.

[0110] The method further includes detecting the location of the solar panel based on the second image by determining whether the solar panel is coplanar with the fixed solar panel and at a predetermined offset from the fixed solar panel (5710). In some embodiments, the solar panel and the fixed solar panel have substantially rectangular shapes. In some embodiments, the solar panel and the fixed solar panel have substantially similar shapes. In some embodiments, the picked solar panel and the fixed solar panel have substantially similar shapes along only one side, and the system may utilize symmetry for the calculations and / or estimations described herein. For example, if glare is absent along the longest visible Hough line, scaling by utilizing symmetry is possible because the panel shape is regular. Because the solar panel is rectangular, distance and angle measurements for the invisible side are always known. However, if glare partially or completely spans the longest "visible" Hough line, the length of the longest Hough line is estimated by interpolating the geometry line by drawing an energy line.

[0111] In some embodiments, detecting the solar panel position includes acquiring a third image of the fixed solar panel when the fixed solar panel is in a perspective view. For example, the image is in a perspective view when the near edge appears longer than the far edge. Some embodiments continuously record the images. In some embodiments, detecting the solar panel position further includes identifying two centers including (i) a center of the solar panel when the solar panel is floating based on the second image and (ii) a center of the fixed solar panel based on the third image. In some embodiments, detecting the solar panel position further includes drawing a line between the two centers as a function of time and establishing Euler angles and radius vectors at different times (e.g., each time point). In some embodiments, detecting the solar panel position further includes generating one or more control signals to operate the first robotic controller to move the solar panel so that five degrees of freedom (relative to the fixed solar panel) approach the numerical value 0. In some embodiments, detecting the placement of the solar panels further includes utilizing computer vision (e.g., edge detection, interest point detection, contour mapping, bounding box techniques, etc.) to verify coplanarity of the solar panels and fixed solar panel points and Hough lines. Examples of these steps are described above with reference to Figures 52 and 53.

[0112] The method further includes generating 5712 a second set of control signals for operating a second robotic controller based on the detected placement to align the solar panels with respect to the fixed solar panels. In some embodiments, generating the second set of control signals includes generating control signals for sliding the solar panels so that the distance between the two centers is approximately equal to the sum of a predetermined safety offset distance and the width of each panel when the solar panels are floating. The second robotic controller and the first robotic controller may be different controllers or the same controller. These controllers may control the upper robotic arm and / or the lower robotic arm in various embodiments. In some embodiments, generating the second set of control signals includes generating control signals for securing the solar panels (e.g., by positioning and tightening clamps, etc.).

[0113] In some embodiments, the method further includes detecting a supporting mechanical. This may include acquiring a fourth image of the solar panel in a second perspective view or a top view, which may be a perspective view different from the other perspective views described above with reference to step 5708. In some embodiments, detecting the supporting mechanical includes converting the fourth image to grayscale to obtain a grayscale image, and performing noise reduction and a bilateral filter on the grayscale image to obtain a processed image. In some embodiments, detecting the supporting mechanical includes eroding and thresholding the processed image to obtain a candidate image and drawing a convex hull of the candidate image. In some embodiments, detecting the supporting mechanical includes determining an outer contour exceeding a predetermined length based on the convex hull and dilating edges of the candidate image. In some embodiments, detecting the supporting mechanical includes performing blurring and Canny edge detection on the candidate image and testing for a convex hull in an orthogonal direction. In some embodiments, detecting the supporting mechanical implement includes detecting the supporting mechanical implement according to a determination that multiple convex hulls are generated and orthogonality is observed. Generally, these steps may include image processing, image preparation, and / or image alignment. Examples of these steps are described above with reference to Figures 54 and 55.

[0114] In some embodiments, the supporting mechanical device comprises a clamp (sometimes referred to as a clamp assembly, e.g., clamp assembly 602 in FIGS. 6A and 6B ) or a fan gear (e.g., fan gear 5402), although other supporting mechanical devices are contemplated, such as a ground support, an electrical box, and electrical connections. In some embodiments, the supporting mechanical device is a clamp, and the method further includes detecting the position of the clamp and assisting an operator with moving the clamp based on the detected position of the solar panel to protect the solar panel from collision with the clamp. For example, a notification / warning is provided to the operator (e.g., via an alarm, etc.). When a clamp is detected, operation is paused. In some embodiments, the supporting mechanical device is a fan gear, and the method further includes detecting the position of the fan gear and assisting an operator with safely positioning the solar panel based on the detected position of the solar panel to protect the solar panel from collision. Examples of these steps are described above with reference to FIGS. 54 and 55 .

[0115] In some embodiments, determining whether the solar panel is coplanar with the fixed solar panel and a predetermined offset from the fixed solar panel includes determining (i) whether the solar panel is coplanar with the fixed solar panel, and (ii) whether the solar panel and the fixed panel are approximately coplanar in the lateral direction.

[0116] FIG. 58 is a flow diagram of another example method 5800 of autonomous solar power installation, according to some embodiments. The method includes acquiring 5802 a first image of a solar panel (e.g., solar panel 4414) in a staging area (e.g., cradle 4410) using a viewpoint camera (e.g., a camera disposed on an upper robot). The solar panel may be approximately perpendicular or normal to the viewpoint camera, the ground, and / or the staging area. The viewpoint camera may be a visible light camera and / or an infrared camera. Example systems and methods for image acquisition and / or image processing, according to some embodiments, are described above with reference to FIG. 40. The method further includes estimating 5804 a plurality of regions / features of the solar panel based on the first image using at least one of distance simulation, geometric correction, and angle adjustment. Examples of these steps, according to some embodiments, are described above with reference to FIG. 51A. The method further includes generating 5806 a first set of control signals for operating a first robotic controller to pick the solar panel. The first set of control signals is based on one or more of the estimated regions / features. The method further includes acquiring 5808 a second image of the solar panel when it has been picked and is in a perspective orientation relative to the viewpoint camera. The method further includes detecting 5810 an orientation in space of the picked solar panel based on the second image. The method further includes generating 5812 a second set of control signals based on the detected orientation to move the picked solar panel to an installation position. At the installation position, the picked solar panel is aligned with the previously installed solar panel by determining whether the picked solar panel is flush with the previously installed solar panel and at a predetermined offset from the previously installed solar panel. Examples of these steps are described above with reference to FIGS. 52, 53, and 57.

[0117] Some embodiments of the present invention have been described above with the aid of functional building blocks illustrating the implementation of specific functions and relationships thereof. The boundaries of these functional building blocks are arbitrarily defined herein for the convenience of description. Alternative boundaries can be defined so long as the specific functions and relationships thereof are appropriately implemented.

[0118] It will be apparent to those skilled in the art that various modifications and variations can be made to the system for installing solar panels of the present invention without departing from the spirit or scope of the present invention. Therefore, it is intended that the present invention encompasses the modifications and variations of the present invention within the scope of the appended claims and their equivalents. It should be understood that the phrases or terms used herein are for the purpose of description and not limitation, and should therefore be interpreted by those skilled in the art in light of the teachings and guidance.

[0119] The breadth and scope of the present invention should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents. [Explanation of symbols]

[0120] 100 Arm Assembly Tools, Robot Arms 102 frames 102-A Truss 104 Wearable Devices 106 Linear guide assembly 108 Clamping tool, linear movable clamping tool 108-A Engagement member 108-B Proximity Sensor 110 Force torque transducer, force torque actuator 112 Junction box, controller 120 solar panels 602 Clamp Assembly 604 Torque tube 604 Installation structure 606 Roller 608 Spring Mechanism 610 Receptor Channel 802 Optical Sensor 802-A position 802-B position 804 Orientation Assembly 903 Assembly Mobile Robot 905 Storage Container 907 Land Vehicles 908 Upper Section 909 Lower Section 911 Second Robot Arm 1005 Module Vehicle 3500 Image Processing Pipeline 3502 Module 3504 Module 3506 Module 3508 Module 3510 Module 3512 Module 3514 Module 3516 Module 3518 Module 3520 Corrected acquired image 3522 Solar Panels 3524-2 Object 3524-4 Object 3526 Output 3528 Panel corner detection 3530-2 Corner 3530-4 Corner 3532-4 Horizontal line 3532-8 Horizontal line 3532-2 vertical line 3532-6 Vertical Line 3600 Example images 3602, 3606, and 3610 of a road under various lighting conditions and image segmentation masks 3604, 3608, and 3612 3602 Road Images 3604 Segmentation Mask 3606 Road Images 3608 Segmentation Mask 3610 Road Images 3612 Segmentation Mask 3700 Captured Images 3702 Solar Panels 3704 Torque tube 3706 annotated images 3708 Black Background 3710 Solar Panels 3712 Torque tube 3714 Predicting a single example using a trained model 3716 objects 3802 bottles 3804 Cube 3806 cups 3808 Rectangle 3810 rectangle 3812 rectangle 3814-2 Rectangle 3814-4 Rectangle 3816 Object Localization 3818 Semantic Segmentation 3820 Label 3822 Label 3824 Label 3826 Instance Segmentation 3828 Label 3830 Label 3834 Label 3836 Label 3900 Segmentation 3902 Panel Instance 3904 Panel Instance 3906 Panel Instance 3908 Panel Instance 3910 Panel Instance 4000 Image Handling System 4002 Camera 4004 Camera 4006 Camera 4008 Image Sensor 4010 Image Sensor 4012 Image Sensor 4014 GPU 4016 GPU 4018 Robot Controller 4100 Trailer System 4102 Coarse Resolution Camera 4200 Histogram 4202 Histogram 4300 Example of approximate solar panel locations (e.g., locations 4304, 4306, 4308, and 4310) using the B Mask R-CNN model. 4302 B Example of approximate solar panel locations (e.g., locations 4304, 4306, 4308, and 4310) using the Mask R-CNN model. 4304 Approximate location of solar panels 4306 Approximate location of solar panels 4308 Approximate location of solar panels 4310 Approximate location of solar panels 4400 System 4402 Battery Enclosure 4404 Main Enclosure 4406 Upper Robot End Effector (EOAT) 4408 Downward Robot EOAT 4410 Cradle 4412 Trailer 4414 Solar Panels 4500 Structures 4502 Vision System 4600 Module Picking 4602 Vision System 4700 System 4702 position 4704 position 4800 Laser Line Generator System 4802 Laser Line Generator System 4900 Vision System 4902 Socket wrench 5102 images 5104 Panel 5106 longest visible horizon 5108 Longest visible vertical line 5112 Label 5114 Label 5116 Glare 5300 placement 5302 Panels picked 5304 Dashed line position 5306 Placed Panels, Fixed Panels 5308 Panel center 5310 Panel center 5312 top right corner 5314 top left corner 5316 bottom right corner 5318 bottom left corner 5400 Solar Panel Installation Infrastructure 5402 Fan Gear 5404 Torque tube 5524 Solar Panel Installation Infrastructure 5526 Torque Tube 5528 Fan Gear 5600 An example application of the centroid method to estimate the center of a solar panel

Claims

1. 1. A method for autonomous solar power installation, comprising: acquiring a first image of a solar panel during an ongoing solar installation; estimating a plurality of characteristics of the solar panel based on the first image using distance simulation, geometric correction, and angle adjustment; generating a first set of control signals for operating a first robotic controller to pick the solar panel based on the estimated characteristics; acquiring a second image of the solar panel when the solar panel is in a perspective view; detecting the location of the solar panel based on the second image by determining whether the solar panel is flush with and at a predetermined offset from a fixed solar panel; generating a second set of control signals for operating a second robotic controller to align the solar panel with the fixed solar panel based on the detected placement; A method comprising:

2. The method of claim 1 , wherein the plurality of features includes a center and a plurality of corners of the solar panel.

3. The step of estimating the plurality of characteristics of the solar panel includes: In response to a determination that the bar code label is visible within the first image, detecting the position of the bar code label; estimating the plurality of characteristics of the solar panel using the location; 10. The method of claim 1, further comprising:

4. The step of estimating the plurality of characteristics of the solar panel includes: Identifying and drawing a longest visible horizontal line and a longest visible vertical line representing an edge of the solar panel based on the first image; For each point in the first image that is located inside the solar panel, calculating a horizontal distance and a vertical distance from each point to the longest visible horizontal line and the longest visible vertical line, respectively; identifying a subset of points based on the calculated horizontal and vertical distances, where for the subset of points (i) the horizontal distance is greater than a first predetermined percentage of the width of the solar panel, and (ii) the vertical distance is greater than a second predetermined percentage of the height of the solar panel; calculating horizontal and vertical intersection angles from each point in the subset of points with the longest visible horizontal and longest visible vertical lines, respectively; correcting the calculated horizontal distance and vertical distance based on the calculated horizontal crossing angle and vertical crossing angle to obtain corrected horizontal distance and vertical distance; identifying a set of candidate points based on the corrected horizontal and vertical distances, wherein the set of candidate points have (i) a horizontal distance greater than a first predetermined percentage of a width of the solar panel, and (ii) a vertical distance greater than a second predetermined percentage of a height of the solar panel; obtaining the center of the solar panel by calculating the centroid of the set of candidate points; estimating candidate corners of the solar panel based on the centroid and the candidate point set using Pythagoras' theorem; correcting the distance of the candidate corner based on the calculated horizontal intersection angle and vertical intersection angle to obtain a final corner of the solar panel; 2. The method of claim 1, comprising:

5. The method of claim 4 , wherein the steps of identifying and rendering are performed using a Hough transform.

6. 5. The method of claim 4, further comprising determining whether a point is located inside the solar panel by determining whether a ray of light emanating from the point intersects the longest visible horizontal line and the longest visible vertical line at an odd point.

7. The step of detecting the arrangement of the solar panels includes: acquiring a third image of the fixed solar panel when the fixed solar panel is in a perspective view; Identifying two centers including (i) a center of the solar panel when the solar panel is floating based on the second image, and (ii) a center of the fixed solar panel based on the third image; drawing a line between the two centers as a function of time; establishing Euler angles and radius vectors at each time point; generating one or more control signals to move the solar panel by operating the first robotic controller so that five degrees of freedom approach a value of zero; utilizing computer vision to verify coplanarity of points and Hough lines of the solar panel and the fixed solar panel; 7. The method of any one of claims 1 to 6, comprising:

8. 8. The method of claim 7, wherein generating one or more control signals comprises generating a control signal to slide the solar panels so that the distance between the two centers is approximately equal to the sum of a predetermined safety offset distance and a width of each panel when the solar panels are floating.

9. acquiring a fourth image of the solar panel from a second perspective view or a top view; obtaining a grayscale image by converting the fourth image to grayscale; performing noise reduction and applying a bilateral filter to the grayscale image to obtain a processed image; obtaining a candidate image by eroding and thresholding the processed image; drawing a convex hull of the candidate image; determining an outer contour exceeding a predetermined length based on the convex hull; dilating the edges of the candidate image; blurring the candidate image and applying Canny edge detection; testing for a convex hull in orthogonal directions; generating the plurality of convex hulls and detecting the supporting mechanical implements in accordance with a determination that orthogonality is observed; 9. The method of claim 1, further comprising:

10. The method of claim 9 , wherein the supporting mechanical device comprises a clamp or a fan gear.

11. The supporting mechanical device is a clamp, and the method comprises: detecting the position of the clamp; assisting an operator in moving the clamp based on the detected position of the solar panel to protect the solar panel from collision with the clamp; 10. The method of claim 9, further comprising:

12. the support device is a fan gear, and the method comprises: detecting a position of a fan gear; assisting an operator in safely positioning the solar panels based on the detected solar panel positions to protect the solar panels from collisions; 10. The method of claim 9, further comprising:

13. 13. The method of claim 1, wherein the solar panel and the fixed solar panel have a substantially rectangular shape.

14. 14. The method of claim 1, wherein the solar panel and the fixed solar panel have substantially similar shapes.

15. 15. The method of claim 1, wherein the solar panel comprises a reflector that causes reflection of sunlight by the solar panel, and the first image or the second image includes a dark white spot.

16. 16. The method of claim 1, wherein the first image includes a view of the solar panel and a background.

17. Prior to the step of estimating the plurality of regions / features of the solar panel, extracting the relative position of the sun in the sky using the physical properties of the sun, the elevation angle of the sun, and the azimuth angle of the sun; estimating glare based on the estimated sun position; applying pseudocolor to said glare by utilizing high fidelity noise reduction and image correction algorithms; 17. The method of any one of claims 1 to 16, further comprising:

18. Prior to the step of estimating the plurality of regions / features of the solar panel, removing solar glare on said solar panel by utilizing image masking and segmentation, or filtering, or image enhancement techniques.

18. The method of any one of claims 1 to 17, further comprising:

19. 19. The method of claim 1, wherein the first set of control signals causes the first robotic controller to move the picked solar panel toward the fixed solar panel.

20. determining whether the solar panel is flush with the fixed solar panel and at a predetermined offset from the fixed solar panel; (i) determining whether the solar panel is coplanar with the fixed solar panel; and (ii) determining whether the solar panel and the fixed panel are substantially coplanar laterally.

20. The method of any one of claims 1 to 19, comprising:

21. 1. A method for autonomous solar power installation, comprising: acquiring a first image of the solar panels within the staging area using a viewpoint camera; estimating a plurality of regions / features of the solar panel based on the first image using at least one of distance simulation, geometric correction, and angle adjustment; generating a first set of control signals for operating a first robotic controller to pick the solar panel, the first set of control signals being based on one or more of the estimated regions / features; acquiring a second image of the solar panel when the solar panel is picked and in a perspective orientation relative to the viewpoint camera; Detecting a spatial orientation of the picked solar panel based on the second image; generating a second set of control signals based on the detected orientation to operate a second robotic controller to move the picked solar panel to an installation position; Including, At the installation location, the picked solar panel is aligned with the previously installed solar panel by determining whether the picked solar panel is flush with and at a predetermined offset from the previously installed solar panel.

Images