Autonomous solar installation using artificial intelligence

Abstract

Systems and methods for mounting a solar panel are provided. The method obtains an image of the solar panel during ongoing solar installation and estimates a feature of the solar panel based on the first image using distance simulation, geometric correction, and / or angular adjustment. The method also generates a control signal based on the estimated feature for operating the robotic controller to pick up the solar panel. The method also obtains a second image of the solar panel while the solar panel is in the perspective view, and detects placement of the solar panel based on the image by determining whether the solar panel is coplanar with a fixed solar panel and has a predetermined offset. Based on the detected placement, a control signal is generated for operating a second robotic controller to align the solar panel with the fixed solar panel.

Description

[0001] Related application data

[0002] This application is based on U.S. Provisional Application No. 63 / 397,125 filed on August 11, 2022 and claims priority under 35 U.S.C. §119, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to solar panel handling systems, and more particularly to systems and methods for mounting solar panels on mounting structures. Background Art

[0004] In the following discussion, reference is made to certain structures and / or methods. However, the following references should not be construed as an admission that these structures and / or methods constitute prior art. Applicants expressly reserve the right to demonstrate that these structures and / or methods do not conform to prior art against the present invention.

[0005] The installation of a photovoltaic array typically involves securing the solar panels to a mounting structure. The bottom support provides attachment points for each solar panel, as well as assisting in the layout of the electrical system and, if applicable, any mechanical components. Due to the fragile nature and large size of the solar panels, the process of securing the solar panels to the mounting structure poses unique challenges. For example, in many cases, the solar panels of a photovoltaic array are mounted on a rotatable structure that allows the solar panels to rotate about an axis so that the array can track the sun. In this case, it is difficult to ensure that all solar panels in the array are coplanar and leveled relative to the axis of the rotatable structure. In addition, the installation cost of a photovoltaic array may account for a considerable portion of the total construction cost of the photovoltaic array. Therefore, a more efficient and reliable solar panel processing system is needed to install solar panels in a photovoltaic array. When the environment is ideal, conventional computer vision techniques can be used. However, glare, overexposure, or underexposure may negatively affect target detection algorithms.

[0006] The use of solar panels is particularly suited to tropical and / or equatorial installations, which are ideal for sunlight availability but have difficult working environments. Therefore, autonomous robotic solutions are needed to install solar panels in such environments. But doing so also means that improvements in robotic installation are needed, such as those related to one or more of the following: guidance and navigation for automated panel picking from shipping or storage containers (e.g., crates); and proper placement on mounting hardware (e.g., torque tubes); and alignment with any previously placed panels; and detection (and avoidance) of possible mechanical structures, jigs, and fixtures, such as clamps and fan gears. Summary of the invention

[0007] Accordingly, the present invention is directed to a solar panel processing system that substantially obviates one or more problems due to limitations and disadvantages of the related art.

[0008] The solar panel handling system disclosed herein facilitates installation of solar panels of a photovoltaic array on a pre-existing mounting structure (e.g., a torque tube). By combining tools for handling solar panels with components that enable the solar panels to mate with the solar panel support structure, installation of solar panels can be made more efficient and reliable. Some embodiments use machine learning techniques to overcome environmental inconsistencies. The system can learn from examples with glare and lighting issues and can generalize to new data during inference.

[0009] Additional features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or can be learned by practicing the present invention. The purpose and other advantages of the present invention will be realized and obtained through the structures particularly pointed out in the written description and its claims and the drawings.

[0010] To achieve these and other advantages and in accordance with the purposes of the present invention, as embodied and broadly described, a system for mounting a solar panel may include an end-of-arm assembly tool including a frame and a suction cup coupled to the frame, and a linear guide assembly coupled to the end-of-arm assembly tool, wherein the linear guide assembly includes: a linearly movable clamping tool, the clamping tool including an engagement member, the engagement member configured to engage a clamp assembly slidably coupled to a mounting structure; a force torque sensor configured to move the clamping tool along the mounting structure; and a junction box, which is coupled to the frame and includes a controller configured to control the force torque sensor and the suction cup, and a power source.

[0011] On the other hand, a method for installing a solar panel may include: engaging an end-of-arm assembly tool with a solar panel, the end-of-arm assembly tool including a frame and a suction cup coupled to the frame; positioning the solar panel relative to a mounting structure having a clamp assembly slidably coupled thereto; engaging a linear guide assembly coupled to the end-of-arm assembly tool with a clamp assembly, the linear guide assembly including a linearly movable clamping tool including an engaging member configured to engage the clamp assembly, and a torque sensor configured to move the clamping tool along the mounting structure; and actuating the torque sensor to move the clamp assembly along the mounting structure so as to engage with a side of the solar panel, thereby fixing the solar panel relative to the mounting structure.

[0012] In another aspect, a method of installing solar panels may include using a machine learning algorithm to automatically detect the center and corners of solar panels (of varying sizes) to guide a robot to accurately pick up and place the solar panels. In some embodiments, the method includes isolating the center and corners of the solar panel and placing the solar panel relative to a mount (e.g., a torque tube) and a previously placed panel. In some embodiments, the method includes detecting auxiliary equipment or structures, such as clamps and / or fan gears. In some embodiments, the techniques described herein do not use synthetic images and are therefore simulation-based methods that employ a predictor-corrector scheme with possible simulations.

[0013] In another aspect, a method of installing a solar panel may include obtaining a first image of the solar panel during an ongoing solar installation. The method also includes estimating a plurality of features of the solar panel based on the first image using distance simulation, geometric correction, and angle adjustment, and generating a first set of control signals based on the estimated plurality of features for operating a first robotic controller to pick up the solar panel. The method also includes obtaining a second image of the solar panel when the solar panel is in a perspective view, and detecting the position of the solar panel based on the second image by determining whether the solar panel is coplanar with a fixed solar panel (i.e., an already installed solar panel) and has a predetermined offset. The method also includes generating a second set of control signals based on the detected placement for operating a second robotic controller to align the solar panel with the fixed solar panel.

[0014] On the other hand, a method for installing a solar panel may include obtaining a first image of a solar panel in a staging area using an observation camera. The method also includes estimating multiple regions / features of the solar panel based on the first image using at least one of distance simulation, geometric correction, and angle adjustment. The method also includes generating a first set of control signals to operate a first robot controller to pick up the solar panel, wherein the first set of control signals is based on one or more of the estimated multiple regions / features. The method also includes obtaining a second image of the solar panel when the solar panel is picked up and is in a perspective orientation relative to the observation camera, and detecting the orientation of the picked up solar panel in space based on the second image. The method also includes generating a second set of control signals based on the detected orientation for moving the picked up solar panel to an installation position. The installation position aligns the picked up solar panel with a previously installed solar panel. Move the floating / picked up panel into position to align with the fixed panel on the torque tube. "Moving into position" means that the floating panel is aligned with the fixed (reference) panel in terms of direction, distance, and orientation.

[0015] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated herein and constitute a part of the specification, illustrate the present invention and, together with the description, further serve to explain the principles of the present invention and enable those skilled in the relevant art to make and use the present invention. The exemplary embodiments are best understood by the following detailed description when read in conjunction with the accompanying drawings. It is emphasized that, according to convention, the features of the drawings are not drawn to scale. On the contrary, the sizes of the various features are arbitrarily enlarged or reduced for clarity.

[0017] The accompanying drawings include the following diagrams:

[0018] Figure 1 A perspective view of a solar panel handling system and a container for solar panels is shown according to an embodiment of the present disclosure.

[0019] Figure 2A-2C It is shown accordingly Figure 1 Top view, front view and side view of a solar panel handling system and container for solar panels.

[0020] Figure 3A Accordingly, top, front and side views of a solar panel handling system coupled to a single solar panel according to an embodiment of the present disclosure are shown.

[0021] Figure 4A and 4B A perspective view of a solar panel handling system is shown according to an embodiment of the present disclosure.

[0022] Figure 5A and 5B A top view and a front view of a solar panel handling system according to an embodiment of the present disclosure are shown accordingly.

[0023] Figure 5C A side view of a solar panel handling system is shown in a retracted position with a gripping tool according to an embodiment of the present disclosure.

[0024] Figure 5D A side view of a gripping tool in an extended or advanced position is shown according to an embodiment of the present disclosure.

[0025] Fig. 6A and 6B A perspective view of a clamping tool of a solar panel handling system engaged with a clamp assembly coupled to a mounting structure is shown in accordance with an embodiment of the present disclosure.

[0026] Fig. 7AA top view of a clamping tool of a solar panel handling system engaged with a clamp assembly coupled to a mounting structure is shown in accordance with an embodiment of the present disclosure.

[0027] Figure 7B A front view of a clamping tool of a solar panel handling system engaged with a clamp assembly coupled to a mounting structure is shown in accordance with an embodiment of the present disclosure.

[0028] Figure 7C A side view of a clamping tool of a solar panel handling system engaged with a clamp assembly coupled to a mounting structure is shown in accordance with an embodiment of the present disclosure.

[0029] Fig.7D A rear view of a clamping tool of a solar panel handling system engaged with a clamp assembly coupled to a mounting structure is shown in accordance with an embodiment of the present disclosure.

[0030] Figure 8 A solar panel handling system during the process of installing a solar panel according to an embodiment of the present disclosure is schematically illustrated in a top view.

[0031] Fig. 9 A solar panel handling system is illustrated that includes an assembly tool coupled to an assembly mobile robot using a robotic arm.

[0032] Fig.10 A solar panel handling system with two robotic arms is illustrated, wherein two assembly tools are coupled to an assembly mobile robot using respective robotic arms.

[0033] Figures 11A to 11C Illustrated is a process for installing solar panels.

[0034] Fig. 12A and 12B An arrangement for a mobile robotic system including two modular vehicles and a ground vehicle having two robotic arms is illustrated.

[0035] Fig.13 The installation achieved using computer vision registration is schematically illustrated.

[0036] Fig.14 An arrangement is schematically illustrated in which a modular vehicle is replaced with a new modular vehicle having additional solar panels for supplementation.

[0037] Figure 15-34 Detailed illustrations of example configurations of systems for mounting solar panels according to embodiments of the present disclosure are provided.

[0038] Fig.35A A block diagram of an exemplary image processing pipeline is shown in accordance with some embodiments.

[0039] Fig.35B An exemplary rectified acquired image is shown in accordance with some embodiments.

[0040] Fig.35C A method for Fig.35B Example output of a neural network image segmentation for obtaining a rectified image is shown in FIG.

[0041] Fig.35D Exemplary panel corner detection is shown in accordance with some embodiments.

[0042] Fig.36 Examples of road images under different lighting conditions and segmentation masks of the images are shown in accordance with some embodiments.

[0043] Fig.37A An example of a captured image including a solar panel and a torque tube is shown in accordance with some embodiments.

[0044] Fig.37B According to some embodiments Fig.37A An example of annotated images of captured images shown in .

[0045] Fig.38A An example of image classification is shown.

[0046] Fig.38B Shows Fig.38A Example of object localization for the image shown in .

[0047] Fig.38C An example of semantic segmentation in accordance with some embodiments is shown.

[0048] Fig.39 An example of instance segmentation of a solar panel is shown in accordance with some embodiments.

[0049] Fig.40 An exemplary image processing system is shown in accordance with some embodiments.

[0050] FIG. 41 illustrates a trailer system according to some embodiments.

[0051] Fig.42A and 42B Shown are histograms of pose error norms for a neural network with and without coarse position, according to some embodiments.

[0052] Fig.43A and 43B An example of the rough location of solar panels using the B Mask R-CNN model is shown in accordance with some embodiments.

[0053] Fig.44A system for solar panel installation according to some embodiments is shown.

[0054] According to some embodiments, Fig.45A shows a vision system for tracking the position of a trailer, and Fig.45B An enlarged view of the visual system is shown.

[0055] According to some embodiments, Fig.46A shows a vision system for module picking, and Fig.46B An enlarged view of the visual system is shown.

[0056] Fig.47A , 47B and 47C illustrate a system for distance measurement at module angles according to some embodiments.

[0057] Fig.48A A system for laser line generation for detecting tube and fixture positions is shown in accordance with some embodiments.

[0058] According to some embodiments, Fig.48B Shows Fig.48A An enlarged view of the laser line generation system shown in FIG. Fig.48C A view of the laser line generation is shown (the horizontal line detects the fixture, and the vertical line detects the tube).

[0059] According to some embodiments, Fig.49A A vision system 4900 for estimating tube and fixture positions is shown, and Fig.49B An enlarged view of the visual system is shown.

[0060] Fig.50A A flow chart of a method for autonomous solar installation is shown according to some embodiments.

[0061] Fig.50B A flow chart illustrating a method of training a neural network for autonomous solar installations according to some embodiments.

[0062] Fig.51A is a schematic diagram of an exemplary method for estimating the center and corners of a solar panel, according to some embodiments.

[0063] Fig.51B is a schematic diagram of an exemplary method for Hough line estimation when glare (or other optical phenomena) is present, according to some embodiments.

[0064] Fig.52 An exemplary process for placement of solar panels is shown in accordance with some embodiments.

[0065] Fig.53is a schematic diagram of an exemplary placement of solar panels according to some embodiments.

[0066] Fig.54 An exemplary solar panel mounting infrastructure is shown having a fan gear attached to a torque tube 5404 in accordance with some embodiments.

[0067] Fig.55A A top view of an exemplary solar panel mounting infrastructure with a torque tube and clamp or fan gear is shown in accordance with some embodiments.

[0068] Fig.55B An exemplary process for clamp and fan gear detection is shown in accordance with some embodiments.

[0069] Fig.56 An exemplary application of a centroid method for estimating the center of a solar panel is shown in accordance with some embodiments.

[0070] Fig.57 is a flow chart of an exemplary method for autonomous solar installation according to some embodiments.

[0071] Fig.58 is a flow chart of another exemplary method for autonomous solar installation according to some embodiments.

[0072] The features and advantages of the present invention will become more apparent from the detailed description set forth below when taken in conjunction with the accompanying drawings, in which the same reference numerals always indicate corresponding elements. In the accompanying drawings, the same reference numerals generally indicate identical, functionally similar and / or structurally similar elements. DETAILED DESCRIPTION

[0073] Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings.

[0074] Figure 1 A perspective view of a solar panel handling system and a box of solar panels according to an embodiment of the present disclosure is shown. The solar panel handling system may include an end-of-arm assembly tool 100 that can be coupled to individual solar panels 120 from a box of solar panels and move them into position relative to a mounting structure for installation.

[0075] The end-of-arm assembly tool 100 may include a frame 102 and one or more attachment devices 104 coupled to the frame 102. Exemplary attachment devices 104 include suction cups or other structures that can be releasably attached to a surface of a solar panel 120 and, at least generally, maintain attachment during manipulation of the solar panel 120 by the end-of-arm assembly tool 100. The frame 102 may be composed of a plurality of trusses 102-A for providing structural strength and stability to the frame 102. The frame 102 also serves as a base for the end-of-arm assembly tool 100 and other related components of the solar panel handling system disclosed herein.

[0076] Other relevant components of the solar panel handling system disclosed herein may be coupled to the frame 102 to fix the relative positions of the components on the end-of-arm assembly tool 100. One or more of the various components of the solar panel handling system may be coupled to one or more trusses 102-A to fix the relative positions of the components on the end-of-arm assembly tool 100.

[0077] For example, the attachment device 104 is configured to reliably attach to a planar surface, such as a surface of a solar panel, by using a vacuum. In a suction cup embodiment, the suction cup can be actuated by pushing the cup against the planar surface, thereby pushing air out of the cup and forming a vacuum seal with the planar surface. Thus, the planar surface is adhered to the suction cup with a certain adhesion strength, which depends on the size of the suction cup and the integrity of the seal with the planar surface. In some embodiments, the suction cup is engaged with the solar panel to form an airtight seal, and then a vacuum pump sucks air out of the suction cup to create the vacuum required for proper adhesion to the solar panel. In some embodiments, when the planar surface is sealed to the suction cup, an air inlet (not shown) provides air to the planar surface so as to deactivate the vacuum and release the planar surface from the suction cup.

[0078] The system may also include a linear guide assembly 106 coupled to the end-of-arm assembly tool 100. The linear guide assembly 106 includes a linearly movable gripping tool 108 having an engagement member 108-A configured to engage a clamp assembly coupled to a mounting structure. The linear guide assembly 106 may be actuated to move the gripping tool 108 along an axis, such as between an extended position and a retracted position. The axis of movement of the gripping tool 108 may be parallel to the axis of the mounting structure. Thus, the linear guide assembly 106 may move the gripping tool 108 and the engagement member 108-A along the mounting structure.

[0079] In some embodiments, the engagement member 108-A may include an electromagnet that can be actuated to grasp the clamp assembly 602 (see Fig. 6A , 6B). Alternatively or additionally, the engagement member 108-A may include a gripper to prevent disengagement between the clamp assembly 602 and the engagement member 108-A when the linear guide assembly 106 is actuated to move the clamping tool relative to the mounting structure, as described in more detail elsewhere herein.

[0080] The linear guide assembly 106 is actuated using a force torque sensor 110. In some embodiments, the linear guide assembly 106 and the force torque sensor 110 may form a rack and pinion structure such that rotation of the force torque sensor 110 causes advancement or retraction of the gripping tool 108. In some embodiments, the linear guide assembly 106 may be a hydraulic assembly that includes a telescopic shaft coupled to the gripping tool 108. In such embodiments, the force torque sensor 110 may be configured in the form of a pump for pumping hydraulic fluid. In other embodiments, the force torque sensor 110 may be configured in the form of or coupled to a linear drive motor that engages a surface of a telescopic shaft coupled to the gripping tool 108.

[0081] In some embodiments, the linear guide assembly 106 may include an electric rod actuator to move the clamping tool 108 parallel to the axis of the mounting structure.

[0082] In some embodiments, the guide assembly 106 may include a roller 606 to facilitate movement of the gripping tool 108 along the mounting structure 604. For example, the roller may include a bearing or other component designed to reduce friction when the gripping tool 108 moves relative to the mounting structure. The roller may be coupled to a sensor, such as a force sensor or a rotation sensor, to provide feedback to the controller.

[0083] In some embodiments, the guide assembly may include a spring mechanism 608 that enables the gripping tool 108 to tilt a small amount (up to 15 degrees) relative to the mounting structure 604. This tilting may occur when the orientation assembly 804 tilts the end-of-arm assembly tool 100 relative to the mounting structure 604 in order to properly level the solar panel.

[0084] The system may also include a terminal box 112 coupled to the frame 102. The terminal box 112 may include a controller configured to control the force torque sensor 110 and the attachment device 104. In some embodiments, the terminal box 112 may also include a power supply or a power controller for controlling power to various components.

[0085] In some embodiments, the controller 112 may include a processor operatively coupled to a memory. The controller 112 may receive input from a sensor associated with the solar panel handling system (e.g., an optical sensor or proximity sensor 108-B described elsewhere herein). The controller 112 may then process the received signal and output a control command for controlling one or more components (e.g., the linear guide assembly 106, the clamping tool 108, or the attachment device 104). For example, in some embodiments, the controller 112 may receive a signal from a proximity sensor that determines that the clamp assembly is approaching the trailing edge of the solar panel being installed, and accordingly reduce the speed of the linear guide assembly 106 to reduce excessive force and impact on the solar panel.

[0086] refer to Figure 8 In some embodiments, the solar panel processing system may further include an optical sensor 802, such as a camera, a photodetector, or any other optical imaging or light sensing device. The optical sensor is suitably located on the frame 102, for example, at Figure 8 802-A in FIG. 1 , or at an interior position of frame 102 that has a field of view that includes the leading edge of the solar panel, such as Figure 8 802-B in the figure. The optical sensor may be configured to sense the orientation of the solar panel relative to the mounting structure during operation of the arm end assembly tool. In some embodiments, the optical sensor may be configured in the form of one or more light guided levels (not shown). In such an embodiment, one or more light beams (e.g., laser beams) may be projected from one end of the arm end assembly tool 100 (e.g., a first position on the frame 102) along or parallel to the axis of the mounting structure 604. One or more photodetectors may be positioned at the other end of the arm end assembly tool 100, such as a second position on the frame 102, to detect the one or more laser beams. Therefore, if the solar panel 120 being installed is not properly oriented or properly leveled relative to the mounting structure 604, the solar panel 102 may block part or all of the one or more laser beams, thereby causing a change signal from the one or more photodetectors, thereby indicating that the solar panel 120 is not properly oriented or properly leveled relative to the mounting structure 604.

[0087] In some embodiments, one or more sensors (e.g., optical sensors 802) may be used to detect and identify objects in order to locate and control the installation with improved accuracy. The sensors may be implemented in conjunction with a neural network, such as an artificial intelligence (AI) system. For example, the neural network may include acquiring and correcting images associated with the solar panel processing system, solar panels (both installed and to be installed), and the installation environment (both of the following: the natural environment, such as terrain; and installed equipment, such as structures associated with the solar panel array). In addition, for example, the neural network may include acquiring and correcting position or proximity information. The corrected images and / or corrected position or proximity information are input into the neural network and processed to estimate the movement and positioning of equipment of the solar panel processing system, such as the movement and positioning of equipment associated with autonomous vehicles, storage vehicles, robotic equipment, and installation equipment. The estimated movement and positioning are published to a control system associated with each device of the solar panel processing system or a master controller of the entire solar panel processing system.

[0088] In some embodiments, the signal from the optical sensor can be input to the controller. In some embodiments, the solar panel processing system can also include an orientation component 804 (see Figure 8 ), which is configured to tilt the end-of-arm assembly tool 100 relative to the mounting structure 604. In such an embodiment, the controller 112 can control the orientation in response to input from an optical signal indicating that the solar panel being installed is not properly oriented or properly leveled relative to the mounting structure (e.g., the torque tube 604). It will be appreciated that while the orientation assembly 804 is shown as being coupled to the force torque sensor 110, those of ordinary skill in the art will readily recognize other means of implementing the orientation assembly 804.

[0089] In some embodiments, the controller 112 may also be configured to control the attachment device 104 to activate or deactivate its attachment / detachment. For embodiments where the attachment device 104 is a suction cup, vacuum may enable coupling or release of the solar panel 120 to the end-of-arm assembly tool 100 .

[0090] In some embodiments, the mounting structure 604 may have an octagonal cross-section, such as Fig. 6A , 6B 7A-7D, to form a torque tube that prevents accidental sliding of the clamp assembly 602. However, other cross-sectional shapes, such as square, oval or other shapes, may also be used. In addition, the mounting structure 604 may use a circular cross-sectional shape.

[0091] In some embodiments, the assembly tool 100 may be configured to be used with an assembly mobile robot 903 (an example of which is shown in FIG. Fig. 9 and Fig.10) coupled. The assembly mobile robot 903 can be configured to position the end-of-arm assembly tool 100 relative to a stack or storage container 905 of solar panels, move a selected solar panel and position the selected solar panel relative to the mounting structure 604. In some embodiments, the assembly mobile robot 903 can be operably coupled to the end-of-arm assembly tool 100 via the force sensor 110 (or, where applicable, the orientation assembly 804). In some embodiments, the assembly mobile robot can also be operably coupled to a controller, thereby enabling an operator of the assembly mobile robot to control various functions of the end-of-arm assembly tool 100, such as activation and / or deactivation of the connection device 104, advancement and / or retraction of the clamping tool and / or activation and / or deactivation of the engaging member relative to the clamp assembly.

[0092] Reference now Figure 1 , 6A , 6B, 7A-7D, 9 and 10, in operation, the solar panel 120 is obtained and positioned above the mounting structure 604. The solar panel is then tilted relative to the mounting structure 604 so that the leading edge of the solar panel (i.e., the edge that will be adjacent to the edge of the previously installed solar panel, or, for the first solar panel, the edge that will be adjacent to the stopper secured to the mounting structure 604) is oriented closer to the mounting structure 604 than the opposing trailing edge. The leading edge is then placed in a receiving channel (a receiving channel positioned along the edge of the previously installed solar panel, i.e., as part of a clamp assembly, or a receiving channel in a stopper) and the inclination of the solar panel is reduced to a mounted position on the mounting structure. As the solar panel is biased into the receiving channel, the angle of inclination is reduced so that, in this mounted position, an edge region of the top planar surface of the solar panel (i.e., the photovoltaically active surface facing the sun) is captured within the receiving channel. Fig. 6A and 6B An exemplary embodiment of a receiving channel 610 on a clamp assembly 602 is shown in FIG.

[0093] Once the solar panel is in position on the mounting structure, the force torque actuator 110 actuates the guide assembly 106 of the end-of-arm assembly tool 100 to bring the engagement member 108-A of the clamping tool 108 into contact with the clamp assembly 602. The clamp assembly is initially positioned outside of the area on the mounting structure to be occupied by the solar panel being installed, but close enough for the relevant components of the end-of-arm assembly tool 100 to reach. The surfaces and features of the engagement member 108-A can be positioned and sized to match complementary features on the clamp assembly 602. After this contact, the force torque actuator 110 is actuated (continues to be actuated or actuated in a second mode) to cause the clamp assembly 602 to slide axially along a portion of the length of the mounting structure 604. The axial sliding of the clamp assembly 602 engages the receiving channel of the clamp assembly 602 with the trailing edge of the solar panel just installed. Sensors, such as those in the force torque actuator 110 or in the clamping tool 108, can provide feedback to the controller indicating full engagement of the receiving channel of the clamp assembly 602 with the trailing edge of the solar panel. Once the clamp assembly 602 is in place, the guide assembly 106 is retracted and the next solar panel can be installed.

[0094] In some embodiments, the linear guide assembly 106 may include a proximity sensor 108-B configured to sense the distance between the engagement member 108 and the trailing edge of the solar panel 120 during the installation operation of the solar panel 120. The output from the proximity sensor 108-B may be used to appropriately control the speed of the gripping tool 108 during the operation of the linear guide assembly 106 so as to avoid excessive force and impact on the solar panel 120. In some embodiments, the proximity sensor 108-B may be, for example, an optical or audio sensor (e.g., sonar) that detects the distance between the leading edge of the solar panel 120 and the engagement member 108; in other embodiments, the proximity sensor 108-B may be a limit switch that retracts by contact.

[0095] Further references Fig. 9 and Fig.10 , the assembly mobile robot 903 can be implemented using a ground vehicle 907. For example, the ground vehicle 907 can be implemented as an electric vehicle (EV). The ground vehicle 907 can move autonomously to the vicinity of the mounting structure 604. Although not shown, the ground vehicle 907 can move along a track or rail attached to or separated from the mounting structure. In some embodiments, the ground vehicle 907 can be controlled using sensors, or based on input or feedback from sensors. For example, the sensor can be an optical sensor or a proximity sensor. In another embodiment, a neural network using artificial intelligence can be used to control the movement of the ground vehicle 907, for example by analyzing the operating environment and formulating instructions for the movement of the ground vehicle.

[0096] Fig.10 An embodiment of a solar panel handling system having two robotic arms is illustrated, wherein two assembly tools are coupled to an assembly mobile robot using respective robotic arms.

[0097] like Fig. 9 As shown, a storage container 905 containing solar panels to be installed can be placed on a ground vehicle. Here, Fig. 9 A solar panel processing system including an arm assembly tool 100 is illustrated, which uses a robotic arm to couple with an assembly mobile robot. Fig.10 As shown in , one or more storage containers 905 may be disposed on corresponding one or more module vehicles 1005 adjacent to a ground vehicle 907. In this manner, Fig.10 A solar panel handling system with two robotic arms is illustrated, wherein two assembly tools are coupled to an assembly mobile robot using respective robotic arms. In embodiments of the present disclosure, the robotic arm may be an articulated arm having two or more sections coupled using joints, or alternatively may be a truss arm. The illustrations herein are intended to disclose the use of any type of arm according to the present disclosure.

[0098] according to Fig. 9 For example, the robotic arm of the arm assembly tool 100 having the upper section 908 and the lower section 909 can provide increased operational flexibility while maintaining light weight and simple operation. Fig. 9 As additionally shown in FIG. 1 , the second robotic arm 911 may be provided with an arm assembly tool 100 having a nut tightener or nut driver at its end to secure the solar panel to the mounting structure 604. Although any type of robotic arm may be used for the second robotic arm 911, Fig. 9 An example of using an articulated arm with a nut tightener or nut driver at its end is illustrated. Here, the robotic arms 100 and 911 can operate autonomously using computer vision with neural network and artificial intelligence control. Alternatively, the robotic arms 100 and 911 can be manually operated or remotely controlled.

[0099] In some embodiments, the ground vehicle 907 may be an autonomous vehicle in which a neural network and artificial intelligence control movement and operation, and the module vehicle 1005 is towed or coupled to the ground vehicle 907. In other embodiments, the module vehicle 1005 may be an autonomous vehicle in which a neural network and artificial intelligence control movement and operation, and the ground vehicle 907 is towed or coupled to the module vehicle 1005. In addition, in some embodiments, the assembly mobile robot 903 is also mounted on one of the ground vehicle 907 and the module vehicle 1005. In other embodiments, the assembly mobile robot 903 may be mounted on a dedicated robot vehicle.

[0100] Figures 11A to 11C The process for installing solar panels is shown in FIG. Fig.11A As shown in , a pallet of solar panels can be delivered by truck. In some embodiments, the pallet can constitute a storage container 905 for the solar panels. The pallet can include machine-readable markings, such as bar codes, QR codes, or other manufacturing references, which can be read to provide information about the solar panels, installation instructions, or other information to be used during the installation process, particularly information to be used by neural network and artificial intelligence control. For example, this information may include the number of solar panels, the type of solar panels, the physical characteristics of the solar panels (e.g., size), installation-related characteristics (e.g., hardware type and location), installation instructions or other characteristics of the solar panels, the storage of the solar panels on the pallet, and installation-related information. In addition, using machine-readable markings, the system can control the supply or replenishment of panel boxes in the correct order and / or ensure that panels with similar impedance from the factory are used.

[0101] like Fig. 11B As shown in , mechanized equipment such as a forklift can be used to move and position the pallet on a ground vehicle. Here, the forklift can be manually operated, remotely operated, or autonomously driven. Fig. 11B In the embodiment, the pallet is located on a ground vehicle. Alternatively, the pallet may be located on a modular vehicle. Then, as Fig. 11C As shown in FIG, a solar panel is installed using a robot arm. In the illustrated example, two arms are used to handle respective solar panels to be installed on respective mounting structures. Here, a ground vehicle moves between two respective mounting structures. In addition, a module vehicle is provided, which can be separated from the ground vehicle.

[0102] As will be appreciated by those skilled in the art, modifications and variations in the embodiments may be used. Fig. 12A and 12B As shown in , two modular vehicles may be provided for respective robotic arms. In another alternative, the modular vehicle may be connected to the ground vehicle rather than being detached. Thus, as Fig. 12A As shown in , the robotic arm can engage the corresponding solar panel to be installed, such as Fig. 12B as shown in .

[0103] In some embodiments, Fig.13 As shown in , computer vision registration can be used to achieve the installation. For example, as mentioned above, optical sensors and the like can be used with neural networks for artificial intelligence.

[0104] In some embodiments, Fig.14As shown in , if a modular vehicle is used with a ground vehicle, then when all of the solar panels of the modular vehicle are installed, the modular vehicle can be exchanged with a supplementary modular vehicle. Here, a computer vision process can be used to communicate with and control an autonomous independent vehicle (e.g., a forklift) to bring additional solar panel boxes. Thus, the supply of solar panels can be supplemented.

[0105] In a replenishment operation using the example of a forklift, a forklift (whether autonomous, remotely controlled, or manually operated) can be used to return empty boxes or containers of solar panels to a disposal area, remove straps, open lids, or cut off box faces of boxes being transported, pick up boxes to correct the rotation / orientation of solar panels, or other tasks. In addition, the forklift can remain near the ground vehicle to wait for the system to exhaust the next box of solar panels. Thus, the forklift can manually or automatically discard the exhausted box, position the next box on the ground vehicle or module vehicle, open the box (including removing straps, opening lids, or cutting off box faces), and back away from the ground vehicle / module vehicle. As described, for example, the replenishment can be autonomous, remotely controlled, or manually operated.

[0106] Figure 15-34 Detailed illustrations of example configurations of systems for mounting solar panels according to embodiments of the present disclosure are provided.

[0107] Fig.35A A block diagram of an exemplary image processing pipeline 3500 according to some embodiments is shown. The pipeline 3500 includes a module 3502 for acquiring an image, a module 3504 for rectifying the image, a module 3506 for neural network image segmentation of the rectified image, a module 3508 for post-processing the output of module 3506 using computer vision techniques, a module 3510 for performing a Hough transform on the output of module 3508, a module 3512 for filtering and segmenting the Hough lines output by module 3510, a module 3514 for identifying the intersections of the horizontal and vertical Hough lines output by module 3512, a module 3516 for estimating the panel pose based on the horizontal and vertical Hough line intersections (e.g., using the 3D panel geometry and the locations of the corners in the image), and a module 3518 for publishing the pose estimate. Fig.35B An exemplary rectified acquired image 3520 (the output of modules 3502 and 3504) is shown, which includes images of a solar panel 3522 and other objects 3524-2 (eg, tape) and 3524-4 (eg, wire). Fig.35C A method for Fig.35B An exemplary output 3526 (output of module 3506) of obtaining a neural network image segmentation of a rectified image is shown in FIG. Fig.35DAn exemplary panel corner detection 3528 (output of module 3514) is shown in accordance with some embodiments. In this example, corners 3530-2 and 3530-4 are detected based on horizontal lines 3532-4 and 3532-8 and vertical lines 3532-2 and 3532-6.

[0108] Fig.36 An example 3600 of images 3602, 3606, and 3610 of a road under different lighting conditions and segmentation masks 3604, 3608, and 3612 for these images is shown in accordance with some embodiments. Conventional computer vision techniques are useful when the environment is ideal. However, glare, overexposure / underexposure may negatively impact object detection algorithms. Machine learning techniques can overcome environmental inconsistencies, learn from examples with glare and lighting issues, and generalize to new data during inference.

[0109] Example Solar Panel Segmentation

[0110] Some embodiments perform solar panel segmentation by capturing images of the solar panel and torque tube under different lighting conditions. Fig.37A An example of a captured image 3700 including a solar panel 3702 and a torque tube 3704 is shown in accordance with some embodiments. Some embodiments annotate the captured image of the solar panel. Fig.37B An example of an annotated image 3706 (sometimes referred to as an annotated ground truth mask) of a captured image 3700 according to some embodiments is shown. The annotated image includes a black background 3708, an outline of a torque tube 3712 shown in dark gray, and an outline of a solar panel 3710 shown in light gray. Some embodiments create a dataset based on the annotated image, train an image segmentation model using the dataset, and use the trained model to detect solar panels and torque tubes in poor lighting conditions. Fig.37C An exemplary prediction 3714 of a trained model according to some embodiments is shown. The trained model predicts background 3708, solar panel 3710, and torque tube 3712, as well as object 3716 in the background ( Fig.37A and 37B not shown).

[0111] Some embodiments continuously collect images (and build a dataset) and use these images to improve the accuracy of the model. Some embodiments use human annotations to improve the accuracy of the model. Some embodiments allow users to adjust the parameters of the segmentation model.

[0112] Some embodiments include separate models for semantic segmentation and instance segmentation. Fig.38A An example of image classification is shown. In this example, image classification detects the presence of bottle 3802, cup 2806, and cube 3804. Fig.38B Shown for Fig.38A 3816. In this example, rectangle 3808 locates bottle 3802, rectangle 3810 locates first cube, rectangle 3812 locates cup 3806, and rectangles 3814-2 and 3814-4 locate cube 3804. Fig.38C An example of semantic segmentation 3818 is shown in accordance with some embodiments. Semantic segmentation helps identify label 3820 of bottle 3802, label 3822 of cube 3804, and label 3824 of cup 3806. Fig.38D An example of instance segmentation 3826 is shown in accordance with some embodiments. In addition to identifying labels 3828 and 3832 for bottles 3802 and 3806, instance segmentation can distinguish instances of cube 3804, thereby correspondingly determining labels 3830, 3834, and 3836 for cube 3804. Instance segmentation can distinguish multiple solar panel instances in a single image. Instance segmentation generates a mask for each class instance in the camera frame, enabling identification and localization of individual panels, utilizing the same data collected for semantic segmentation, and supporting illumination invariance. Fig.39 An example of instance segmentation 3900 for solar panels is shown in accordance with some embodiments. In this example, panel instances 3902, 3904, 3906, 3908, and 3910 are identified. The example shows instances of panels (e.g., instances 3902 and 3904) having different orientations.

[0113] Fig.40An exemplary image processing system 4000 according to some embodiments is shown. System 4000 includes multiple cameras, including a camera 4002 for coarse positioning, a camera 4004 for capturing images when picking up a panel, and a camera 4006 for capturing images when placing a panel. Camera 4002 includes a narrow field of view lens, and cameras 4004 and 4006 each include a wide field of view lens. Camera 4002 can be used to identify trailer position and initial robot position. In some embodiments, cameras 4004 and 4006 can be the same camera. In some embodiments, camera 4002 can also be used to locate fixtures and center structures during solar panel installation. Cameras 4002, 4004, and 4006 are coupled to corresponding image sensors 4008, 4010, and 4012 (e.g., AR0820 sensors). In some embodiments, the image sensor is optimized for both low light and / or high dynamic range performance. In some embodiments, the system 4000 includes a high-speed digital video interface (e.g., FPD-link) and Ethernet for connecting the camera to one or more GPUs (e.g., a GPU 4014 suitable for edge AI processing, such as Nvidia XT, a GPU suitable for image processing applications, such as Nvidia AGX Xavier TM ). GPU 4016 implements the exemplary image processing pipeline 3500 described above and is connected to the robot controller 4018 using Ethernet. According to some embodiments, GPU 4014 may be removed in certain systems and the output from the sensor may be connected directly to GPU 4016.

[0114] Some embodiments continue to capture training images while installing solar panels. Figure 41 shows a trailer system 4100 with a coarse camera 4102 that can be used to capture training images in accordance with some embodiments. The AI / neural network system takes into account internal parameters (e.g., camera / lens distortion) as well as external parameters (e.g., camera position and angle on the robotic arm and the pose of the robotic arm at the time of image capture) to calculate where each of the four corners of the panel are.

[0115] Fig.42A and 42B Correspondingly, histograms 4200 and 4202 of the norm of the pose error of the neural network when using and not using coarse position are shown in accordance with some embodiments. As shown, with coarse position, the neural network error (the difference between the true position of the corner of the solar panel and the neural network's estimate) is greatly reduced (in some cases, from nearly 5 inches to around 0.7 inches).

[0116] Fig.43A and 43BExamples 4300 and 4302 of rough locations of solar panels (e.g., locations 4304, 4036, 4308, and 4310) using a B Mask R-CNN model are shown in accordance with some embodiments. Mask R-CNN is a convolutional neural network (CNN) for image segmentation and instance segmentation. This deep neural network detects objects in an image and generates a high-quality segmentation mask for each instance. Mask R-CNN is based on a region-based convolutional neural network. Image segmentation is the process of dividing a digital image into multiple segments or sets of pixels corresponding to image objects. This segmentation is used to locate objects and boundaries (lines, curves, etc.). Mask R-CNN can be used for semantic segmentation and instance segmentation. Semantic segmentation classifies each pixel into a fixed set of categories without distinguishing between object instances. In other words, semantic segmentation involves identifying / classifying similar objects as a single class from the pixel level. All objects are classified as a single entity (solar panels). Semantic segmentation is sometimes called background segmentation because it separates the subject of the image (e.g., solar panels, lines) from the background. On the other hand, instance segmentation (sometimes called instance recognition) involves correctly detecting all objects in an image while also accurately segmenting each instance. In this sense, instance segmentation combines object detection, object localization, and object classification, and helps distinguish instances of each object in an image. In addition to having two outputs for each candidate object, including a class label and a bounding box offset, in Mask R-CNN, the third branch also outputs an object mask. This mask output helps to extract a finer spatial layout of the object. When compared to other models, in addition to being easier to train, having good performance and efficiency, Mask R-CNN is also particularly suitable for solar panel recognition because the neural network is able to perform both semantic segmentation and instance segmentation. In addition, the mask branch adds only a small computational overhead, enabling fast solar panel detection and fast experiments. Mask R-CNN can be used for image segmentation, thereby identifying objects in an image and creating a mask within the boundaries of the object.

[0117] Fig.44 A system 4400 for solar panel installation is shown according to some embodiments. According to some embodiments, the system 4400 includes a main housing 4404, a battery housing 4402, an upper robotic end-of-arm tool (EOAT) 4406, a lower robotic EOAT 4408, and a bracket 4410 for holding a solar panel 4414 on a trailer 4412.

[0118] According to some embodiments, Fig.45A A vision system 4502 is shown mounted on a trailer and used to estimate the pose of the structure 4500, and Fig.45BShown is a magnified view of vision system 4502. Various embodiments may mount the vision system on different parts of the ground vehicle, on a robotic arm, or on an end-of-arm tool.

[0119] According to some embodiments, Fig.46A A vision system 4602 for module picking 4600 is shown, and Fig.46B A magnified view of vision system 4602 is shown, which includes a high-resolution camera with laser line generation.

[0120] Fig.47A A method for positioning 4702 (an enlarged view of which is shown in FIG. Fig.47B ) and position 4704 (an enlarged view of which is shown in Fig.47C System 4700 measured as the distance between modules at a module angle (i.e., when facing the module).

[0121] Fig.48A A system 4800 for laser line generation for detecting tube and fixture positions is shown in accordance with some embodiments. According to some embodiments, Fig.48B shows an enlarged view of the laser line generation system 4802, and Fig.48C View 4804 of laser line generation (horizontal lines detect the fixture, and vertical lines detect the tube) is shown.

[0122] Fig.49A A vision system 4900 is shown for estimating the position of a tube and a clamp and locating a nut on the clamp in accordance with some embodiments. Fig.49A Also shown is a socket wrench 4902 for tightening the nut. Fig.49B A magnified view of the visual system is shown. Fig.49B As shown in Figure 1, the camera uses the laser line described above to locate the tube and clamp, and uses a flashing ring light to locate the nut on the clamp. The laser provides an accurate estimate of the position of the tube and clamp. The flashing ring light is used to locate the Fig.49C When tightened, the nut compresses the clamp to hold the panel in place.

[0123] Example solar panel installation using artificial intelligence

[0124] Fig.50AA flow chart of a method 5000 for autonomous solar installation according to some embodiments is shown. The method includes acquiring (5002) an image of an ongoing solar installation. The image includes an image of one or more solar panels and one or more torque tubes. In some embodiments, acquiring the image includes using one or more filters to avoid direct solar glare in order to detect end-of-arm tooling (EOAT). In some embodiments, acquiring the image includes using a high-resolution camera with laser line generation to identify one or more torque tubes and / or fixture locations. In some embodiments, the image includes an image of a fixture and / or a central structure for the ongoing solar installation. In some embodiments, the image includes an image of a fixture and / or a central structure for the ongoing solar installation. In some embodiments, a wide-angle fisheye lens is used to acquire multiple images to create a composite HDR (high dynamic range) image within the camera hardware. The images are sent through the Robot Operating System (ROS), which is a high-level software framework for integrating robots and servos, so that the images are corrected (e.g., from fisheye distortion to a flat image) using OpenCV (image processing framework) modules. The area and bit depth are then selected and used to shrink the HDR image to a standard 8-bit image, effectively cropping it for use as input to train the neural network. At the acquisition point, the robot pose can be stored (using ROS) to create a transformed camera result relative to the trailer (trailer system for solar panel installation). This can include the robot position and the camera position to identify the location of the image in 3D space.

[0125] The method also includes detecting (5004) solar panel segments by inputting the image into a trained neural network, the neural network being trained to detect solar panels under poor lighting conditions. The neural network can be implemented using software and / or hardware (sometimes referred to as neural network hardware) that uses a conventional CPU, GPU, ASIC, and / or FPGA. In some embodiments, the trained neural network includes: (i) a model for semantic segmentation for identifying solar panel segments; and (ii) a model for instance segmentation for identifying multiple solar panels. In some embodiments, the trained neural network uses the Mask R-CNN framework for instance segmentation. The trained neural network detects solar panel segments based on features extracted from images of an ongoing solar installation. In some embodiments, the acquired image is input to the neural network via ROS (e.g., the input image arrives at the neural network module (Detectron) from the OpenCV module). According to some embodiments, the following reference Fig.50BDescribes an exemplary technique for training a neural network. In some embodiments, the neural network performs image segmentation to identify a panel (or multiple panels) without identifying the location of the panels. In some embodiments, there is one model that implements both functions (semantic segmentation and instance segmentation). Some embodiments use two instances of the same model to optimize throughput. In this case, the camera takes two images and each image is passed through one instance. Running two models allows twice as many images to be processed simultaneously.

[0126] The method also includes estimating (5006) the panel pose of the one or more solar panels using a computer vision pipeline based on the solar panel segments. In some embodiments, the computer vision pipeline includes one or more computer vision algorithms for post-processing, Hough transforms, filtering and segmentation of Hough lines, finding horizontal and / or vertical Hough line intersections, and panel pose estimation using predetermined 3D panel geometry and corner positions. In some embodiments, the computer vision pipeline locates a fixture and / or a center structure to estimate the panel pose. In some embodiments, the computer vision pipeline locates one or more torque tubes and / or fixture positions to estimate the panel pose. In some embodiments, the computer vision pipeline locates a nut. After locating the nut, a socket wrench mounted on a smaller robotic arm can engage the nut and tighten it to secure the panel in place. Before performing this step, the fixture may become loose and the panel may fall due to wind.

[0127] In some embodiments, estimation of the panel pose is performed using conventional machine vision hardware in order to locate the position of the panel in 3-D space. In some embodiments, this is a rough identification of circular edges and is not intended to be very precise. The Hough transform can then be used to determine the exact location of the edge, which is followed by inferring the edge lines of the panel, determination of the position of the panel intersections, and identification of the panel corners. The panel corners are released to identify the position of the panel relative to the robot. For example, based on the panel geometry in 3-D, the pose of the panel is calculated based on the location of the corners of the panel in the image.

[0128] In some embodiments, to estimate the panel pose, the computer vision pipeline uses a PnP (Perspective-n-Point) solver with the camera's intrinsic parameters (it knows its own camera distortion and parallax). The extrinsic parameters then capture the camera's position relative to the robot using the robot's and EOAT's poses at the time of image capture. The robot pose can be captured continuously using a timestamp. The timestamp can then be used to match the robot pose to the camera acquisition timestamp. In some embodiments, the computer vision pipeline uses the known poses of the robot and end-of-arm tooling (where the camera is located) at the time of image capture to calculate the position of one or more corners of the panel.

[0129] The method also includes generating (5008) a control signal based on the estimated panel pose for operating a robotic controller to install the one or more solar panels. In some embodiments, after finding the panel, the position is projected along the tube to find the fixture pixel to identify the fixture position (e.g., how far away the fixture is and how close it is to the fixture puller). Some embodiments use the fixture position to verify whether the fixture is within the allowable window required by the fixture puller on the EOAT. Some embodiments use the center structure to determine the order of placing one or two panels to avoid collision with the fan gear. Some embodiments use the panel position to ensure that the trailer is in an effective position relative to the tube so that the robot is within reach of the work that needs to be performed. Some embodiments use the pose from the leading panel to subsequently guide the lower robot in its thin tube acquisition, which drives the position of the upper and lower robots for panel placement and nut driving. In some embodiments, the thin tube acquisition described above uses horizontal and vertical lasers to create a profiler system that finds the tube and fixture position. This refines the working pose from the thick tube from 10-20mm and reduces it to less than plus or minus 5mm. At the first panel the thick tubes are within 5mm but as this is projected the error grows and thin tubes are used to limit this error to plus or minus 5mm.

[0130] Fig.50B A flow chart of a method 5010 for training a neural network for autonomous solar installations is shown in accordance with some embodiments. The method includes: acquiring (5012) a plurality of images of a solar panel installation under different lighting conditions; annotating (5014) the plurality of images to identify solar panel images (instead of or in addition to automatically annotating the images, manually annotated images may be used); and training (5016) one or more image segmentation models using the solar panel images to detect solar panels under adverse lighting conditions. In some embodiments, the neural network is manually trained using various images, such as several images of different backgrounds (e.g., grass, dirt), different numbers of panels, clamps, panels with or without cardboard corners, in various weather conditions (e.g., clear conditions, rainy conditions). Within these images, masks (lines) are drawn to indicate which pixels represent panels, clamps, tubes, and central structures. These images and their masks are used to create a series of pseudo images, which are then used by the neural network during training. The pseudo images are input images with angle distortions to enable multiple training with the same input images. For example, 300-1000 real (input) images may be used for training, and for each real image 10-20 fake images may be created.

[0131] Synthetic images can be used to train machine learning algorithms for solar panel installations. However, synthetic images or other synthetic training data may be cost-prohibitive and may not even be generated in some cases. Producing realistic images may require weeks of training, even on expensive hardware, and may require a lot of supervision. Generating useful synthetic data is often a process of trial and error. There is also a risk associated with overtraining with synthetic data. Since artificial data is often used in areas where real-world data is scarce, the generated data may not accurately reflect real-world scenarios. In view of these limitations, system and method techniques that do not rely on synthetic data are needed. The techniques described herein are not location-restricted because images are not used for learning. These techniques are centered on the panel and background contours do not affect correctness. Some embodiments use a predictor correction mechanism whereby the center is predicted, tested if it fails, and the failure is used as feedback to improve the next estimate of the center.

[0132] Some embodiments use solar physics, the altitude and / or azimuth of the sun to obtain the relative position of the sun in the sky. Some embodiments estimate glare based on an estimate of the sun's position. Some embodiments use high-fidelity noise removal and image correction algorithms to false color glare. Some embodiments identify the center and corners of panels, detect wear on torque tubes and false color wear lines / scratches in shadows (e.g., blue shadows), place panels, and / or detect structures such as fixtures and fan gears.

[0133] Some embodiments pick up solar panels by detecting the corners and center of the panel to assist the top robot (sometimes called the top robot arm) in accurately picking up the solar panel. The panels may be of different sizes. Some embodiments detect the placed panel and assist the bottom robot in accurately aligning the panel relative to it. Some embodiments detect the position of the fixture and assist the operator in the movement of the fixture. The panel is protected from impact. For example, some embodiments detect the fan gear position and assist the operator in safely placing the solar panel relative to it. The panel is also protected from collision. These aspects are described in detail below.

[0134] According to some embodiments, the first step for picking up a solar panel is to estimate the center position of the panel. The upper robotic arm (e.g., upper robotic end-of-arm tool (EOAT) 4406) can use the center estimate to pick up the panel. Solar panels may have different sizes but have the same overall shape, which is primarily rectangular, even if the solar panels come from different manufacturers. The algorithmic techniques described herein utilize the rectangular shape of the solar panel to estimate the center. The size may be different, which means that the width and length of the solar panel may change, which affects the center position of the panel and, therefore, the corner position of the panel. Although described herein with respect to rectangular solar panels, these methods and techniques may also be applied or appropriately modified for use with solar panels of other polygonal shapes.

[0135] After identifying the center and corners, the upper arm can pick up the solar panel and enter placement mode. In placement mode, the picked up panel needs to be aligned with the torque tube and aligned relative to the existing (or previously placed) panel to avoid collision with the panel. Some embodiments also detect other structures in the installation environment, such as fixtures, to avoid collision with such structures. According to some embodiments, the placed panel cannot collide with the existing panel and must be properly aligned with the existing panel so that once the alignment process is completed, the lower robotic arm (e.g., lower robot EOAT 4408) rises and tightens the screws on the fixture. Thus, the upper robotic arm picks up and places the solar panel, and the lower robotic arm ensures alignment, optionally by using a laser. In some embodiments, image recognition software detects other structures in the installation environment, such as fan gears, notifying the operator so that the position of the panel can be offset to avoid collision or impact with the structure.

[0136] During pick, place and / or alignment, the movement of the sun may have an impact. The movement of the sun may depend on the time of day and / or season. Some embodiments use the altitude of the sun and / or the azimuth of the sun. The azimuth is the angle at which the sun hits the ground. The altitude of the sun is the position of the sun in the sky on a particular day and may be particularly important for solar panel installations. Solar panels typically have a coating with reflective properties on the top. Solar physics needs to be considered. High transmittance is desired so that more photons are converted to electrons, but excessive conversion can lead to poor performance of the solar panel. Therefore, in order to protect it from this undesirable illumination, some embodiments include a thin film coating with reflective properties applied to the solar panel. Due to this sun movement on a particular day and due to the orientation of these solar panels during pick, many reflections and glare are seen on the panels. Therefore, when the camera captures this image, there are dark white spots in the image that affect and disrupt the computer vision algorithm.

[0137] Another failure mode is that because the fixture must be adjusted manually, the fixture must be moved along the torque tube. This is a physical movement of the fixture involving metal contact on metal. The movement of the fixture on the torque tube may cause scratches or abrasions on the torque tube. When the lower robotic arm sees this, it reflects white on a white background. The laser line that is supposed to be an indication of alignment is focused on the metal object, so it reflects white. From an image perspective, since the background is also white, it is impossible to know where the laser line is because the two colors blend. In some embodiments, this can be circumvented using colored (e.g., blue) tape so that there is a background that provides contrast. Because of these issues, a system based on non-synthetic data is needed. The technology described herein provides similar functionality to those described above for picking, placing, and aligning, avoiding collisions and / or impacts through the use of geometric shapes.

[0138] Exemplary method for estimating the geometric center and corners of a panel

[0139] Fig.51A 5100 is a schematic diagram of an exemplary method for estimating the center and corners of a solar panel according to some embodiments. The center P of the panel 5104 and its corners may be estimated by distance simulation, geometric correction, and / or angle adjustment. For example, straight lines within the panel may be identified by using Hough lines or Hough transforms. Some embodiments identify and draw the longest visible horizontal and vertical Hough lines representing the edges of the panel. The longest visible lines are useful because the image 5102 may be only a partial image, or the image may cover only a portion of the panel 5104. Some embodiments randomly select any point (e.g., point A as an internal point or point E as an external point) from the image (any point within the boundaries of the image 5102) and determine whether the point is inside the panel or outside the panel. The image is a combination of a foreground and a background, which corresponds to the environment of the solar panel, such as a support on which the solar panel is located. The image may be a top view, a ground outline, or a moving object; and any part of the image may be the background. If the ray originating from the point intersects the longest visible line at an odd point, the point is inside the panel. If the ray intersects the longest visible line at an even point, then the point is outside the panel. If the point is inside the panel, some embodiments calculate the horizontal and vertical distances to the Hough lines.

[0140] refer to Fig.51A, assuming that point A is determined to be an interior point. The two normals AB and AC fall to the longest visible horizontal line 5106 and the longest visible vertical line 5108, respectively. These lines are not normal to the panel, which means that the α and β (angles) where the normals intersect the longest visible horizontal and vertical lines are the cases where the angles subtended by the rays will not be 90 degrees. This is because these angles will only be 90 degrees when the panel is aligned with the image. This may happen if the longest visible horizontal line and the image boundary are parallel to each other. Since this is not a typical case, α and β will be assumed to be values ​​less than or greater than 90 degrees. Some embodiments repeat these steps for all interior points to create the matrix.

[0141] Some embodiments identify a subset of points that satisfy the following properties: (a) horizontal distance (d horizontal ) is greater than or equal to 45% (or the highest value) of the panel width, which may be a user-entered or previously calculated width; and (b) the vertical distance (d vertical ) is greater than or equal to 45% (or the highest value) of the panel height, which may be a user-entered or previously calculated height. Using this level of tolerance is important because solar panels may come from different vendors and panel widths are not fixed numbers. The 45% value is used for illustrative purposes, and other user-defined values ​​may also be used. Various embodiments may use different percentage values ​​depending on the tolerance for error or the need for accuracy.

[0142] Some embodiments calculate both the horizontal angle (α) and the vertical angle (β) of the intersection of each qualifying point with the longest visible Hough line. Some embodiments utilize the calculated angles to correct the estimated distance. The angles α and β may need to be normalized, meaning that if the distance is evaluated as a vertical or horizontal distance, the distance needs to be corrected based on 90 minus α or 90 plus α and / or 90 minus β and 90 plus β. β equal to 0 would mean α equal to 90 degrees. Therefore, the α and β angles are used as a guide.

[0143] Some embodiments isolate a subset of points that satisfy the following properties: (a) the horizontal distance is greater than a predetermined percentage (e.g., 48%) of the panel width or a maximum value; and (b) the vertical distance is greater than a predetermined percentage (e.g., 48%) of the panel height or a maximum value. vertical is approximately 50% of the panel height, d horizontalWhen α and β are 50% of the panel width and / or α and β are each approximately 90 degrees, point A will asymptotically approach the panel center P. They are sometimes referred to as representative points. These points are closer to the center of the solar panel, corresponding to the bubble box shown on the circle (equiprobability circle or CEP). These points are scattered or distributed in a circular form. Points of interest for estimating the geometric center: (i) are inside the solar panel; (ii) are close to the longest visible horizontal and vertical Hough lines; and (iii) form a nearly normal orientation with the longest visible horizontal and vertical Hough lines. 50% will mean the center, so at least 48% of the points have a CEP error or circular error probability of less than 2%. As long as the evaluated points meet the 48% distance condition, these points are within 2% of the geometric center, but these points are still a group of points. They are not a single point. Some embodiments determine the centroid of the point set to identify the center of the panel. Some embodiments use the Pythagorean theorem to estimate the corners of the panel in two directions. Some embodiments use angles to correct the distance to determine the corners of the panel. The value of 48% is used for illustration purposes, and other user-defined values ​​may also be used. Various embodiments may use different percentage values ​​depending on the tolerance for error or the need for accuracy.

[0144] The center of the panel can be used to control the attachment means, such as suction cups (e.g., 8 suction cups), on the upper robotic arm. These suction cups can be symmetrically distributed on the robotic arm. Centering the robotic arm on the solar panel avoids misalignment when picking up the solar panel. If the solar panel is not picked up in the correct orientation, this means that it takes much longer to place the panel if the pick-up causes a tilt, and the panel must then be aligned next to the previously placed panel.

[0145] The goal might be to complete the pick-up process and the place-down process in under 60 seconds.

[0146] Some embodiments use CEP techniques to estimate the center based on proximity and perpendicularity, and may not include all of the steps described above. The CEP is a circle but not a perfect geometry. Since the points that form the circle are determined through simulation, they will form a theoretical circle that can be visualized based on the best fit geometry. Therefore, the estimated center of the panel will be the centroid of these points. CEP implies that some embodiments provide a predetermined percentage (e.g., 2%) deviation from the actual center. Perpendicularity is a condition that ensures that each candidate point used to form the CEP is close to normal to the longest visible Hough line, using corrected glare and / or symmetry or based on the energy lines of the created pixels.

[0147] Some embodiments use the Shi-Tomasi method and / or the Harris method to estimate the center and / or corners of a solar panel.

[0148] The above-described steps for center and / or corner estimation may be applied to stationary panels, picked panels, and / or previously placed panels.

[0149] The next step is to place the picked panel. Panel placement energies and proximity balance the picked panel and the previously placed panel. In some embodiments, the image is captured in perspective. Some embodiments dynamically compare the fixed panel and the moving panel (e.g., when the picked panel moves in the air). The different heights between the picked panel and the placed panel (reference) create an electrical potential. In some embodiments, placement is complete when 6 degrees of freedom (3 rotations and 3 translations) are aligned for both panels.

[0150] Fig.52 An exemplary process 5200 for placement of solar panels is shown in accordance with some embodiments. Some embodiments calculate (5202) the centers of a fixed panel and a floating panel. Some embodiments draw (5204) a line between the two centers as a function of time. Some embodiments establish (5206) Euler angles and radius vectors at different points in time. Some embodiments move (5208) (or cause a robotic arm to move) the floating panel so that the values ​​of degrees of freedom two through six are close to zero. Some embodiments use computer vision to confirm (5210) the coplanarity of the points or Hough lines of the two panels. Some embodiments slide (5212) (or cause a robotic arm to slide) the floating panel so that the distance between the two centers is approximately equal to the safe offset distance (which can be user input or predetermined) plus the width of each panel (which can be pre-calculated or user input).

[0151] Fig.53 is a schematic diagram of an exemplary placement 53 of a solar panel according to some embodiments. When the panel 5302 is picked up, the panel floats in the air with six degrees of freedom (three rotations and three translations). The panel may be tilted, angled and / or skewed relative to the placed panel 5306 (sometimes referred to as a fixed panel). Regardless of the position and / or orientation of the picked panel relative to the placed panel, the picked panel has six degrees of separation, which means that three distances and / or three angles may be different. The goal is to align the panels, so the upper robotic arm needs to place the picked panel in such a way that its longest edge and the longest edge of the placed panel are aligned or flush with each other. Based on the placement orientation, the upper robotic arm may place the picked panel so that its shortest edge and the shortest edge of the placed panel are aligned or flush with each other. Therefore, the robotic arm may initially place the floating panel at a certain angle (for example, relative to the plane on which it will ultimately be installed). After the edges are aligned, the robotic arm may tilt the floating panel so that it is parallel to the ground and / or the placed panel. Some embodiments may use the above reference Fig.52 Describe the algorithm.

[0152] refer to Fig.53, panel center 5308 is estimated for the picked panel 5320. Panel center 5310 for the placed panel 5306 may be predetermined or similarly estimated (eg, similar to how the center is estimated for a floating panel for pick and place). Fig.53 Corresponds to the top view of the two panels. The dynamic positional degrees of freedom (DOF) of the picked panel relative to the placed panel is expressed by θ and 5306. Ψ is not shown because this is a top view. "Distance dapproach,1" corresponds to the distance from the upper right corner 5312 of the picked panel 5302 to the upper left corner 5314 of the fixed panel 5306. "Distance dapproach,2" corresponds to the distance from the lower right corner 5316 of the picked panel 5302 to the lower left corner 5318 of the fixed panel 5306. The goal is to move the floating panel to the dashed position 5304 (referred to as the final position, with DOF [X, 0, 0, 0, 0, 0]. When the DOF vectors take the form [Xsafe, 0, 0, 0, 0, 0], the panel is ready for alignment. This occurs when dapproach, 1 and dapproach, 2 reach Xsafe. The Xsafe distance may take into account the fixture thickness. The other degrees of freedom are 0. The projection axis and the reference axis are aligned. This example is shown for illustrative purposes. Other initial configurations and procedures for placement and / or alignment are also possible. For example, vectors other than the Y or Z axis may be driven to zero, and / or the solar panel may be moved in other directions, with the goal of aligning all vectors except one degree of freedom when it becomes safe to align the panel.

[0153] Some embodiments detect structure, such as clamp position and / or fan gear position, to assist an operator with clamp movement and / or safe placement of a solar panel. Fig.54 An exemplary solar panel mounting infrastructure is shown having a fan gear 5402 attached to a torque tube 5404 in accordance with some embodiments. Fig.55A A top view of an exemplary solar panel mounting infrastructure 5524 with a torque tube 5526 and a clamp or fan gear 5528 is shown in accordance with some embodiments.

[0154] Fig.55B An exemplary process 5500 for fixture and fan gear detection according to some embodiments is shown. Some embodiments obtain (5502) an image in a perspective or top view (an example of which is shown in Fig.55A), converting (5504) the image to grayscale, denoising the grayscale image and applying (5506) a bilateral filter, eroding and thresholding (5508) the image, blurring and applying (5516) canny edge detection, dilating (5514) the edges, finding (5514) outer contours that exceed a certain length (e.g., based on the length of the panel geometry, with at least 50% of the width and length as a reference), and / or drawing (5516) a convex hull. Some embodiments test (5518) the convex hull in orthogonal directions. If multiple convex hulls are created and orthogonality is observed, some embodiments mark (5520) the feature detection. Some embodiments indicate (5522) that an object, such as a clamp or fan gear, was detected.

[0155] Fig.56 An exemplary application 500 of a centroid method for estimating the center of a solar panel in accordance with some embodiments is shown. An image of the panel is obtained, converted to pixel coordinates, and the pixel density is calculated from the X and Y angles. Given the density of these pixels, the center or relative center can be estimated. Since the centroid always points to the center of the image, the method picks a point that is apparently the center of the image, but it may not be the center of the panel. This is because the image capture may not be symmetrical, for example the panel is skewed. In some cases, there are different poses and the system may see something like Fig.56 , which means that the closer edge of the panel appears longer and the farther edge of the panel appears shorter. This indicates perspective (and therefore the image may be skewed on the torque tube). In this case, the estimated center (coordinates (1999.5, 1125.5) in this example) has an offset from the true geometric center. This provides an incorrect estimate, but the resulting offset is known. Once the system detects perspective, it knows that the angle is skew, which means that the estimate of the center cannot be the true center. Or the geometric center of the panel is misaligned with the center of the image. However, in order to accurately pick and place, the system needs to know the geometric center of the panel. Therefore, when the panel angle is skewed, straightening and pose correction are applied. In addition, the background is also suppressed. These images and poses can be in multiple directions in six degrees of freedom (three rotations and three translations), and either the background must be suppressed so that it does not affect the center of mass, or the system must compensate for the background and make adjustments independent of the background.

[0156] The algorithms described above for picking, placing, and detecting structures (e.g., fixtures and fan gears) are independent of each other. Various embodiments use one or more of the algorithms or a combination thereof for solar panel installation. Some embodiments combine one or more of these methods with conventional methods for solar panel installation.

[0157] The techniques described herein have several advantages over conventional techniques. For example, systems and methods according to the techniques described herein use non-synthetic imaging schemes to obtain recommendations for pick and place, and therefore provide scalability to any terrain and conditions. Closed-loop feedback helps to automatically correct and improve the estimation process. The simulation process is conditionally independent. Some embodiments take into account optical and geometric properties to adapt to changes in the environment and manufacturer specifications. Some embodiments use pixel energy and pixel color density components during the estimation process.

[0158] Exemplary Methods for Autonomous Solar Installations

[0159] Fig.57 is a flow chart of an exemplary method 5700 for autonomous solar installation according to some embodiments. The method includes obtaining (5702) a first image of a solar panel (e.g., solar panel 4414 held in bracket 4410) during an ongoing solar installation. According to some embodiments, the above reference Fig.40 Exemplary systems and methods for obtaining images and / or image processing are described. In some embodiments, the solar panel includes reflective material that causes the solar panel to reflect sunlight, and the image of the solar panel may include dark white spots due to such reflections. In some embodiments, the first image includes a view of the solar panel and a background, examples of which are described above with reference to Fig.51A In this case, the above reference is not used. Fig.56 The centroid method described is used to calculate the center of the solar panel because the centroid method assumes that all pixels are equally important and therefore causes distortion. An analog method, such as method 5700, can be used.

[0160] The method also includes estimating (5704) a plurality of features of the solar panel based on the first image using distance simulation, geometric correction, and angle adjustment. In some embodiments, the plurality of features include interior points, centers, corners, and / or edges of the solar panel. In some embodiments, an initial set of features may be estimated, and other features may be calculated based on the estimation.

[0161] In some embodiments, estimating the plurality of features of the solar panel further comprises: detecting a position of a panel identifier (e.g., a barcode label) based on determining that the panel identifier is visible in the first image; and using the position to estimate the plurality of features of the solar panel. For example, the panel identifier may be placed on an edge at one of the edges of the solar panel. In some cases, the panel identifier, such as a barcode, may be detected based on a cluster of Hough lines. If the density of vertical Hough lines is high in a region, it indicates the presence of a barcode.

[0162] In some embodiments, estimating the multiple features of the solar panel includes: identifying and drawing (e.g., drawing Hough lines on detected edge boundaries) the longest visible horizontal line and the longest visible vertical line representing the edge of the solar panel based on the first image; for each point in the first image that is inside the solar panel in the first image, correspondingly calculating the horizontal and vertical distances from the corresponding point to the longest visible horizontal line and the longest visible vertical line; and identifying a subset of points based on the calculated horizontal and vertical distances, for which: (i) the horizontal distance is greater than a first predetermined percentage (e.g., 45% or a maximum value) of the width of the solar panel; and (ii) the vertical distance is greater than a second predetermined percentage (e.g., 45% or a maximum value) of the height of the solar panel; correspondingly calculating the distance from each point in the point subset to the longest visible horizontal line and the longest visible vertical line. The method comprises the steps of: determining the horizontal and vertical angles of intersection of the solar panel and the vertical angles of intersection of the solar panel; correcting the calculated horizontal and vertical distances based on the calculated horizontal and vertical angles of intersection to obtain corrected horizontal and vertical distances (the range is the difference between the orthogonal and calculated angles); identifying a set of candidate points based on the corrected horizontal and vertical distances, for which: (i) the horizontal distance is greater than a first predetermined percentage (e.g., 48% or a maximum value) of the width of the solar panel; and (ii) the vertical distance is greater than a second predetermined percentage (e.g., 48% or a maximum value) of the height of the solar panel; calculating the centroid of the candidate point set to obtain the center of the solar panel; estimating candidate corners of the solar panel based on the centroid and the candidate point set using the Pythagorean theorem; and correcting the distances of the candidate corners based on the calculated horizontal and vertical angles of intersection to obtain the final corners of the solar panel. Reference above Fig.51A Examples of these steps are described.

[0163] In some embodiments, identification and drawing are performed using the Hough transform. The longest visible horizontal line and the longest visible vertical line are Hough lines. Hough lines are good indicators of geometric feature detection. Both standard and probabilistic Hough processes can be used in estimation. The parameters of these methods can be modified for specific applications. Hough lines are used to detect straight lines in an image. A straight line can be mathematically defined by (i) slope and offset, (ii) angle and radius, and (iii) poles (e.g., the starting and ending coordinates of a line segment). In the standard formula of the Hough transform, the angle and radius vectors are estimated as parameters in a polar coordinate system; this produces two degrees of freedom that can be modified. In the probabilistic formula of the Hough transform, the coordinates of the endpoints are estimated as parameters in a Cartesian coordinate system; this produces four degrees of freedom that can be modified. Other techniques for detecting straight lines can be used instead of the Hough transform technique.

[0164] In some embodiments, the method further includes determining whether the point is inside the solar panel by determining whether a ray originating from the point intersects the longest visible horizontal line and the longest visible vertical line at an odd point.

[0165] In some embodiments, the method further comprises: before estimating the multiple regions / features of the solar panel: obtaining the relative position of the sun in the sky using, for example, solar physics, the altitude and / or azimuth of the sun; estimating the glare based on the estimate of the sun's position; and false coloring the glare using noise removal and image correction algorithms. Glare is characterized by the presence of a high density of white pixels in the image, which will become noise in this particular application. Therefore, noise removal will mean removing this cluster of white pixels with the panel color. Some embodiments use filtering techniques such as Gaussian or wavelet or Kalman or false color for this purpose. Fig.51B 51 is a schematic diagram of an exemplary method 5110 for Hough line estimation with glare according to some embodiments. Image boundaries are represented by markers 5112 and panel boundaries are represented by markers 5114. The edge between corners A and D (line AD) shows glare 5116. If the Hough line detection ratio is not possible or incorrect, as is the case in this example, an energy balance line BC is drawn as a representation of the Hough lines. The energy balance line is the locus of all pixels that share approximately the same energy value in a particular direction. These steps are performed if the solar altitude and azimuth on a particular day indicate that glare is formed. The energy balance line (sometimes called an energy line) is the locus of points along the Hough lines or the direction of the panel width / length where pixels share nearly the same energy value. Some embodiments monitor two conditions: (i) the energy values ​​of the pixels must be relatively close to each other (e.g., within 2%); and (ii) the pixel centers must be collinear in the direction. The starting and ending points of the energy lines are critical. Some embodiments monitor for drops in energy levels between adjacent pixels (e.g., step changes, such as changes greater than 10%). When this drop occurs and the pixels are collinear, it is safe to conclude that glare is no longer present. When the visible Hough lines are supplemented with energy lines, the pixel intensities captured by the energy, the drop in this energy, and / or the collinearity can be used to establish the lines.

[0166] In some embodiments, the method further comprises: before estimating the plurality of regions / features of the solar panel: removing solar glare on the solar panel using image masking and segmentation or filtering or image correction techniques. Fig.36 , 38, 43 and 50 describe exemplary methods for masking and / or segmentation.

[0167] Return to reference Fig.57 The method further includes generating (5706) a first set of control signals based on the estimated features for operating the first robotic controller to pick up the solar panel. In some embodiments, the first set of control signals also causes the first robotic controller to move the picked up solar panel toward the fixed solar panel.

[0168] The method also includes obtaining (5708) a second image of the solar panel when the solar panel is in the perspective view. Fig.56 An example of a perspective view is described.

[0169] The method also includes detecting placement of the solar panel based on the second image by determining whether the solar panel is coplanar with the fixed solar panel and has a predetermined offset (5710). In some embodiments, the solar panel and the fixed solar panel are substantially rectangular in shape. In some embodiments, the solar panel and the fixed solar panel are substantially similar in shape. In some embodiments, the picked solar panel and the fixed solar panel are substantially similar in shape only along one side, and the system can use symmetry for calculations and / or estimates described herein. For example, when there is no glare along the longest visible Hough line, symmetry allows scaling because the panel geometry is regular. The solar panel is rectangular, and therefore the distance and angle measurements are always known for the invisible side. However, when glare partially or completely covers the longest "visible" Hough line, energy lines are then drawn to supplement the geometric lines to estimate the length of the longest Hough line.

[0170] In some embodiments, detecting the placement of the solar panel includes obtaining a third image of the fixed solar panel when the fixed solar panel is in perspective. For example, the image is in perspective when the near edge appears longer than the far edge. Some embodiments record images continuously. In some embodiments, detecting the placement of the solar panel also includes identifying two centers, including: (i) the center of the solar panel when the solar panel is floating based on the second image; and (ii) the center of the fixed solar panel based on the third image. In some embodiments, detecting the placement of the solar panel also includes drawing a line between the two centers based on time; establishing Euler angles and radius vectors at different time points (for example, at each time point). In some embodiments, detecting the placement of the solar panel also includes generating one or more control signals for operating the first robotic controller to move the solar panel so that the five degrees of freedom (relative to the fixed solar panel) are close to a value of 0. In some embodiments, detecting the placement of the solar panel also includes using computer vision (for example, using edge detection, point of interest detection, contour mapping, bounding box methods) to confirm the coplanarity of points and Hough lines of the solar panel and the fixed solar panel. Reference above Fig.52 and 53 Examples of these steps are described.

[0171] The method also includes generating (5712) a second set of control signals based on the detected placement for operating a second robotic controller to align the solar panel with the fixed solar panel. In some embodiments, generating the second set of control signals includes generating control signals to slide the solar panel when the solar panel is floating so that the distance between the two centers is approximately equal to the predetermined safe offset distance plus the width of each panel. The second robotic controller and the first robotic controller can be different controllers or the same controller. In various embodiments, the controller can control the upper robotic arm and / or the lower robotic arm. In some embodiments, generating the second set of control signals includes generating control signals for fixing the solar panel (e.g., by positioning and tightening the clamp).

[0172] In some embodiments, the method further includes detecting supporting mechanical equipment. This may include obtaining a fourth image of the solar panel in a second perspective or in a top view. This may be a perspective different from the other perspectives previously described with reference to step 5708. In some embodiments, detecting supporting mechanical equipment includes converting the fourth image to grayscale to obtain a grayscale image; denoising the grayscale image and applying a bilateral filter to obtain a processed image. In some embodiments, detecting supporting mechanical equipment includes corroding and thresholding the processed image to obtain a candidate image; drawing a convex hull for the candidate image. In some embodiments, detecting supporting mechanical equipment includes determining an outer contour greater than a predetermined length based on the convex hull; and enlarging the edge of the candidate image. In some embodiments, detecting supporting mechanical equipment includes blurring the candidate image and applying Canny edge detection; and testing the convex hull in orthogonal directions. In some embodiments, detecting supporting mechanical equipment includes detecting supporting mechanical equipment based on determining that multiple convex hulls are created and orthogonality is observed. These steps may generally include image processing, image preparation, and / or image adjustment. Reference above Fig.54 and 55 describe examples of these steps.

[0173] In some embodiments, the support mechanism includes a clamp (sometimes referred to as a clamp assembly, e.g., clamp assembly 602, Fig. 6A , 6B) or a fan gear (e.g., fan gear 5402), but other supporting mechanical devices are also contemplated, such as grounding posts, electrical boxes, and electrical connections. In some embodiments, the supporting mechanical device is a clamp, and the method also includes detecting the position of the clamp and assisting an operator in moving the clamp based on the detected placement of the solar panel so as to protect the solar panel from impact with the clamp. For example, providing a notification / alert to the operator (e.g., via an alarm). Pausing operation when a clamp is detected. In some embodiments, the supporting device is a fan gear, and the method also includes detecting the position of the fan gear; and assisting an operator in safely placing the solar panel based on the detected placement of the solar panel so as to protect the solar panel from collision. Reference above Fig.54 and 55 describe examples of these steps.

[0174] In some embodiments, determining whether a solar panel is coplanar with a fixed solar panel and has a predetermined offset includes determining (i) whether the solar panel is in the same plane as the fixed solar panel; and (ii) whether the solar panel and the fixed panel are nearly flush in a lateral direction.

[0175] Fig.58 is a flow chart of another exemplary method 5800 for autonomous solar installation according to some embodiments. The method includes obtaining (5802) a first image of a solar panel (e.g., solar panel 4414) in a staging area (e.g., bracket 4410) using an observation camera (e.g., a camera located in an upper robot). The solar panel can be approximately perpendicular or normal to the observation camera, the ground and / or the staging area. The observation camera can be a visual camera and / or an IR camera. According to some embodiments, the above reference Fig.40 Exemplary systems and methods for obtaining images and / or image processing are described. The method also includes estimating (5804) multiple regions / features of the solar panel based on the first image using at least one of distance simulation, geometric correction, and angle adjustment. According to some embodiments, the above reference Fig.51AAn example of these steps is described. The method also includes generating (5806) a first set of control signals for operating a first robotic controller to pick up a solar panel. The first set of control signals is based on one or more of the estimated multiple regions / features. The method also includes obtaining (5808) a second image of the solar panel when the solar panel is picked up and is in a perspective orientation relative to the observation camera. The method also includes detecting (5810) the orientation of the picked up solar panel in space based on the second image. The method also includes generating (5812) a second set of control signals based on the detected orientation for moving the picked up solar panel to an installation position. The installation position aligns the picked up solar panel with the previously installed solar panel by determining whether the picked up solar panel is coplanar with the previously installed solar panel and has a predetermined offset. Reference above Fig.52 , 53 and 57 describe examples of these steps.

[0176] Embodiments of the present invention have been described above by means of functional building blocks that illustrate implementation methods of specific functions and their relationships. For ease of description, the boundaries of these functional building blocks are arbitrarily defined herein. Alternative boundaries may be defined as long as the specific functions and their relationships are properly performed.

[0177] It will be apparent to those skilled in the art that various modifications and variations can be made in the system for mounting solar panels of the present invention without departing from the spirit or scope of the invention. Therefore, the present invention is intended to cover modifications and variations of the present invention as long as they fall within the scope of the appended claims and their equivalents. It is to be understood that the terms or terminology herein are for descriptive purposes rather than limiting, so that those skilled in the art should interpret the terms or terminology of this specification in accordance with the teachings and guidance described.

[0178] The breadth and scope of the present invention should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

Claims

1. A method for autonomous solar installation, the method comprising: obtaining a first image of a solar panel during an ongoing solar installation; estimating a plurality of features of the solar panel based on the first image using distance simulation, geometric correction, and angle adjustment; generating a first set of control signals based on the estimated features for operating a first robotic controller to pick up the solar panel; obtaining a second image of the solar panel when the solar panel is in perspective view; detecting placement of the solar panel based on the second image by determining whether the solar panel is coplanar with a fixed solar panel and has a predetermined offset from the fixed solar panel; as well as A second set of control signals is generated based on the detected placement for operating a second robotic controller to align the solar panel with the fixed solar panel.

2. The method according to claim 1, wherein: The plurality of features include a center and a plurality of corners of the solar panel.

3. The method of claim 1 , estimating the plurality of characteristics of the solar panel further comprising: Based on determining that the barcode label is visible in the first image: Detecting the position of the barcode label; as well as The locations are used to estimate the plurality of features of the solar panel.

4. The method according to claim 1, wherein: Estimating the plurality of characteristics of the solar panel comprises: Based on the first image, identifying and drawing a longest visible horizontal line and a longest visible vertical line representing an edge of the solar panel; For each point in the first image that is inside the solar panel in the first image, calculating the horizontal distance and the vertical distance from the corresponding point to the longest visible horizontal line and the longest visible vertical line, respectively; and Based on the calculated horizontal distance and vertical distance, identifying a subset of points for which: (i) the horizontal distance is greater than a first predetermined percentage of the width of the solar panel; and (ii) the vertical distance is greater than a second predetermined percentage of the height of the solar panel; Correspondingly calculating the horizontal intersection angle and the vertical intersection angle of each point from the subset of points with the longest visible horizontal line and the longest visible vertical line; Based on the calculated horizontal intersection angle and vertical intersection angle, correct the calculated horizontal distance and vertical distance to obtain a corrected horizontal distance and vertical distance; Based on the corrected horizontal and vertical distances, identifying a set of candidate points for which: (i) the horizontal distance is greater than a first predetermined percentage of the width of the solar panel; and (ii) the vertical distance is greater than a second predetermined percentage of the height of the solar panel; Calculating the centroid of the set of candidate points to obtain the center of the solar panel; estimating candidate corners of the solar panel using the Pythagorean theorem based on the centroid and the set of candidate points; and The distances of the candidate corners are corrected based on the calculated horizontal and vertical intersection angles to obtain a final corner of the solar panel.

5. The method according to claim 4, wherein: The identification and rendering are performed using Hough transform.

6. The method according to claim 4, further comprising: Whether a point is inside the solar panel is determined by determining whether a ray originating from the point intersects the longest visible horizontal line and the longest visible vertical line at an odd point.

7. A method according to any one of the preceding claims, wherein: Detecting the placement of the solar panel includes: obtaining a third image of the fixed solar panel when the fixed solar panel is in the perspective view; identifying two centers, including (i) a center of the solar panel when the solar panel is floating based on the second image and (ii) a center of the fixed solar panel based on the third image; draw a line between said two centers according to time; Establish the Euler angles and radius vector at each time point; generating one or more control signals for operating the first robotic controller to move the solar panel so that the five degrees of freedom approach a value of zero; and Computer vision is used to confirm the coplanarity of the points and Hough lines of the solar panel with the fixed solar panel.

8. The method according to claim 7, wherein: Generating the one or more control signals comprises: A control signal is generated to slide the solar panel when the solar panel is floating so that the distance between the two centers is approximately equal to a predetermined safety offset distance plus the width of each panel.

9. The method according to any one of the preceding claims, further comprising: obtaining a fourth image of the solar panel in a second perspective or top view; Converting the fourth image into grayscale to obtain a grayscale image; Denoising the grayscale image and applying a bilateral filter to obtain a processed image; performing erosion and threshold processing on the processed image to obtain a candidate image; Draw the convex hull of the candidate image; determining an outer contour having a length greater than a predetermined length based on the convex hull; expanding the edge of the candidate image; Blurring the candidate image and applying Canny edge detection; Test the convex hull in orthogonal directions; as well as The support mechanism is detected by creating multiple convex hulls and observing orthogonality.

10. The method according to claim 9, wherein: The supporting mechanical device includes a clamp or a fan gear.

11. The method according to claim 9, wherein: The supporting mechanical device is a clamp, and wherein the method further comprises: detecting the position of the fixture; and Based on the detected placement of the solar panel, an operator is assisted in moving the clamp so as to protect the solar panel from impact by the clamp.

12. The method according to claim 9, wherein: The support device is a fan gear, and wherein the method further comprises: detecting the position of the fan gear; and Based on the detected placement of the solar panel, an operator is assisted in safely placing the solar panel so as to protect the solar panel from collision.

13. A method according to any one of the preceding claims, wherein: The solar panel and the fixed solar panel are substantially rectangular in shape.

14. A method according to any one of the preceding claims, wherein: The solar panel and the fixed solar panel are substantially similar in shape.

15. A method according to any one of the preceding claims, wherein: The solar panel includes a reflective material that causes the solar panel to reflect sunlight, and the first image or the second image includes a dark white spot.

16. A method according to any one of the preceding claims, wherein: The first image includes a view of the solar panel and a background.

17. The method according to any one of the preceding claims, further comprising: Prior to evaluating the plurality of regions / features of the solar panel: Using solar physics, the sun's altitude and azimuth to obtain the sun's relative position in the sky; estimating glare based on an estimate of the sun's position; and The glare is false-colored using a high-fidelity noise removal and image correction algorithm.

18. The method according to any one of the preceding claims, further comprising: Prior to evaluating the plurality of regions / features of the solar panel: Image masking and segmentation or filtering or image correction techniques are used to remove sun glare on the solar panel.

19. A method according to any one of the preceding claims, wherein: The first set of control signals causes the first robotic controller to move the picked-up solar panel toward the fixed solar panel.

20. A method according to any one of the preceding claims, wherein: Determining whether the solar panel is coplanar with the fixed solar panel and has a predetermined offset from the fixed solar panel includes: Determine whether (i) the solar panel is in the same plane as the fixed solar panel; and (ii) the solar panel and the fixed panel are nearly flush in the lateral direction.

21. A method for autonomous solar installation, the method comprising: obtaining a first image of the solar panels in the staging area using a viewing camera; estimating a plurality of regions / features of the solar panel based on the first image using at least one of distance simulation, geometric correction, and angle adjustment; generating a first set of control signals to operate a first robotic controller to pick up the solar panel, wherein the first set of control signals is based on one or more of the estimated plurality of regions / features; obtaining a second image of the solar panel when the solar panel is picked up and in a perspective orientation relative to the viewing camera; detecting an orientation of the picked-up solar panel in space based on the second image; and generating a second set of control signals based on the detected orientation for operating a second robotic controller to move the picked-up solar panel to an installation position, Wherein, the installation position aligns the picked-up solar panel with the previously installed solar panel by determining whether the picked-up solar panel is coplanar with the previously installed solar panel and has a predetermined offset from the previously installed solar panel.

Images