Photovoltaic robot control method and device with embodied visual-motor coupling

Abstract

The present application relates to a kind of photovoltaic robot control method and device with embodiment visual-motion coupling, method includes the following steps: feature extraction is carried out to the photovoltaic installation site image acquisition result obtained by vision sensor, generates photovoltaic panel installation visual data;Step S2, the detection result of mechanical arm gripper mechanical sensor when mechanical arm picks up photovoltaic panel, generates photovoltaic panel installation mechanical data;Step S3, the photovoltaic panel installation visual data and the photovoltaic panel installation mechanical data are carried out space-time alignment, and carry out fusion credibility evaluation;Step S4, according to fusion credibility evaluation pass, utilize the path planning algorithm based on RRT*, planning obtains the movement trajectory that satisfies no obstacle and dynamics feasible;Step S5, according to the movement trajectory generated photovoltaic panel installation instruction, drives photovoltaic robot to carry out photovoltaic panel installation.The present application has the characteristics of high reliability, high adaptability and high precision.

Description

Technical Field

[0001] The present invention relates to the technical field of photovoltaic panel installation, and in particular to a photovoltaic robot control method and device with embodied vision-motion coupling. Background Art

[0002] A photovoltaic robot is a robotic system that can autonomously or semi-autonomously perform tasks in a photovoltaic power station. By integrating multiple sensors and actuators, it can realize intelligent monitoring, maintenance and management of photovoltaic power stations, involving functions such as photovoltaic panel installation, photovoltaic panel dust removal and cleaning, inspection and testing, fault diagnosis and repair.

[0003] Traditional photovoltaic panel robot motion control during operations typically utilizes positioning and navigation methods, such as the SLAM-based photovoltaic robot positioning and navigation method and device provided in Chinese invention patent CN116448094A. This method utilizes environmental point cloud maps and scene images to generate a three-dimensional photovoltaic site map. The robot's obstacle avoidance sensors measure obstacle locations, and the robot uses maps and high-precision GPS or Beidou positioning for on-site navigation, controlling the robot to reach its destination and perform handling and installation operations. This control method relies on pre-built maps and satellite positioning technology, requiring high sensor accuracy and computing resources. It is less adaptable to sudden obstacles or environmental changes, and strong electromagnetic interference or extreme weather can affect sensor performance, leading to positioning drift or data loss. This, in turn, affects the accuracy of installation operations and makes it difficult to meet photovoltaic panel installation requirements. Summary of the Invention

[0004] In order to solve the technical problems existing in the above-mentioned prior art, the purpose of the present invention is to provide an embodied vision-motion coupled photovoltaic robot control method and device, which has the characteristics of high reliability, high adaptability and high precision.

[0005] To achieve the above-mentioned object of the invention, the present invention provides a photovoltaic robot control method with embodied vision-motor coupling, comprising the following steps:

[0006] Step S1: extracting features from the photovoltaic installation site image acquisition results obtained by the visual sensor to generate photovoltaic panel installation visual data;

[0007] Step S2: obtaining detection results of the mechanical sensor on the gripper of the robotic arm when the robotic arm picks up the photovoltaic panel, and generating photovoltaic panel installation mechanical data;

[0008] Step S3: performing spatiotemporal alignment on the photovoltaic panel installation visual data and the photovoltaic panel installation mechanical data, and performing a fusion credibility assessment;

[0009] Step S4, according to the fusion credibility evaluation pass, the path planning algorithm based on RRT* is used to plan a motion trajectory meeting the barrier-free and dynamic feasibility;

[0010] Step S5, according to the obtained motion trajectory, photovoltaic panel installation instructions are generated to drive the photovoltaic robot to install the photovoltaic panel.

[0011] According to one of the technical solutions of the application, further comprising:

[0012] Step S6, error detection is performed on the photovoltaic panel installation result, and error compensation instructions are generated to drive the mechanical arm to perform error correction according to the installation error exceeding the error threshold.

[0013] According to one of the technical solutions of the application, in step S1, the visual sensor includes a 2D camera and a 3D depth camera, the 3D depth camera is a split binocular 3D camera; the 2D camera is used to acquire a two-dimensional image of the photovoltaic panel and the photovoltaic support, and the 3D depth camera is used to obtain a depth image of the photovoltaic panel and the photovoltaic support.

[0014] The photovoltaic panel installation visual data is a photovoltaic panel installation scene model, and at least includes position information and angle information of the photovoltaic support.

[0015] According to one of the technical solutions of the application, in step S2, specifically comprising:

[0016] Step S21, according to the contact of the mechanical arm's gripper with the photovoltaic panel, real-time detection data of the mechanical sensor on the mechanical arm is acquired, and the contact effectiveness of the gripper with the photovoltaic panel is judged;

[0017] When the gripper on the mechanical arm contacts the photovoltaic panel, the mechanical sensor on the gripper obtains detection data F, and when the detection data F satisfies the following conditions, it is determined that the gripper and the photovoltaic panel are in effective contact:

[0018] The detection data |F| is greater than or equal to 1N, and the duration T of the detection data F is greater than 10ms;

[0019] Step S22, according to the effective contact of the gripper with the photovoltaic panel, the gripper clamps the photovoltaic panel, and real-time detection data of the mechanical sensor on the mechanical arm is acquired, including the detection data F of the mechanical sensor and the duration T of the detection data F.

[0020] According to one of the technical solutions of the application, in step S3, specifically comprising:

[0021] Step S31, the obtained photovoltaic panel installation scene model and the detection data of the mechanical sensor are spatio-temporally aligned;

[0022] Step S32: performing fusion credibility assessment on the spatiotemporally aligned data, the fusion credibility assessment including visual credibility assessment and mechanical credibility assessment;

[0023] Step S33: Compare the visual confidence and the mechanical confidence with the visual confidence threshold and the mechanical confidence threshold respectively. If the confidence is greater than the threshold, it is determined to be passed, otherwise it is failed. If both the visual confidence and the mechanical confidence are passed, execute step S4. If only one of the visual confidence and the mechanical confidence is passed, return to step S1 according to the failure of the visual confidence, and return to step S2 according to the failure of the mechanical confidence, re-acquire the visual data or re-pick up the photovoltaic panel. If both fail, control the robot to stop and notify the staff to check.

[0024] According to a technical solution of the present invention, the calculation method of the mechanical confidence is as follows:

[0025] B=b1*f+b2*t

[0026] Among them, B is the mechanical confidence index, b1 and b2 are coefficients, f is the force investigation coefficient of the mechanical sensor, which is equal to f=F / F0, and F0 is the reference value of the photovoltaic panel mechanical sensor; t is equal to the duration investigation coefficient, which is equal to t=T / T0, and T0 is the duration reference value.

[0027] According to a technical solution of the present invention, in step S4, the steps specifically include:

[0028] Acquiring environmental data and dynamic obstacle motion information, wherein the environmental data includes a target installation position of the photovoltaic panel, a current position of the fixture, and a photovoltaic panel installation scenario model;

[0029] Define the workspace, cost function, and configure RRT tree parameters;

[0030] The target bias sampling strategy is adopted to sample the working area with a fixed probability, and the path planning is performed by the RTT* algorithm. The termination condition of the RTT* algorithm is

[0031]

[0032] According to a technical solution of the present invention, defining a workspace includes:

[0033] Define the configuration space: map the 7-DOF joint space of the robot arm to the SE(3) workspace;

[0034] Define safe operation areas: Use Kalman filter to predict motion trajectories based on dynamic obstacle motion information; Based on the photovoltaic panel installation scenario model and motion trajectory prediction results, define static obstacle installation operation areas and dynamic obstacle installation operation areas according to the minimum static safety distance and minimum dynamic safety distance.

[0035] The dynamic obstacle installation operation area is obtained based on the dynamic obstacle motion trajectory, which passes through: the minimum static safety distance from the static obstacle;

[0036] According to a technical solution of the present invention, the cost function is set to:

[0037] f(n)=α*L+β*E+γ*S

[0038] Among them, L represents the path length index, E represents the energy consumption index, and S represents the safety margin index;

[0039] The path length index L is calculated as follows:

[0040]

[0041] Where n is the total number of discrete pose transformation matrices in the path, Ti∈SE(3) represents the manipulator pose, and se(3) is the Lie algebra space norm.

[0042] The calculation method of energy consumption index E is:

[0043]

[0044] Among them, t0 and t f are the start time and end time of the path, τ j is the joint torque, is the joint angular velocity, k=0.1 is the friction coefficient.

[0045] The calculation method of safety margin index S is:

[0046]

[0047] Where m is the number of obstacles in the environment, d i is the distance between the path point and the nearest obstacle, v i is the relative speed of the dynamic obstacle, and σ=0.2m is the attenuation coefficient.

[0048] According to one aspect of the present invention, there is provided a photovoltaic robot control device with embodied vision-motor coupling, which is used to implement the above-mentioned control method, comprising:

[0049] A visual data processing module is electrically connected to the visual sensor and is used to extract features from the image data acquired by the visual sensor to obtain a photovoltaic panel installation scene model and photovoltaic panel target position information;

[0050] A mechanical data processing module is electrically connected to the mechanical sensor and is used to obtain and process detection data of the mechanical sensor;

[0051] Confidence assessment module, used for spatiotemporal alignment and fusion credibility assessment of visual data and mechanical data;

[0052] Path planning module, used for path planning based on RRT* path planning algorithm;

[0053] The instruction output module is electrically connected to the visual sensor, mechanical sensor, mobile device and robotic arm driving device, and is used to output control instructions to drive the visual sensor, mechanical sensor, mobile device and robotic arm driving device to operate.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] This paper proposes a photovoltaic robot control method and device with embodied vision-action coupling. This method performs spatiotemporal alignment based on visual and mechanical data acquired by visual and mechanical sensors, and utilizes the RRT* path planning algorithm to generate an obstacle-free planned path that meets dynamic requirements. This constructs an embodied control architecture integrating "perception-cognition-action." The control process does not require the use of obstacle avoidance sensors and navigation equipment, nor does it rely on a global map of the installation site. This effectively improves the accuracy and reliability of the photovoltaic robot's control, making it suitable for the operational requirements of complex photovoltaic installation sites.

[0056] The present invention utilizes a split binocular 3D camera for multi-scene recognition. By combining binocular 3D depth images with 2D planar images, the system achieves precise object recognition and positioning. By aligning the visual and mechanical data in time and space, a unified coordinate system is established, improving the reliability and spatial and temporal consistency of feature extraction and command output.

[0057] In the present invention, the RRT* path planning algorithm quickly responds to environmental changes by expanding the tree structure in real time, without relying on the global map. It fully considers the scheduling of static and dynamic obstacles in the workspace, and takes into account the safety and resource utilization of the multi-dimensional robotic arm. The definition and parameter configuration of the RRT* path planning algorithm are suitable for dynamic obstacle scenarios and the high-dimensional motion space of photovoltaic panel installation robots with multi-joint robotic arms. It significantly improves the safety, energy efficiency and efficiency of path planning, making the planned path more practical, economical and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be derived from these drawings without inventive effort.

[0059] Figure 1 Schematically showing a flow chart of a photovoltaic robot control method with embodied vision-action coupling provided in an embodiment of the present invention;

[0060] Figure 2 A schematic diagram showing the working principle of the photovoltaic robot control method with embodied vision-action coupling provided in an embodiment of the present invention;

[0061] Figure 3 The figure schematically shows the structure of a photovoltaic robot control device with embodied vision-action coupling according to an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The description of the embodiments in this specification should be combined with the corresponding drawings, which should be considered a complete part of this specification. In the drawings, the shapes and thicknesses of the embodiments may be exaggerated and indicated for simplicity or convenience. Furthermore, the various structural components in the drawings will be described separately. It is worth noting that components not shown in the drawings or not described in words are known to those of ordinary skill in the art.

[0063] The description of the embodiments herein and any references to directions and orientations are for ease of description only and are not to be construed as limiting the scope of the present invention. The following description of the preferred embodiments may involve combinations of features, which may exist independently or in combination. The present invention is not specifically limited to the preferred embodiments. The scope of the present invention is defined by the claims.

[0064] like Figure 1 and Figure 2 As shown, the present invention provides an embodied vision-motion coupling photovoltaic robot control method, which is applied to the photovoltaic robot for photovoltaic panel installation. The photovoltaic robot includes a visual sensor, a robotic arm, a mobile device, and a central control module. The robotic arm is provided with a gripper and a drive device electrically connected to the central control module. The gripper is provided with a mechanical sensor electrically connected to the central control module. The visual sensor is connected to the central control module. The control method includes the following steps:

[0065] Step S1: extracting features from the photovoltaic installation site image acquisition results obtained by the visual sensor to generate photovoltaic panel installation visual data; the photovoltaic panel installation visual data is a photovoltaic panel installation scene model, including the position and angle of the photovoltaic bracket;

[0066] Step S2: obtaining detection results of the mechanical sensor on the gripper of the robotic arm when the robotic arm picks up the photovoltaic panel, and generating photovoltaic panel installation mechanical data;

[0067] The position of the photovoltaic panel is obtained through the visual sensor, and the robotic arm is driven to pick up the photovoltaic panel, and the picking feedback result is obtained through the mechanical sensor;

[0068] Step S3: performing spatiotemporal alignment on the photovoltaic panel installation visual data and the photovoltaic panel installation mechanical data, and performing a fusion credibility assessment;

[0069] Step S4: Based on the fusion credibility evaluation, a path planning algorithm based on RRT* is used to plan a motion trajectory that satisfies obstacle-free and dynamic feasibility.

[0070] Step S5: generating photovoltaic panel installation instructions according to the obtained motion trajectory, and driving the photovoltaic robot to install the photovoltaic panels;

[0071] Step S6: Perform error detection on the photovoltaic panel installation result, and generate an error compensation instruction based on the installation error exceeding the error threshold, and drive the robotic arm to perform error correction.

[0072] The present invention constructs a photovoltaic panel installation scene model by using images obtained by visual sensors to obtain the installation position and angle of the photovoltaic panels. According to the photovoltaic panel installation scene model, a path planning algorithm based on RRT* is used to obtain a motion trajectory that meets the requirements of obstacle-free and dynamically feasible motion. The reliability of the photovoltaic robot installation operation is improved, and there is no need to rely on satellite navigation equipment and obstacle avoidance sensors during the installation process, which improves the operational flexibility and adaptability of the photovoltaic panel robot. The reliability of the robot arm operation is improved by performing spatiotemporal alignment and fusion credibility evaluation on the photovoltaic installation scene model and the feedback results obtained by the mechanical sensor, and the accuracy of the robot's installation operation is improved.

[0073] In an embodiment of the present invention, in step S1, the visual sensor includes a 2D camera and a 3D depth-of-field camera, the 2D camera is used to obtain two-dimensional images of the photovoltaic panels and photovoltaic brackets, and the 3D depth-of-field camera is used to obtain depth-of-field images of the photovoltaic panels and photovoltaic brackets.

[0074] In step S1, constructing photovoltaic panel installation visual data specifically includes:

[0075] Step S11: extract edge features from the two-dimensional image acquired by the 2D camera to obtain edge features of the photovoltaic panel and the photovoltaic bracket;

[0076] Step S12: obtaining point cloud data of the photovoltaic panel and the photovoltaic bracket from the depth-of-field image acquired by the 3D depth-of-field camera;

[0077] Step S13: performing feature matching on the point cloud data according to the edge features to construct a photovoltaic panel installation scene model.

[0078] In the step S2, it specifically includes:

[0079] Step S21: Based on the contact between the gripper of the robotic arm and the photovoltaic panel, the detection data of the mechanical sensor on the robotic arm is obtained in real time to determine the contact effectiveness between the gripper and the photovoltaic panel;

[0080] When the gripper on the robotic arm contacts the photovoltaic panel, the mechanical sensor on the gripper obtains a detection reading F. When the detection reading F meets the following conditions, it is determined that the gripper and the photovoltaic panel are in effective contact:

[0081] The detection reading |F| ≥ 1N, and the duration T of the detection reading F is greater than 10ms;

[0082] Step S22: Based on the effective contact between the gripper and the photovoltaic panel and the gripper clamping the photovoltaic panel, the detection data of the mechanical sensor on the robotic arm is obtained in real time, including the detection reading F of the mechanical sensor and the duration T of the detection reading F.

[0083] In the step S3, it specifically includes:

[0084] Step S31: performing spatiotemporal alignment on the obtained photovoltaic panel installation scene model and the detection readings of the mechanical sensor;

[0085] Step S32: performing fusion credibility assessment on the spatiotemporally aligned data, the fusion credibility assessment including visual credibility assessment and mechanical credibility assessment;

[0086] Among them, the visual confidence is used to evaluate the reliability of the generated photovoltaic panel installation scene model, which is obtained by evaluating the results of feature extraction. For example, when using target detection algorithms such as YOLO for feature extraction, the confidence comprehensively considers the object existence rate and bounding box matching. In this embodiment, when the visual confidence is greater than 0.7, the photovoltaic panel installation scene module is considered reliable and can be used for subsequent path planning.

[0087] Mechanical confidence is used to evaluate the reliability of the robotic arm picking up the photovoltaic panel. The calculation method of mechanical confidence is as follows:

[0088] B=b1*f+b2*t

[0089] Wherein, B is the mechanical confidence index, b1 and b2 are coefficients, f is the force investigation coefficient of the mechanical sensor, which is equal to f = F / F0, and F0 is the reference value of the photovoltaic panel mechanical sensor; t is the duration investigation coefficient, which is equal to t = T / T0, and T0 is the duration reference value;

[0090] Step S33: Compare the visual confidence and the mechanical confidence with the visual confidence threshold and the mechanical confidence threshold respectively. If the confidence is greater than the threshold, it is determined to be passed, otherwise it is failed. If both the visual confidence and the mechanical confidence are passed, execute step S4. If only one of the visual confidence and the mechanical confidence is passed, return to step S1 according to the failure of the visual confidence, and return to step S2 according to the failure of the mechanical confidence, re-acquire the visual data or re-pick up the photovoltaic panel. If both fail, control the robot to stop and notify the staff to check.

[0091] In step S4, it specifically includes:

[0092] Step S41: Acquire environmental data and dynamic obstacle motion information. The environmental data includes the target installation position of the photovoltaic panel, the current position of the fixture, and the photovoltaic panel installation scenario model. The target installation position of the photovoltaic panel is acquired in real time using a visual sensor. The current position of the fixture is fed back by the robotic arm joint encoder and converted to a global coordinate system using a hand-eye calibration matrix. Dynamic obstacle motion information includes the position and velocity of the dynamic obstacle.

[0093] Step S42, initialization: set the starting point and end point of the path, initialize the RRT tree, and use the starting point as the root node; define the workspace and cost function, and configure the RRT tree parameters;

[0094] Defining the workspace includes defining the configuration space and defining the safe operation area;

[0095] The configuration space is defined as: mapping the 7-DOF joint space of the manipulator to the SE(3) workspace;

[0096] The safe operating area is defined based on the photovoltaic panel installation scenario model. The safe operating area includes the static obstacle safe operating area and the dynamic obstacle safe operating area. The static obstacle safe operating area should meet the minimum static safety distance from the static obstacle, and the dynamic obstacle safe operating area should meet the minimum dynamic safety distance from the dynamic obstacle. The minimum static safety distance is greater than the minimum dynamic safety distance.

[0097] In this embodiment, the minimum safe distance for static obstacles (such as support structures) is set to 5 cm; the minimum safe distance for dynamic obstacles (such as mobile transportation equipment) is increased to 10 cm. The motion trajectory of dynamic obstacles is predicted using a Kalman filter.

[0098] In this embodiment, the RRT tree parameters are configured as follows:

[0099] Set the maximum number of iterations Nmax = 1000, the initial step size ΔS = 0.3m, the target tolerance ∈ = 0.05m, and introduce a target bias sampling strategy to directly sample the target area with a probability of 15% to accelerate convergence;

[0100] Among them, the cost function is set to:

[0101] f(n)=α*L+β*E+γ*S

[0102] Among them, L represents the path length index, E represents the energy consumption index, and S represents the safety margin index;

[0103] The path length index L is calculated as follows:

[0104]

[0105] Where n is the total number of discrete pose transformation matrices in the path, Ti∈SE(3) represents the manipulator pose, and se(3) is the Lie algebra space norm.

[0106] The calculation method of energy consumption index E is:

[0107]

[0108] Among them, t0 and t f are the start time and end time of the path, τ j is the joint torque, is the joint angular velocity, k=0.1 is the friction coefficient.

[0109] The calculation method of safety margin index S is:

[0110]

[0111] Where m is the number of obstacles in the environment, d i is the distance between the path point and the nearest obstacle, v i is the relative velocity of the dynamic obstacle, σ=0.2m is the attenuation coefficient;

[0112] Step S42, sampling and expansion: randomly sampling new nodes in the state space with a probability of 15%; generating a series of candidate nodes within a certain time and speed range starting from the current node according to the dynamic window method; calculating the evaluation value of each candidate node through the cost function;

[0113] Step S43: Select the optimal node: select the node with the best evaluation value from the candidate nodes as a child node and add it to the RRT tree; update the parent node information of the node to facilitate subsequent path backtracking.

[0114] Step S44, conflict detection and resolution: Check whether the new node conflicts with an obstacle or an existing node.

[0115] If a conflict occurs, the node is abandoned and resampled.

[0116] Among them, conflicts can be avoided by dynamically adjusting the parameters of the dynamic window method.

[0117] Step S45: Path backtracking and optimization: When the end point is found or the preset search depth is reached, backtracking is started from the end point, selecting the parent node with the best evaluation value until returning to the starting point. The backtracking path is smoothed to reduce path tortuosity and sudden changes.

[0118] Step S46, algorithm termination condition:

[0119] The algorithm terminates when a path that meets the conditions is found or the preset maximum number of iterations is reached.

[0120] The algorithm termination condition is set as:

[0121]

[0122] In the step S6, it specifically includes:

[0123] Step S61: Obtaining the installation result information of the photovoltaic panels, including the locations of the photovoltaic panels after installation;

[0124] The image data of photovoltaic panels after installation is obtained through visual sensors, and feature extraction is performed on the collected images to obtain the position and angle of the photovoltaic panels after installation;

[0125] Step S62: Compare the installed position of the photovoltaic panel with the target installation position of the photovoltaic panel and calculate the posture error. If the posture error is less than 0.1 mm, the installation task is terminated; otherwise, step S63 is executed.

[0126] Step S63: Generate a control instruction based on the posture error;

[0127] Step S64: According to the control instruction, the robot arm is controlled to move to the location of the photovoltaic panel, clamp the photovoltaic panel, and perform a contact validity test. If valid, the control instruction is executed and the process returns to step S61.

[0128] like Figure 3 As shown, the present invention provides an embodied vision-motion coupled photovoltaic robot control device for implementing the above control method, including:

[0129] A visual data processing module is electrically connected to the visual sensor and is used to extract features from the image data acquired by the visual sensor to obtain a photovoltaic panel installation scene model and photovoltaic panel target position information;

[0130] A mechanical data processing module is electrically connected to the mechanical sensor and is used to obtain and process detection data of the mechanical sensor;

[0131] Confidence assessment module, used for spatiotemporal alignment and fusion credibility assessment of visual data and mechanical data;

[0132] Path planning module, used for path planning based on RRT* path planning algorithm;

[0133] The instruction output module is electrically connected to the visual sensor, mechanical sensor, mobile device and robotic arm driving device, and is used to output control instructions to drive the visual sensor, mechanical sensor, mobile device and robotic arm driving device to operate.

[0134] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements but also other elements not explicitly listed, or also includes elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0135] Finally, it should be noted that the above is a preferred embodiment of the present invention. It should be noted that although the preferred embodiment of the present invention has been described, it is clear that those skilled in the art, once they understand the basic inventive concept of the present invention, can make various improvements and modifications without departing from the principles of the present invention. Such improvements and modifications should also be considered as within the scope of protection of the present invention. Therefore, the appended claims are intended to be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the embodiments of the present invention.

Claims

1. A photovoltaic robot control method with embodied vision-motion coupling, characterized in that: The following steps are involved: Step S1: extracting features from the photovoltaic installation site image acquisition results obtained by the visual sensor to generate photovoltaic panel installation visual data; Step S2: obtaining detection results of the mechanical sensor on the gripper of the robotic arm when the robotic arm picks up the photovoltaic panel, and generating photovoltaic panel installation mechanical data; Step S3: performing spatiotemporal alignment on the photovoltaic panel installation visual data and the photovoltaic panel installation mechanical data, and performing a fusion credibility assessment; Step S4: Based on the fusion credibility evaluation, a path planning algorithm based on RRT* is used to plan a motion trajectory that satisfies obstacle-free and dynamic feasibility. Step S5: generating photovoltaic panel installation instructions according to the obtained motion trajectory, and driving the photovoltaic robot to install the photovoltaic panels; In step S3, it specifically includes: Step S31: performing spatiotemporal alignment on the obtained photovoltaic panel installation scene model and the detection readings of the mechanical sensor; Step S32: performing fusion credibility assessment on the spatiotemporally aligned data, the fusion credibility assessment including visual credibility assessment and mechanical credibility assessment; Step S33: Compare the visual confidence and the mechanical confidence with the visual confidence threshold and the mechanical confidence threshold, respectively. If the confidence is greater than the threshold, it is determined to be passed; otherwise, it is determined to be failed. If both the visual confidence and the mechanical confidence are passed, execute step S4. If only one of the visual confidence and the mechanical confidence is passed, return to step S1 based on the failure of the visual confidence, and return to step S2 based on the failure of the mechanical confidence, reacquire the visual data or re-pick up the photovoltaic panel. If both fail, control the robot to stop and notify the staff to conduct an inspection. The calculation method of the mechanical confidence is as follows: B=b1*f+b2*t Among them, B is the mechanical confidence index, b1 and b2 are coefficients, f is the force investigation coefficient of the mechanical sensor, which is equal to f=F / F0, F is the detection reading of the mechanical sensor, and F0 is the reference value of the photovoltaic panel mechanical sensor; t is equal to the duration investigation coefficient, which is equal to t=T / T0, T is the duration of the detection reading F, and T0 is the duration reference value.

2. The photovoltaic robot control method with embodied vision-motion coupling according to claim 1, characterized in that: Also includes: Step S6: Perform error detection on the photovoltaic panel installation result, and generate an error compensation instruction based on the installation error exceeding the error threshold, and drive the robotic arm to perform error correction.

3. The photovoltaic robot control method with embodied vision-motion coupling according to claim 1, characterized in that: In step S1, the visual sensor includes a 2D camera and a 3D depth-of-field camera. The 2D camera is used to obtain a two-dimensional image of the photovoltaic panel and the photovoltaic bracket, and the 3D depth-of-field camera is used to obtain a depth-of-field image of the photovoltaic panel and the photovoltaic bracket. The photovoltaic panel installation visual data is a photovoltaic panel installation scene model, which at least includes position information and angle information of the photovoltaic bracket.

4. The photovoltaic robot control method with embodied vision-motion coupling according to claim 3, characterized in that: In the step S2, it specifically includes: Step S21: Based on the contact between the gripper of the robotic arm and the photovoltaic panel, the detection data of the mechanical sensor on the robotic arm is obtained in real time to determine the contact effectiveness between the gripper and the photovoltaic panel; When the gripper on the robotic arm contacts the photovoltaic panel, the mechanical sensor on the gripper obtains a detection reading F. When the detection reading F meets the following conditions, it is determined that the gripper and the photovoltaic panel are in effective contact: The detection reading |F| ≥ 1N, and the duration T of the detection reading F is greater than 10ms; Step S22: Based on the effective contact between the gripper and the photovoltaic panel and the gripper clamping the photovoltaic panel, the detection data of the mechanical sensor on the robotic arm is obtained in real time, including the detection reading F of the mechanical sensor and the duration T of the detection reading F.

5. The photovoltaic robot control method with embodied vision-motion coupling according to claim 1, characterized in that: In the step S4, it specifically includes: Acquiring environmental data and dynamic obstacle motion information, wherein the environmental data includes a target installation position of the photovoltaic panel, a current position of the fixture, and a photovoltaic panel installation scenario model; Define the workspace, cost function, and configure RRT tree parameters; A target bias sampling strategy is adopted to sample the working area with a fixed probability, and path planning is performed using the RTT* algorithm. The termination conditions of the RTT* algorithm include the static safety distance termination condition, the joint torque margin termination condition, and the path maximum curvature termination condition, which are expressed as follows:

6. The photovoltaic robot control method with embodied vision-motion coupling according to claim 5, characterized in that: The definition workspace includes: Define the configuration space: map the 7-DOF joint space of the robot arm to the SE(3) workspace; Define safe operating areas: Use a Kalman filter to predict the motion trajectory of dynamic obstacles based on their motion information. Based on the photovoltaic panel installation scenario model and the motion trajectory prediction results, define the static obstacle installation operation area and the dynamic obstacle installation operation area according to the minimum static safety distance and the minimum dynamic safety distance. The dynamic obstacle installation operation area is obtained according to the dynamic obstacle motion trajectory. The dynamic obstacle motion trajectory passes: the minimum static safety distance from the static obstacle.

7. The photovoltaic robot control method with embodied vision-motion coupling according to claim 5, characterized in that: The cost function is set to: f(n)=α*L+β*E+γ*S Among them, L represents the path length index, E represents the energy consumption index, and S represents the safety margin index; The path length index L is calculated as follows: Where n is the total number of discrete pose transformation matrices in the path, Ti∈SE(3) represents the manipulator pose, and se(3) is the Lie algebra space norm; The calculation method of energy consumption index E is: Among them, t0 and t f are the start time and end time of the path, τ j is the joint torque, is the joint angular velocity, k = 0.1 is the friction coefficient; The calculation method of safety margin index S is: Where m is the number of obstacles in the environment, d i is the distance between the path point and the nearest obstacle, v i is the relative speed of the dynamic obstacle, and σ=0.2m is the attenuation coefficient.

8. An embodied vision-motor coupling photovoltaic robot control device, for implementing the control method according to any one of claims 1 to 7, comprising: A visual data processing module is electrically connected to the visual sensor and is used to extract features from the image data acquired by the visual sensor to obtain a photovoltaic panel installation scene model and photovoltaic panel target position information; A mechanical data processing module is electrically connected to the mechanical sensor and is used to obtain and process detection data of the mechanical sensor; Confidence assessment module, used for spatiotemporal alignment and fusion credibility assessment of visual data and mechanical data; Path planning module, used for path planning based on RRT* path planning algorithm; The instruction output module is electrically connected to the visual sensor, mechanical sensor, mobile device and robotic arm driving device, and is used to output control instructions to drive the visual sensor, mechanical sensor, mobile device and robotic arm driving device to operate.

Images