Photovoltaic robot control method and device with body vision-action coupling

Abstract

The invention relates to a control method and device for a photovoltaic robot with body vision-action coupling, and the method comprises the following steps: carrying out the feature extraction of a photovoltaic installation site image collection result obtained by a vision sensor, and generating the installation vision data of a photovoltaic panel; s2, obtaining a detection result of a mechanical sensor on a mechanical arm clamping jaw when the mechanical arm picks up the photovoltaic panel, and generating photovoltaic panel installation mechanical data; s3, carrying out space-time alignment on the photovoltaic panel installation visual data and the photovoltaic panel installation mechanical data, and carrying out fusion credibility evaluation; s4, according to the fact that the fusion credibility evaluation is passed, a path planning algorithm based on RRT * is utilized, and a barrier-free and dynamics-feasible movement track is planned and obtained; and S5, a photovoltaic panel installation instruction is generated according to the obtained motion trail, and a photovoltaic robot is driven to carry out photovoltaic panel installation. The method has the characteristics of high reliability, high adaptability and high precision.

Description

Technical Field

[0001] The present invention relates to the technical field of photovoltaic panel installation, and particularly to a control method and device for a photovoltaic robot with embodied vision-action coupling. Background Art

[0002] A photovoltaic robot is a robot system that can autonomously or semi-autonomously perform tasks in a photovoltaic power station. By integrating various sensors and actuators, it realizes intelligent monitoring, maintenance, and management of the photovoltaic power station, involving functions such as photovoltaic panel installation, photovoltaic panel dust removal and cleaning, inspection and detection, fault diagnosis and repair, etc.

[0003] In the traditional mobile control during the operation of a photovoltaic panel robot, positioning and navigation methods are usually adopted. For example, the photovoltaic robot positioning and navigation method and device based on SLAM technology provided by Chinese invention patent CN116448094A obtain a three-dimensional photovoltaic site map by using an environmental point cloud map and a scene image, measure the position of obstacles by combining obstacle avoidance sensors installed on the photovoltaic robot, and at the same time combine the map and high-precision GPS or Beidou positioning to achieve on-site navigation, control the photovoltaic robot to reach the destination and perform handling and installation operations. This control method relies on pre-built maps and satellite positioning technology, has high requirements for sensor accuracy and computing resources, has weak adaptability to sudden obstacles or environmental changes, and strong electromagnetic interference or extreme weather affects the performance of sensors, resulting in positioning drift or data loss, thereby affecting the accuracy of installation operations and making it difficult to meet the installation requirements of photovoltaic panels. Summary of the Invention

[0004] To solve the above technical problems existing in the prior art, the purpose of the present invention is to provide a control method and device for a photovoltaic robot with embodied vision-action coupling, which have the characteristics of high reliability, high adaptability, and high precision.

[0005] To achieve the above invention purpose, the present invention provides a control method for a photovoltaic robot with embodied vision-action coupling, including the following steps:

[0006] Step S1: Extract features from the acquisition result of the photovoltaic installation site image obtained by the vision sensor to generate photovoltaic panel installation vision data;

[0007] Step S2: Obtain the detection result of the mechanical sensor on the mechanical arm gripper when the mechanical arm picks up the photovoltaic panel to generate photovoltaic panel installation mechanical data;

[0008] Step S3: Perform spatio-temporal alignment on the photovoltaic panel installation vision data and the photovoltaic panel installation mechanical data, and perform fusion credibility evaluation;

[0009] Step S4: Based on the passing of the fusion credibility assessment, use the RRT*-based path planning algorithm to plan a motion trajectory that meets the requirements of obstacle-free and dynamic feasibility.

[0010] Step S5: Generate a photovoltaic panel installation instruction according to the obtained motion trajectory, and drive the photovoltaic robot to install the photovoltaic panel.

[0011] According to one technical solution of the present invention, it further includes:

[0012] Step S6: Detect the error of the photovoltaic panel installation result, and generate an error compensation instruction according to the installation error exceeding the error threshold, and drive the robotic arm to perform error correction.

[0013] According to one technical solution of the present invention, in step S1, the vision sensor includes a 2D camera and a 3D depth camera, and the 3D depth camera is a split-type binocular 3D camera; the 2D camera is used to obtain the two-dimensional image of the photovoltaic panel and the photovoltaic bracket, and the 3D depth camera is used to obtain the depth image of the photovoltaic panel and the photovoltaic bracket;

[0014] The visual data for photovoltaic panel installation is a photovoltaic panel installation scene model, which at least includes the position information and angle information of the photovoltaic bracket.

[0015] According to one technical solution of the present invention, in the step S2, it specifically includes:

[0016] Step S21: According to the gripper of the robotic arm contacting the photovoltaic panel, obtain the detection data of the force sensor on the robotic arm in real time, and judge the contact effectiveness between the gripper and the photovoltaic panel;

[0017] When the gripper on the robotic arm contacts the photovoltaic panel, the force sensor on the gripper obtains a detection reading F. When the detection reading F meets the following conditions, it is determined that the gripper and the photovoltaic panel are in effective contact:

[0018] The absolute value of the detection reading |F|≥1N, and the duration T of the detection reading F is greater than 10ms;

[0019] Step S22: According to the effective contact between the gripper and the photovoltaic panel and the gripper clamping the photovoltaic panel, obtain the detection data of the force sensor on the robotic arm in real time, including the detection reading F of the force sensor and the duration T of the detection reading F.

[0020] According to one technical solution of the present invention, in step S3, it specifically includes:

[0021] Step S31: Perform spatio-temporal alignment on the obtained photovoltaic panel installation scene model and the detection reading of the force sensor.

[0022] Step S32: Perform a fusion credibility assessment on the data after spatio-temporal alignment. The fusion credibility assessment includes a visual confidence assessment and a mechanical confidence assessment;

[0023] Step S33: Compare the visual confidence and the mechanical confidence with the visual confidence threshold and the mechanical confidence threshold respectively. If the confidence is greater than the threshold, it is determined to pass; otherwise, it fails. If both the visual confidence and the mechanical confidence pass, execute Step S4; if only one of the visual confidence and the mechanical confidence passes, return to Step S1 according to the visual confidence failing, and return to Step S2 according to the mechanical confidence failing to re-acquire visual data or re-pick up the photovoltaic panel; if both fail, control the robot to stop and notify the staff for inspection.

[0024] According to a technical solution of the present invention, the calculation method of the mechanical confidence is as follows:

[0025] B = b1 * f + b2 * t

[0026] Wherein, B is the mechanical confidence index, b1 and b2 are both coefficients, f is the force investigation coefficient of the mechanical sensor, equal to f = F / F0, F0 is the reference value of the photovoltaic panel mechanical sensor; t is equal to the duration investigation coefficient, equal to t = T / T0, T0 is the reference value of the duration.

[0027] According to a technical solution of the present invention, in the said Step S4, it specifically includes:

[0028] Obtain environmental data and dynamic obstacle motion information. The environmental data includes the target installation position of the photovoltaic panel, the current pose of the fixture, and the photovoltaic panel installation scene model;

[0029] Define the workspace, cost function, and configure the RRT tree parameters;

[0030] Adopt the target bias sampling strategy to sample the working area with a fixed probability, and perform path planning through the RTT* algorithm. Among them, the termination condition of the RTT* algorithm is

[0031]

[0032] According to a technical solution of the present invention, the definition of the workspace includes:

[0033] Define the configuration space: Map the 7-degree-of-freedom joint space of the robotic arm to the SE(3) workspace;

[0034] Define the safe operation area: Predict the motion trajectory through the Kalman filter according to the dynamic obstacle motion information; Based on the photovoltaic panel installation scenario model and the motion trajectory prediction result, define the static obstacle installation operation area and the dynamic obstacle installation operation area according to the minimum static safety distance and the minimum dynamic safety distance.

[0035] The dynamic obstacle installation operation area is obtained according to the dynamic obstacle motion trajectory, and the dynamic obstacle motion trajectory is obtained by: the distance from the static obstacle is the minimum static safety distance.

[0036] According to one technical solution of the present invention, the cost function is set as:

[0037] f(n) = α * L + β * E + γ * S

[0038] Wherein, L represents the path length index, E represents the energy consumption index, and S represents the safety margin index;

[0039] The calculation method of the path length index L is:

[0040]

[0041] Wherein, n is the total number of discrete pose transformation matrices in the path, Ti ∈ SE(3) represents the manipulator pose, and se(3) is the Lie algebra space norm.

[0042] The calculation method of the energy consumption index E is:

[0043]

[0044] Wherein, t0 and t f Are the start time and end time of the path respectively, τ j Is the joint torque, Is the joint angular velocity, and k = 0.1 is the friction coefficient.

[0045] The calculation method of the safety margin index S is:

[0046]

[0047] Wherein, m is the number of obstacles in the environment, d i Is the distance between the path point and the nearest obstacle, v i Is the relative velocity of the dynamic obstacle, and σ = 0.2m is the attenuation coefficient.

[0048] According to one aspect of the present invention, a photovoltaic robot control device with embodied vision-action coupling is provided for implementing the above control method, including:

[0049] A visual data processing module, electrically connected to a visual sensor, for extracting features from the image data acquired by the visual sensor to obtain a photovoltaic panel installation scene model and photovoltaic panel target position information;

[0050] A mechanical data processing module, electrically connected to a mechanical sensor, for acquiring and processing the detection data of the mechanical sensor;

[0051] A confidence evaluation module, for performing spatio-temporal alignment of visual data and mechanical data and evaluating the fusion credibility;

[0052] A path planning module, for performing path planning based on the RRT* path planning algorithm;

[0053] An instruction output module, electrically connected to the visual sensor, mechanical sensor, mobile device, and robotic arm driving device, for outputting control instructions to drive the visual sensor, mechanical sensor, mobile device, and robotic arm driving device to act.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] The present invention proposes an embodied vision-action coupling photovoltaic robot control method and device, which performs spatio-temporal alignment according to the visual data and mechanical data acquired by the visual sensor and the mechanical sensor, and uses the RRT* path planning algorithm to generate an obstacle-free planned path that meets the dynamic requirements, constructs an embodied control architecture of "perception-cognition-action", and the control process does not require the use of obstacle avoidance sensors and navigation devices, does not rely on the global map of the installation site, effectively improves the accuracy and reliability of the photovoltaic robot control, and is applicable to the operation requirements of complex photovoltaic installation sites.

[0056] In the present invention, the visual sensor system adopts a split binocular 3D camera, which is suitable for multi-scene recognition. By combining the binocular 3D depth image and the 2D plane image, accurate object recognition and positioning are realized. Through the spatio-temporal alignment of visual data and mechanical data, a unified coordinate system is constructed, improving the reliability and spatio-temporal unity of feature extraction and instruction output.

[0057] In the present invention, the RRT* path planning algorithm quickly responds to environmental changes through a real-time expanding tree structure, does not rely on the global map, fully considers the scheduling of static and dynamic obstacles in the working space, and simultaneously considers the safety and resource utilization of the multi-dimensional robotic arm, defines and configures the parameters of the RRT* path planning algorithm, is suitable for dynamic obstacle scenarios and the high-dimensional motion space of photovoltaic panel installation robots with multi-joint robotic arms, significantly improves the safety, energy efficiency and efficiency of path planning, and makes the planned path more practical, economical and reliable. Description of the Drawings

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0059] Figure 1 Schematically showing the flowchart of the embodied vision-action coupling photovoltaic robot control method provided in the embodiments of the present invention;

[0060] Figure 2 Schematically showing the working principle diagram of the embodied vision-action coupling photovoltaic robot control method provided in the embodiments of the present invention;

[0061] Figure 3 Schematically showing the structural diagram of the embodied vision-action coupling photovoltaic robot control device provided in the embodiments of the present invention. Detailed implementation manners

[0062] The description of the embodiments of this specification should be combined with the corresponding drawings, and the drawings should be regarded as part of the complete specification. In the drawings, the shape or thickness of the embodiments can be enlarged, and simplified or conveniently marked. Furthermore, each part of the structure in the drawings will be described separately. It should be noted that the elements not shown or not described in words in the drawings are in the forms known to those of ordinary skill in the art.

[0063] In the description of the embodiments herein, any reference to directions and orientations is only for convenience of description and should not be construed as any limitation on the protection scope of the present invention. The following description of the preferred embodiments involves combinations of features, which may exist independently or in combination. The present invention is not particularly limited to the preferred embodiments. The scope of the present invention is defined by the claims.

[0064] As Figure 1 and Figure 2 shown, a kind of embodied vision-action coupling photovoltaic robot control method provided by the present invention is applied to the photovoltaic robot for installing photovoltaic panels. The photovoltaic robot includes a vision sensor, a robotic arm, a mobile device, and a central control module. A gripper and a driving device electrically connected to the central control module are arranged on the robotic arm, and a mechanical sensor electrically connected to the central control module is arranged on the gripper. The vision sensor is connected to the central control module point-to-point. The control method includes the following steps:

[0065] Step S1: Extract features from the image acquisition results of the photovoltaic installation site obtained by the vision sensor to generate visual data for photovoltaic panel installation; the visual data for photovoltaic panel installation is a photovoltaic panel installation scene model, including the position and angle of the photovoltaic support.

[0066] Step S2: Obtain the detection results of the mechanical sensor on the robotic arm gripper when the robotic arm picks up the photovoltaic panel to generate mechanical data for photovoltaic panel installation.

[0067] Obtain the position of the photovoltaic panel through the vision sensor, drive the robotic arm to pick up the photovoltaic panel, and obtain the picking feedback result through the mechanical sensor.

[0068] Step S3: Align the visual data for photovoltaic panel installation and the mechanical data for photovoltaic panel installation in space and time, and perform a fusion credibility assessment.

[0069] Step S4: According to the passing of the fusion credibility assessment, use the RRT*-based path planning algorithm to plan a motion trajectory that meets the requirements of obstacle-free and dynamic feasibility.

[0070] Step S5: Generate a photovoltaic panel installation instruction according to the obtained motion trajectory, and drive the photovoltaic robot to install the photovoltaic panel.

[0071] Step S6: Detect the error of the photovoltaic panel installation result, and generate an error compensation instruction according to the installation error exceeding the error threshold, and drive the robotic arm to perform error correction.

[0072] In the present invention, by using the image obtained by the vision sensor to construct a photovoltaic panel installation scene model, obtaining the installation position and angle of the photovoltaic panel, and using the RRT*-based path planning algorithm according to the photovoltaic panel installation scene model to obtain a motion trajectory that meets the requirements of obstacle-free and dynamic feasibility, the reliability of the photovoltaic robot installation operation is improved, and there is no need to rely on satellite navigation equipment and obstacle avoidance sensors during the installation process, improving the operation flexibility and adaptability of the photovoltaic panel robot; by performing space-time alignment and fusion credibility assessment on the photovoltaic installation scene model and the feedback results obtained by the mechanical sensor, the reliability of the robotic arm operation is improved, and the installation accuracy of the robot is improved.

[0073] In an embodiment of the present invention, in step S1, the vision sensor includes a 2D camera and a 3D depth camera. The 2D camera is used to obtain the two-dimensional image of the photovoltaic panel and the photovoltaic support, and the 3D depth camera is used to obtain the depth image of the photovoltaic panel and the photovoltaic support.

[0074] In step S1, specifically constructing the visual data for photovoltaic panel installation includes:

[0075] Step S11: Extract edge features from the two-dimensional image obtained by the 2D camera to obtain the edge features of the photovoltaic panel and the photovoltaic support.

[0076] Step S12: Obtain the point cloud data of the photovoltaic panel and the photovoltaic support from the depth image acquired by the 3D depth camera;

[0077] Step S13: Perform feature matching on the point cloud data according to the edge features to construct a photovoltaic panel installation scene model.

[0078] In the said Step S2, it specifically includes:

[0079] Step S21: When the gripper of the robotic arm contacts the photovoltaic panel, obtain the detection data of the force sensor on the robotic arm in real time, and judge the contact effectiveness between the gripper and the photovoltaic panel;

[0080] When the gripper on the robotic arm contacts the photovoltaic panel, the force sensor on the gripper obtains a detection reading F. When the detection reading F meets the following conditions, it is determined that the gripper and the photovoltaic panel are in effective contact:

[0081] The absolute value of the detection reading |F| ≥ 1N, and the duration T of the detection reading F is greater than 10 ms;

[0082] Step S22: When the gripper and the photovoltaic panel are in effective contact and the gripper clamps the photovoltaic panel, obtain the detection data of the force sensor on the robotic arm in real time, including the detection reading F of the force sensor and the duration T of the detection reading F.

[0083] In the said Step S3, it specifically includes:

[0084] Step S31: Perform spatio-temporal alignment on the obtained photovoltaic panel installation scene model and the detection reading of the force sensor;

[0085] Step S32: Perform a fusion credibility assessment on the data after spatio-temporal alignment. The fusion credibility assessment includes a visual confidence assessment and a mechanical confidence assessment;

[0086] Among them, the visual confidence is used to evaluate the reliability of the generated photovoltaic panel installation scene model, and is obtained by evaluating the result of feature extraction. For example, when using object detection algorithms such as YOLO for feature extraction, the confidence comprehensively considers the object existence rate and the bounding box matching degree. In this embodiment, when the visual confidence > 0.7, it is considered that the photovoltaic panel installation scene module is reliable and can be used for subsequent path planning.

[0087] The mechanical confidence is used to evaluate the reliability of the robotic arm picking up the photovoltaic panel. The calculation method of the mechanical confidence is as follows:

[0088] B = b1 * f + b2 * t

[0089] Wherein, B is the mechanical confidence index, both b1 and b2 are coefficients, f is the inspection coefficient of the force of the mechanical sensor, and f = F / F0, where F0 is the reference value of the mechanical sensor of the photovoltaic panel; t is equal to the inspection coefficient of the duration, and t = T / T0, where T0 is the reference value of the duration;

[0090] Step S33: Compare the visual confidence and the mechanical confidence with the visual confidence threshold and the mechanical confidence threshold respectively. If the confidence is greater than the threshold, it is determined to pass; otherwise, it fails. If both the visual confidence and the mechanical confidence pass, execute Step S4; if only one of the visual confidence and the mechanical confidence passes, return to Step S1 according to the visual confidence failure, and return to Step S2 according to the mechanical confidence failure to re-acquire visual data or re-pick up the photovoltaic panel; if both fail, control the robot to stop and notify the staff for inspection.

[0091] In Step S4, it specifically includes:

[0092] Step S41: Obtain environmental data and dynamic obstacle movement information. The environmental data includes the target installation position of the photovoltaic panel, the current pose of the fixture, and the photovoltaic panel installation scene model. The target installation position of the photovoltaic panel is obtained in real time through a visual sensor, and the current pose of the fixture is fed back by the robotic arm joint encoder and converted to the global coordinate system in combination with the hand-eye calibration matrix. The dynamic obstacle movement information includes the position information and speed information of the dynamic obstacle.

[0093] Step S42: Initialization: Set the start point and end point of the path, initialize the RRT tree, and use the start point as the root node; Define the workspace and the cost function, and configure the RRT tree parameters;

[0094] Defining the workspace includes defining the configuration space and defining the safe operation area;

[0095] The configuration space is defined as: mapping the 7-degree-of-freedom joint space of the robotic arm to the SE(3) workspace;

[0096] Based on the photovoltaic panel installation scene model, define the safe operation area. The safe operation area includes the static obstacle safe operation area and the dynamic obstacle safe operation area. The static obstacle safe operation area should satisfy the distance from the static obstacle to meet the minimum static safety distance, and the dynamic obstacle safe operation area should satisfy the distance from the dynamic obstacle to meet the minimum dynamic safety distance; the minimum static safety distance is greater than the minimum dynamic safety distance.

[0097] In this embodiment, the minimum safety distance of the static obstacle (such as the bracket structure) is set to 5 cm; the minimum safety distance of the dynamic obstacle (such as the mobile transportation equipment) is increased to 10 cm. The movement trajectory of the dynamic obstacle is predicted through a Kalman filter.

[0098] In this embodiment, the RRT tree parameter configuration is as follows:

[0099] Set the maximum number of iterations Nmax = 1000, the initial step size ΔS = 0.3m, the target tolerance ∈ = 0.05m, and introduce a target bias sampling strategy to directly sample the target area with a probability of 15% to accelerate convergence;

[0100] Among them, the cost function is set as:

[0101] f(n) = α * L + β * E + γ * S

[0102] Among them, L represents the path length index, E represents the energy consumption index, and S represents the safety margin index;

[0103] The calculation method of the path length index L is:

[0104]

[0105] Among them, n is the total number of discrete pose transformation matrices in the path, Ti ∈ SE(3) represents the manipulator pose, and se(3) is the Lie algebra space norm.

[0106] The calculation method of the energy consumption index E is:

[0107]

[0108] Among them, t0 and t f are the start time and end time of the path respectively, τ j is the joint torque, is the joint angular velocity, and k = 0.1 is the friction coefficient.

[0109] The calculation method of the safety margin index S is:

[0110]

[0111] Among them, m is the number of obstacles in the environment, d i is the distance between the path point and the nearest obstacle, v i is the relative velocity of the dynamic obstacle, and σ = 0.2m is the attenuation coefficient;

[0112] Step S42, Sampling and Expansion: Randomly sample new nodes in the state space with a probability of 15%; according to the dynamic window method, generate a series of candidate nodes from the current node within a certain time and speed range; calculate the evaluation value of each candidate node through the cost function;

[0113] Step S43. Select the optimal node: Select the node with the optimal evaluation value from the candidate nodes as the child node and add it to the RRT tree; update the parent node information of the node for subsequent path backtracking.

[0114] Step S44. Conflict detection and resolution: Check whether the new node conflicts with obstacles or existing nodes.

[0115] If a conflict occurs, discard the node and resample.

[0116] Among them, conflicts can be avoided by dynamically adjusting the parameters of the dynamic window method.

[0117] Step S45. Path backtracking and optimization: When the end point is found or the preset search depth is reached, backtrack from the end point, select the parent node with the optimal evaluation value until returning to the start point. Smooth the backtracked path to reduce the twists and mutations of the path.

[0118] Step S46. Algorithm termination condition:

[0119] When a path that meets the conditions is found or the preset maximum number of iterations is reached, the algorithm terminates.

[0120] The algorithm termination condition is set as:

[0121]

[0122] In step S6, it specifically includes:

[0123] Step S61. Obtain the installation result information of the photovoltaic panel, including the position of the photovoltaic panel after installation;

[0124] Obtain the image data of the photovoltaic panel after installation through a vision sensor, and perform feature extraction on the collected image audit bureau to obtain the position and angle of the photovoltaic panel after installation;

[0125] Step S62. Compare the position of the photovoltaic panel after installation with the target installation position of the photovoltaic panel and calculate the pose error. If the pose error < 0.1 mm, end the installation task; otherwise, execute step S63;

[0126] Step S63. Generate a control command according to the pose error;

[0127] Step S64. According to the control command, control the robotic arm to move to the position where the photovoltaic panel is located, clamp the photovoltaic panel, and perform contact effectiveness detection. If it is effective, execute the control command and return to step S61.

[0128] As Figure 3 shown, the present invention provides an embodied vision-action coupling photovoltaic robot control device for implementing the above control method, including:

[0129] A visual data processing module, electrically connected to the visual sensor, for extracting features from the image data acquired by the visual sensor to obtain a photovoltaic panel installation scenario model and photovoltaic panel target position information;

[0130] A mechanical data processing module, electrically connected to the mechanical sensor, for acquiring and processing the detection data of the mechanical sensor;

[0131] A confidence evaluation module, for performing spatio-temporal alignment of visual data and mechanical data and evaluating the fusion credibility;

[0132] A path planning module, for performing path planning based on the RRT* path planning algorithm;

[0133] An instruction output module, electrically connected to the visual sensor, the mechanical sensor, the mobile device, and the robotic arm driving device, for outputting control instructions to drive the visual sensor, the mechanical sensor, the mobile device, and the robotic arm driving device to act.

[0134] It should be noted that in this article, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or terminal device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article or terminal device including the element.

[0135] Finally, it should be noted that the above is the preferred embodiment of the present invention. It should be pointed out that although the preferred embodiments of the present invention have been described, for those skilled in the art of this technology, once the basic creative concept of the present invention is known, without departing from the principle described in the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention. Therefore, the appended claims are intended to be construed to include the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present invention.

Claims

1. A photovoltaic robot control method with embodied vision-action coupling, characterized in that, It includes the following steps: Step S1: Extract features from the acquisition result of the photovoltaic installation site image obtained by the vision sensor to generate photovoltaic panel installation vision data; Step S2: Obtain the detection result of the mechanical sensor on the robotic arm gripper when the robotic arm picks up the photovoltaic panel to generate photovoltaic panel installation mechanical data; Step S3: Perform spatio-temporal alignment on the photovoltaic panel installation vision data and the photovoltaic panel installation mechanical data, and conduct a fusion credibility assessment; Step S4: According to the passing of the fusion credibility assessment, use the path planning algorithm based on RRT* to plan a motion trajectory that meets the requirements of obstacle-free and dynamic feasibility; Step S5: Generate a photovoltaic panel installation instruction based on the obtained motion trajectory to drive the photovoltaic robot to install the photovoltaic panel.

2. The photovoltaic robot control method with embodied vision-action coupling according to claim 1, wherein It also includes: Step S6: Detect the error of the photovoltaic panel installation result, and generate an error compensation instruction according to the installation error exceeding the error threshold to drive the robotic arm to perform error correction.

3. The photovoltaic robot control method for embodied vision-action coupling according to claim 1, characterized in that, In step S1, the vision sensor includes a 2D camera and a 3D depth camera. The 2D camera is used to obtain the two-dimensional images of the photovoltaic panel and the photovoltaic bracket, and the 3D depth camera is used to obtain the depth images of the photovoltaic panel and the photovoltaic bracket; The photovoltaic panel installation vision data is a photovoltaic panel installation scene model, which at least includes the position information and angle information of the photovoltaic bracket.

4. The embodied vision-action coupled photovoltaic robot control method according to claim 3, characterized in that In the said step S2, it specifically includes: Step S21: According to the gripper of the robotic arm contacting the photovoltaic panel, continuously obtain the detection data of the mechanical sensor on the robotic arm to judge the contact effectiveness between the gripper and the photovoltaic panel; When the gripper on the robotic arm touches the photovoltaic panel, the mechanical sensor on the gripper obtains a detection reading F. When the detection reading F meets the following conditions, it is determined that the gripper and the photovoltaic panel are in effective contact: The absolute value of the detection reading |F| ≥ 1N, and the duration T of the detection reading F is greater than 10 ms; Step S22: According to the effective contact between the gripper and the photovoltaic panel and the gripper clamping the photovoltaic panel, continuously obtain the detection data of the mechanical sensor on the robotic arm, including the detection reading F of the mechanical sensor and the duration T of the detection reading F.

5. The photovoltaic robot control method for embodied vision-action coupling according to claim 4, characterized in that, In step S3, it specifically includes: Step S31: Perform spatio-temporal alignment on the obtained photovoltaic panel installation scene model and the detection reading of the mechanical sensor; Step S32: Conduct a fusion credibility assessment on the data after spatio-temporal alignment. The fusion credibility assessment includes a vision confidence assessment and a mechanical confidence assessment; Step S33: Compare the vision confidence and the mechanical confidence with the vision confidence threshold and the mechanical confidence threshold respectively. If the confidence is greater than the threshold, it is determined to pass, otherwise it fails; if both the vision confidence and the mechanical confidence pass, execute step S4; if only one of the vision confidence and the mechanical confidence passes, return to step S1 according to the vision confidence not passing, and return to step S2 according to the mechanical confidence not passing to re-obtain the vision data or re-pick up the photovoltaic panel; if both do not pass, control the robot to stop and notify the staff for inspection.

6. The photovoltaic robot control method for embodied vision-action coupling according to claim 5, wherein The calculation method of the said mechanical confidence is as follows: B = b1 * f + b2 * t Wherein, B is the mechanical confidence index, b1 and b2 are both coefficients, f is the inspection coefficient of the force of the mechanical sensor, and f = F / F0, where F0 is the reference value of the photovoltaic panel mechanical sensor; t is equal to the inspection coefficient of the duration, and t = T / T0, where T0 is the reference value of the duration.

7. The photovoltaic robot control method for embodied vision-action coupling according to claim 5, characterized in that In the step S4, it specifically includes: Obtaining environmental data and dynamic obstacle motion information, where the environmental data includes the target installation position of the photovoltaic panel, the current pose of the fixture, and the photovoltaic panel installation scenario model; Defining the workspace, cost function, and configuring the RRT tree parameters; Adopting the target bias sampling strategy to sample the working area with a fixed probability, and performing path planning through the RTT* algorithm. Among them, the termination condition of the RTT* algorithm is 8. The method for controlling a photovoltaic robot with embodied vision-action coupling according to claim 7, wherein The definition of the workspace includes: Defining the configuration space: mapping the 7-degree-of-freedom joint space of the robotic arm to the SE(3) workspace; Defining the safe operation area: predicting the motion trajectory through the Kalman filter according to the dynamic obstacle motion information; based on the photovoltaic panel installation scenario model and the motion trajectory prediction result, defining the static obstacle installation operation area and the dynamic obstacle installation operation area according to the minimum static safety distance and the minimum dynamic safety distance. The dynamic obstacle installation operation area is obtained according to the dynamic obstacle motion trajectory, and the dynamic obstacle motion trajectory is obtained by: the distance from the static obstacle is the minimum static safety distance.

9. The method for controlling a photovoltaic robot with embodied vision-action coupling according to claim 7, wherein The cost function is set as: f(n) = α*L + β*E + γ*S Wherein, L represents the path length index, E represents the energy consumption index, and S represents the safety margin index; The calculation method of the path length index L is: Where n is the total number of discrete pose transformation matrices in the path, Ti ∈ SE(3) represents the pose of the robotic arm, and se(3) is the Lie algebra space norm; The calculation method of the energy consumption index E is: where t0 and t f are the start time and end time of the path respectively, τ j is the joint torque, is the joint angular velocity, and k = 0.1 is the friction coefficient; The calculation method of the safety margin index S is: where m is the number of obstacles in the environment, d i is the distance between the path point and the nearest obstacle, v i is the relative velocity of the dynamic obstacle, and σ = 0.2 m is the attenuation coefficient.

10. A photovoltaic robot control device with embodied vision-action coupling for implementing the control method according to any one of claims 1 to 9, including: A visual data processing module, electrically connected to the visual sensor, for extracting features from the image data acquired by the visual sensor to obtain the photovoltaic panel installation scenario model and the photovoltaic panel target position information; A mechanical data processing module, electrically connected to the mechanical sensor, for acquiring and processing the detection data of the mechanical sensor; A confidence evaluation module, for performing spatio-temporal alignment of visual data and mechanical data and evaluating the fusion credibility; A path planning module, for performing path planning based on the RRT* path planning algorithm; An instruction output module, electrically connected to the visual sensor, mechanical sensor, mobile device, and robotic arm driving device, for outputting control instructions to drive the visual sensor, mechanical sensor, mobile device, and robotic arm driving device to act.

Images