A Robot Cooperative Control Method and Related Devices Based on Visual Servo

Abstract

The present invention discloses a robot cooperative control method and related device based on visual servo, which relates to the technical field of robot control. The method includes: allocating target tasks to each robot based on a task allocation scheme; establishing a task logic process to construct the movement trajectories of each robot and establishing an overall movement trajectory layout; during the execution of the target tasks, calculating the error value of the movement trajectory, constructing a visual servo controller to generate an optimized control quantity to adjust the movement trajectory; calculating a pose compensation value based on the generated rotational error and translational error and adjusting its control joints; parsing the operation data on the cloud device and transmitting it to several robots that need to share it based on a preset data sharing logic; the corresponding robots adjust the working parameters in real time for cooperative cooperation. The present invention improves the task operation efficiency and quality of the robots, enabling the control of the robots to achieve a more ideal effect.

Description

Technical Field

[0001] The present invention relates to the technical field of robot control, and in particular, to a robot cooperative control method and related device based on visual servo. Background Art

[0002] Robots can be used for operations such as grasping, processing, assembling, and transporting task components in various industries. Through robots, high-difficulty work can be executed efficiently and accurately, thereby improving production efficiency and reducing labor costs. And with the continuous development of robot research and technology development, it has been able to perform cooperative control on robots through visual servo technology. However, in the existing robot cooperative control method based on visual servo technology, after assigning corresponding target tasks to the robots, for the movement trajectories of each robot, only the movement trajectory of each robot is planned, but the consideration of the trajectory intersection points is lacking, and an overall movement layout of the robots cannot be formed, resulting in collisions when the robots move. At the same time, in the further control of the robots' work using visual servo technology, the existing visual servo control overly relies on the calibration of camera parameters, and the calibration of camera parameters overly depends on the professional level of calibration technicians. At the same time, in the existing visual servo control, there are problems such as imperfect system establishment and unknown parameter variables in the visual servo controller, resulting in the reliability and stability of the further control of the robots by visual servo not being guaranteed. And there are also problems of lack of data sharing logic and large data processing burden in the cooperative control of robots, resulting in the cooperative control of robots not being able to achieve the ideal effect and affecting the task operation quality of the robots. Summary of the Invention

[0003] The purpose of the present invention is to overcome the deficiencies of the prior art. The present invention provides a robot cooperative control method and related device based on visual servo, which improves the task operation efficiency and quality of the robots, and enables the control of the robots to achieve a more ideal effect.

[0004] To solve the above technical problems, the present invention provides a robot cooperative control method based on visual servo, and the method includes:

[0005] Obtain a task assignment plan, and assign corresponding target tasks to each robot based on the task assignment plan, where the target tasks include task components assigned to each robot;

[0006] Determine the working range of each robot based on the target tasks, establish a task logic process, construct the movement trajectories of each robot within the working range based on the task logic process, and establish an overall movement trajectory layout based on the movement trajectories of each robot;

[0007] Control each robot to execute the target task according to the overall movement trajectory layout. During the process of each robot executing the target task according to the overall movement trajectory layout, collect the real-time video stream, extract the current movement trajectory of each robot based on the real-time video stream, calculate the error value between the current movement trajectory and the movement trajectories of each robot in the overall movement trajectory layout, and construct a visual servo controller. Based on the visual servo controller, use the error value to generate an optimized control amount, and adjust the movement trajectory of the robot based on the optimized control amount;

[0008] During the process of each robot executing the target task according to the overall movement trajectory layout, obtain the current processing pose of each robot for the task component, obtain the preset expected processing pose of each robot, generate the corresponding rotational error and translational error based on the current processing pose and the preset expected processing pose, calculate the corresponding pose compensation value based on the rotational error and translational error, and adjust the control joints of each robot based on the pose compensation value;

[0009] During the process of each robot executing the target task according to the overall movement trajectory layout, obtain the operation data uploaded by the robot based on the cloud device, parse the operation data, obtain the data to be shared, and transmit the data to be shared to several robots that need to share the data to be shared based on the preset data sharing logic;

[0010] The corresponding robot adjusts the working parameters in real time based on the data to be shared, and cooperates with each other based on the working parameters adjusted in real time.

[0011] Optionally, the establishment of the task logic process, the construction of the movement trajectory of each robot within the working range based on the task logic process, and the establishment of the overall movement trajectory layout based on the movement trajectories of each robot include:

[0012] Determine the target task process based on the task assignment scheme, and establish a task logic process based on the logical relationship between different target tasks and the positions of different target tasks in the task process;

[0013] Construct the working model of each robot based on the task logic process, obtain the current position point of each robot, and use Cartesian space to plan the movement trajectory of each robot within the working range based on the working model and the current position point;

[0014] Obtain the trajectory intersection points based on the movement trajectories of each robot, calculate the time difference for the corresponding several robots to move from the current position point to the trajectory intersection points, determine whether there will be a collision between each robot at the trajectory intersection points based on the time difference, and adjust the movement trajectory of each robot based on the judgment result to obtain the optimized movement trajectories of each robot;

[0015] Establish an overall movement trajectory layout based on the optimized movement trajectories of each robot.

[0016] Optionally, constructing the visual servo controller to generate an optimized control quantity based on the error value using the visual servo controller includes:

[0017] Construct a visual servo error system based on the neurodynamics of constrained global optimization combined with a preset robot kinematic model;

[0018] Perform Euler discretization on the visual servo error system to obtain a discretized visual servo error system;

[0019] Define an objective function based on sequential quadratic programming using a positive definite weighting matrix, and construct a visual servo control model based on the objective function using the discretized visual servo error system;

[0020] Generate a performance constraint function based on performance constraints, and construct a visual servo controller based on the visual servo control model combined with the performance constraint function;

[0021] Generate an adaptive control rate based on the error value using the visual servo controller, and generate an optimized control quantity based on the adaptive control rate.

[0022] Optionally, generating the corresponding rotational error and translational error based on the current machining pose and the preset desired machining pose includes:

[0023] Obtain the first pixel coordinates corresponding to the current machining pose and the second pixel coordinates corresponding to the preset desired machining pose;

[0024] Construct a projection model, obtain the conversion relationship between the first pixel coordinates and the second pixel coordinates based on the projection model, and generate a corresponding projection homography matrix based on the conversion relationship;

[0025] Decompose the projection homography matrix to obtain the corresponding rotational error and translational error.

[0026] Optionally, calculating a pose compensation value based on the rotational error and translational error, and adjusting the control joints of each robot based on the pose compensation value includes:

[0027] Calculate a joint arm movement angle compensation value, a control torque compensation value, and a robot joint movement speed compensation value based on the rotational error and translational error, and the joint arm movement angle compensation value, the control torque compensation value, and the robot joint movement speed compensation value constitute the pose compensation value;

[0028] Perform linear fitting on the pose compensation value to obtain a linear fitting result, convert the linear fitting result into a control signal, and transmit the control signal to the corresponding robot. The corresponding robot adjusts the control joints based on the control signal.

[0029] Optionally, the performing linear fitting on the pose compensation value to obtain a linear fitting result includes:

[0030] Perform convergence processing on the pose compensation value based on an approximation function with equiripple characteristics to obtain the pose compensation value after convergence processing;

[0031] Perform least squares linear fitting processing on the pose compensation value after convergence processing based on a fitting factor to obtain a linear fitting result.

[0032] Optionally, the parsing the operation data to obtain data to be shared and transmitting the data to be shared to several robots that need to share the data based on a preset data sharing logic includes:

[0033] Parse the operation data based on a preset data parsing logic to obtain data to be shared;

[0034] Obtain the mapping relationship between the robot identifier and the data tag of the data to be shared based on a preset data sharing logic, and determine several robots that need to share the data to be shared based on the mapping relationship;

[0035] Obtain the network protocol addresses and ports of several robots that need to share the data to be shared, and transmit the data to be shared to several robots that need to share the data to be shared based on the network protocol addresses and ports.

[0036] In addition, the present invention also provides a robot cooperative control device based on visual servo. The device includes:

[0037] Task allocation module: used to obtain a task allocation plan, and allocate corresponding target tasks to each robot based on the task allocation plan. The target tasks include task components allocated to each robot;

[0038] Moving trajectory layout module: used to determine the working range of each robot based on the target task, establish a task logic process, construct the moving trajectories of each robot within the working range based on the task logic process, and establish an overall moving trajectory layout based on the moving trajectories of each robot;

[0039] Moving trajectory adjustment module: It is used to control each robot to execute the target task according to the overall moving trajectory layout. During the process of each robot executing the target task according to the overall moving trajectory layout, it collects real-time video streams, extracts the current moving trajectory of each robot based on the real-time video streams, calculates the error value between the current moving trajectory and the moving trajectories of each robot in the overall moving trajectory layout, constructs a visual servo controller, generates an optimized control quantity using the error value based on the visual servo controller, and adjusts the moving trajectory of the robot based on the optimized control quantity;

[0040] Processing pose adjustment module: It is used to obtain the current processing pose of each robot for the task component during the process of each robot executing the target task according to the overall moving trajectory layout, obtain the preset expected processing pose of each robot, generate corresponding rotation error and translation error based on the current processing pose and the preset expected processing pose, calculate the corresponding pose compensation value based on the rotation error and translation error, and adjust the control joints of each robot based on the pose compensation value;

[0041] Data sharing module: It is used to obtain the operation data uploaded by the robot based on the cloud device during the process of each robot executing the target task according to the overall moving trajectory layout, parse the operation data, obtain the data to be shared, and transmit the data to be shared to several robots that need to share the data to be shared based on the preset data sharing logic;

[0042] Cooperative coordination module: It is used for the corresponding robot to adjust the working parameters in real time based on the data to be shared, and perform cooperative coordination based on the working parameters adjusted in real time.

[0043] In addition, the present invention also provides an electronic device, which includes a processor and a memory. The memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the electronic device executes the above-mentioned robot cooperative control method based on visual servo.

[0044] In addition, the present invention also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions run on an electronic device, the electronic device is enabled to execute the above-mentioned robot cooperative control method based on visual servo.

[0045] In the embodiments of the present invention, the consideration of trajectory intersection points is added to the movement trajectory of each robot, and the calculation of time difference is introduced, which avoids collisions of robots during the execution of target tasks and enables the overall movement trajectory layout of robots. During the execution of target tasks by robots, a visual servo error model based on constrained global optimization neural dynamics is established. On this basis, a visual servo controller is constructed by combining quadratic sequential programming and performance constraints. This visual servo controller is more perfect and can avoid sudden changes in control quantities, enabling the visual servo controller to more effectively drive the robot to the target point. At the same time, the pose compensation values generated by rotational error and translational error are used to adjust the control joints of each robot, greatly reducing the dependence on parameter calibration. At the same time, by using an approximation function and fitting factor with equiripple characteristics to linearly fit the pose compensation values, the adjustment amount of the robot control joints is simplified. Meanwhile, linear fitting can eliminate the influence of robot vibration on the calculation results, further improving the reliability and stability of robot control. By presetting data sharing logic, the data to be shared is sent to the robots that need data sharing, realizing the cooperation between robots, reducing the data processing burden of robots, improving the task operation efficiency and quality of robots, and enabling the control of robots to achieve a more ideal effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0047] Figure 1 is a schematic flowchart of a visual servo-based robot cooperative control method in the embodiments of the present invention;

[0048] Figure 2 is a schematic structural composition diagram of a visual servo-based robot cooperative control device in the embodiments of the present invention;

[0049] Figure 3 is a schematic structural composition diagram of an electronic device in the embodiments of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0050] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0051] Embodiment 1

[0052] Please refer to Figure 1 , Figure 1 which is a schematic flow chart of a robot cooperative control method based on visual servo in the embodiments of the present invention.

[0053] As Figure 1 shown, a robot cooperative control method based on visual servo, the method includes:

[0054] S11: Obtain a task allocation scheme, and allocate corresponding target tasks to each robot based on the task allocation scheme. The target tasks include task components allocated to each robot.

[0055] In the specific implementation process of the present invention, task feature analysis is performed on all target tasks to obtain target task feature information. Cooperative task allocation is performed based on the target task feature information to obtain an initial task allocation scheme. At the same time, the initial task allocation scheme is adjusted using the task load upper limit of the robot to obtain a task allocation scheme. Corresponding target tasks are allocated to each robot based on the task allocation scheme. The target tasks include task components allocated to each robot.

[0056] S12: Determine the working range of each robot based on the target task, establish a task logic process, construct the movement trajectory of each robot within the working range based on the task logic process, and establish an overall movement trajectory layout based on the movement trajectories of each robot.

[0057] In the specific implementation process of the present invention, the establishment of the task logic process, the construction of the movement trajectories of each robot within the working range based on the task logic process, and the establishment of the overall movement trajectory layout based on the movement trajectories of each robot include: determining the target task process based on the task allocation scheme, and establishing the task logic process based on the logical relationship between different target tasks and the positions of different target tasks in the task process; constructing the working models of each robot based on the task logic process, obtaining the current position points of each robot, and using Cartesian space to plan the movement trajectories of each robot within the working range based on the working models and the current position points; obtaining the trajectory intersection points based on the movement trajectories of each robot, calculating the time differences for the corresponding several robots to move from the current position points to the trajectory intersection points, determining whether there will be collisions between each robot at the trajectory intersection points based on the time differences, and adjusting the movement trajectories of each robot based on the judgment results to obtain the optimized movement trajectories of each robot; and establishing the overall movement trajectory layout based on the optimized movement trajectories of each robot.

[0058] Specifically, based on the target tasks, determine the working ranges of each robot, determine the target task process based on the task allocation scheme, and there is a corresponding task process set in the task allocation scheme. Establish the task logic process based on the logical relationship between different target tasks and the positions of different target tasks in the task process. Analyze each target task through the task engine, and the task engine is configured based on a directed acyclic graph. Each node in the directed acyclic graph is associated with a target task, and the processing logic of each target task is implemented by a corresponding operator. Each edge in the directed acyclic graph is used to describe the logical relationship between related target tasks. Thus, the logical relationship between different target tasks can be obtained, and the corresponding positions of different target tasks in the task process can be found. Based on the corresponding logical relationship and the corresponding positions, a specific target task logic process can be established. Construct the working models of each robot based on the task logic process, and establish the working models of each robot according to the task logic process and the working attribute information of each robot. The working attribute information of the robot includes the movement speed and the maximum load-bearing capacity for task components, etc. Obtain the current position points of each robot. Each robot is equipped with a positioning device, and its current position point can be obtained through the positioning device. Use Cartesian space to plan the movement trajectories of each robot within the working range based on the working models and the current position points. The initial movement trajectory of the Cartesian space of the robot is formed by the working models and the current position points, and the trajectory constraints of the robot's Cartesian space and the objective function of the optimized trajectory planning are established. The objective function of the optimized trajectory planning is:

[0059]

[0060] where, t totalThe motion time of the robot for trajectory planning is \(t\), and the moving speed of the robot is \(v\). The initial moving trajectory is subjected to time integration through the trajectory constraints in the Cartesian space of the robot and the objective function for optimizing the trajectory planning, so as to obtain the moving trajectories of each robot within the working range. Trajectory intersection points are obtained based on the moving trajectories of each robot. The trajectory intersection points are searched according to the moving trajectories of each robot, and the time differences for the corresponding several robots to move from the current position point to the trajectory intersection points are calculated. The moving time differences for moving from the current position point to the estimated intersection points are calculated according to the moving speed of each robot. Based on the time differences, it is judged whether there will be collisions between the robots at the trajectory intersection points. If the time difference is greater than the preset time difference threshold, it means that at this trajectory intersection point, the corresponding robots pass through successively, so there will be no collisions between the corresponding several robots. If the time difference is less than or equal to the preset time difference threshold, it means that at this trajectory intersection point, the corresponding several robots do not have enough time difference to pass through successively, resulting in collisions of the robots at this trajectory intersection point. And the moving trajectories of each robot are adjusted based on the judgment result. When it is judged that the trajectory robot will collide when moving according to its moving trajectory, the passing order of the robots is preferentially graded according to the priorities of the target tasks assigned to each robot. The robot with a higher task priority passes first, and the robot with a lower task priority waits for the previous robot to pass before moving. Until all the trajectories that will collide are adjusted, the optimized moving trajectories of each robot are obtained. An overall moving trajectory layout is established based on the optimized moving trajectories of each robot. The optimized moving trajectories of each robot are combined as a whole to form the overall moving trajectory layout.

[0061] S13: Control each robot to execute the target task according to the overall moving trajectory layout. During the process of each robot executing the target task according to the overall moving trajectory layout, collect the real-time video stream, extract the current moving trajectory of each robot based on the real-time video stream, calculate the error value between the current moving trajectory and the moving trajectories of each robot in the overall moving trajectory layout, and construct a visual servo controller. Based on the visual servo controller, use the error value to generate an optimized control quantity, and adjust the moving trajectory of the robot based on the optimized control quantity;

[0062] In the specific implementation process of the present invention, constructing the visual servo controller and generating an optimized control quantity based on the error value by using the visual servo controller includes: constructing a visual servo error system based on constrained global optimization neural dynamics combined with a preset robot kinematic model; performing Euler discretization on the visual servo error system to obtain a discretized visual servo error system; defining an objective function based on sequential quadratic programming using a positive definite weighting matrix, and constructing a visual servo control model based on the objective function by using the discretized visual servo error system; generating a performance constraint function based on performance constraints, and constructing a visual servo controller based on the visual servo control model in combination with the performance constraint function; generating an adaptive control law based on the visual servo controller by using the error value, and generating an optimized control quantity based on the adaptive control law.

[0063] Specifically, controlling each robot to execute a target task according to the overall movement trajectory layout. During the process of each robot executing the target task according to the overall movement trajectory layout, a real-time video stream is collected through the camera device of the robot, and the current movement trajectory of each robot is extracted based on the real-time video stream. By extracting each frame of the image in the real-time video stream, the current movement direction and route of each robot can be obtained based on each frame of the image. Calculate the error value between the current movement trajectory and the movement trajectories of each robot in the overall movement trajectory layout, that is, calculate the direction deviation value between the current movement trajectory and the movement trajectories of each robot in the overall movement trajectory layout and the distance deviation value between the current movement trajectory and the corresponding movement trajectory in the overall movement trajectory layout. Construct a visual servo error system based on constrained global optimization neural dynamics combined with a preset robot kinematic model. Considering that the robot may have a speed jump, which will affect the control effect, constrained global optimization neural dynamics is added to construct the visual servo error system, making the control signal smoother. Constrained global optimization neural dynamics can be regarded as a simulation of the rapid thinking process of the brain. Each individual neural network performs local search according to its own neural dynamics and converges to a desired target solution. The error signal is processed by constrained global optimization neural dynamics to obtain an error signal vector, and the error signal vector obtained by the constrained global optimization neural dynamics is introduced in combination with the kinematic model of the robot to construct a visual servo error system. Perform Euler discretization on the visual servo error system, discretize the continuous time variable in the visual servo error system, convert the differential into a difference, and obtain a discretized visual servo error system. Define an objective function based on sequential quadratic programming using a positive definite weighting matrix, and use sequential quadratic programming to minimize the control input of the quadratic objective function by using the positive definite weighting matrix. Define the objective function through the obtained control input, and construct a visual servo control model based on the objective function by using the discretized visual servo error system. The expression of the visual servo control model is:

[0064] E(k+i|k) = h(E(k+i-1|k)) + u(k+i-1|k),

[0065] where E(k+i|k) is the predicted value of the error signal vector E obtained from the discretized visual servo error system at time k for time k+i, h is the corresponding objective function, E(k+i-1|k) is the predicted value of the error signal vector E obtained from the discretized visual servo error system at time k for time k+i-1, and u(k+i-1|k) is the predicted value of the control input u obtained from the objective function at time k for k+i-1. A performance constraint function is generated based on performance constraints. According to the expected movement trajectory of the robot during task execution, that is, the movement trajectories corresponding to each robot in the overall movement trajectory layout, the task execution coefficient and dynamic constraint conditions for executing the target task are determined through this movement trajectory. A performance constraint function is constructed using the task execution coefficient and dynamic constraint conditions based on the hyperplane constraint conditions in the generalized space. A visual servo controller is constructed by combining the visual servo control model with the performance constraint function. The performance constraint function is also added to the visual servo control model, that is, the performance constraint function is introduced as a constraint condition into the visual servo control model, thereby constructing the overall visual servo controller. An adaptive control rate is generated based on the error value using the visual servo controller. The error value is input into the visual servo controller, and the visual servo controller can obtain the corresponding adaptive control rate. An optimized control quantity is generated based on the adaptive control rate. The change of the corresponding variable of the movement trajectory can be adjusted in real time through the adaptive control rate, that is, the optimized control quantity is obtained, and the movement trajectory of the robot is adjusted based on the optimized control quantity.

[0066] S14: During the process of each robot executing the target task according to the overall movement trajectory layout, obtain the current machining pose of each robot for the task component, obtain the preset expected machining pose of each robot, generate the corresponding rotational error and translational error based on the current machining pose and the preset expected machining pose, and calculate the corresponding pose compensation value based on the rotational error and translational error, and adjust the control joints of each robot based on the pose compensation value;

[0067] In the specific implementation process of the present invention, the generating the corresponding rotational error and translational error based on the current machining pose and the preset expected machining pose includes: obtaining the first pixel coordinates corresponding to the current machining pose and the second pixel coordinates corresponding to the preset expected machining pose; constructing a projection model, obtaining the conversion relationship between the first pixel coordinates and the second pixel coordinates based on the projection model, and generating the corresponding projection homography matrix based on the conversion relationship; decomposing the projection homography matrix to obtain the corresponding rotational error and translational error.

[0068] Further, calculating a pose compensation value based on the rotation error and the translation error, and adjusting the control joints of each robot based on the pose compensation value, includes: calculating a joint arm movement angle compensation value, a control torque compensation value, and a robot joint movement speed compensation value based on the rotation error and the translation error, where the joint arm movement angle compensation value, the control torque compensation value, and the robot joint movement speed compensation value constitute the pose compensation value; performing linear fitting on the pose compensation value to obtain a linear fitting result, converting the linear fitting result into a control signal, and transmitting the control signal to the corresponding robot, and the corresponding robot adjusts the control joints based on the control signal.

[0069] Further, performing linear fitting on the pose compensation value to obtain a linear fitting result includes: performing convergence processing on the pose compensation value based on an approximation function with an equiripple characteristic to obtain a pose compensation value after convergence processing; performing least squares linear fitting processing on the pose compensation value after convergence processing based on a fitting factor to obtain a linear fitting result.

[0070] Specifically, during the process of each robot executing the target task according to the overall movement trajectory layout, obtain the current machining pose of each robot with respect to the task component. The current machining pose of the robot with respect to the task component is obtained through the binocular vision cameras of each robot in its corresponding coordinate system. Obtain the preset expected machining pose of each robot, obtain the first pixel coordinates corresponding to the current machining pose and the second pixel coordinates corresponding to the preset expected machining pose. The corresponding pixel coordinates can be obtained through the corresponding coordinate system. Use the global position parameters of the robot to construct a projection model by means of a projection function, and obtain the conversion relationship between the first pixel coordinates and the second pixel coordinates based on the projection model. Calculate the conversion relationship between the first pixel coordinates and the second pixel coordinates according to the relationship expression of the expected pixel coordinates using the projection model. Generate the corresponding projective homography matrix based on the conversion relationship, that is, the projective homography matrix converted to the projected pixel coordinates through the conversion relationship. Decompose the projective homography matrix, and determine that three of the columns of the projective homography matrix are the first matrix. Determine the homography matrix and rotation matrix corresponding to the image plane tilt according to the first matrix. Decompose the homography matrix and rotation matrix of the image plane tilt into an orthogonal matrix and an upper triangular matrix, and optimize and iterate the orthogonal matrix and the upper triangular matrix to obtain the corresponding rotation error and translation error. Calculate the joint arm movement angle compensation value, the control torque compensation value, and the robot joint movement speed compensation value based on the rotation error and the translation error. According to the rotation error and the translation error, use the inverse kinematics of the robot to calculate the angular deviation, torque deviation, and movement speed deviation of the joint arm of the robot from the current machining pose to the characteristic expected point of the preset expected machining pose. Generate the joint arm movement angle compensation value, the control torque compensation value, and the robot joint movement speed compensation value according to the angular deviation, torque deviation, and movement speed deviation. The joint arm movement angle compensation value, the control torque compensation value, and the robot joint movement speed compensation value constitute the pose compensation value. Perform a convergence process on the pose compensation value based on an approximation function with equiripple characteristics to obtain the pose compensation value after the convergence process. The characteristic of the Butterworth filter is that the frequency response curve in the passband is maximally flat without fluctuations, and gradually drops to zero in the stopband. When the boundary of the passband meets the index requirements, there will be a large margin in the passband, and an approximation function with equiripple characteristics can evenly distribute the approximation accuracy throughout the passband, or evenly distribute it throughout the stopband, or evenly distribute it within both at the same time. In this way, the filter order can be greatly reduced. Therefore, perform a convergence process on the pose compensation value using the Butterworth filter based on an approximation function with equiripple characteristics to obtain the pose compensation value after the convergence process.Perform least - squares linear fitting on the pose compensation value after convergence processing based on the fitting factor, establish a linear mapping relationship by combining least - squares linear fitting with the fitting factor, generate a dynamic mapping function based on the linear mapping relationship, optimize and correct the pose compensation value after convergence processing according to the dynamic mapping function to obtain the optimized and corrected pose compensation value, generate a linear fitting result based on the optimized and corrected pose compensation value, and the linear fitting result can be used for the precise compensation control of the robot. Convert the linear fitting result into a control signal, transmit the control signal to the corresponding robot, the corresponding robot adjusts the control joint based on the control signal, the corresponding robot obtains the joint adjustment amount through the control signal, and adjusts the control joint through the joint adjustment amount.

[0071] S15: During the process of each robot performing the target task according to the overall movement trajectory layout, obtain the operation data uploaded by the robot based on the cloud device, parse the operation data to obtain the data to be shared, and transmit the data to be shared to several robots that need to share the data to be shared based on the preset data sharing logic.

[0072] In the specific implementation process of the present invention, the step of parsing the operation data to obtain the data to be shared and transmitting the data to be shared to several robots that need to share the data to be shared based on the preset data sharing logic includes: parsing the operation data based on the preset data parsing logic to obtain the data to be shared; obtaining the mapping relationship between the robot identifier and the data label of the data to be shared based on the preset data sharing logic, and determining several robots that need to share the data to be shared based on the mapping relationship; obtaining the network protocol address and port of several robots that need to share the data to be shared, and transmitting the data to be shared to several robots that need to share the data to be shared based on the network protocol address and port.

[0073] Specifically, during the process of each robot executing the target task according to the overall movement trajectory layout, the operation data uploaded by the robot is obtained based on the cloud device, and the operation data is parsed based on the preset data parsing logic. The data parsing logic can be pre-configured in the cloud device and is adaptively set according to the application scenarios of the collaborative operation of several robots. The data to be shared obtained by parsing the operation data according to the preset data parsing logic is data that can be directly used for data sharing. According to the preset data parsing logic, the structure and form of the operation data can be converted, and then data redundancy is removed to obtain the data to be shared. The data to be shared may include the labeled map, the position information of the robots that need to perform collaborative operations, and the task operation requirements, etc. Based on the preset data sharing logic, the mapping relationship between the robot identifier and the data label of the data to be shared is obtained. Similarly, the data sharing logic can be pre-configured in the cloud device and is adaptively set according to the application scenarios of the collaborative operation of several robots. The preset sharing logic includes the mapping relationship between the robot identifier and the corresponding data label. The robot identifier is the unique identity certificate of each robot, and the data label is the unique identifier of the data. Based on the mapping relationship, several robots that need to share the data to be shared are determined. The network protocol addresses and ports of several robots that need to share the data to be shared are obtained through the cloud device, the corresponding socket interfaces are determined according to the network protocol addresses and ports, and the data to be shared is transmitted to several robots that need to share the data to be shared through the corresponding socket interfaces using the network protocol addresses and ports.

[0074] S16: The corresponding robot adjusts the working parameters in real time based on the data to be shared and performs collaborative cooperation based on the real-time adjusted working parameters.

[0075] In the specific implementation process of the present invention, the corresponding robot combines the data to be shared with the task data collected in real time by itself to generate task combined data, and uses the preset decision model to adjust the working parameters in real time through the task combined data, and completes the collaborative cooperation of the target task based on the real-time adjusted working parameters. Each robot has the ability of decision-making and control. Through the real-time sharing of data and the real-time adjustment of working parameters, each robot realizes collaborative control.

[0076] In the embodiments of the present invention, the moving trajectories of each robot take into account the trajectory intersection points, and the calculation of the time difference is introduced to avoid collisions of the robots during the execution of the target tasks. It is possible to achieve the overall moving trajectory layout of the robots. During the execution of the target tasks by the robots, a visual servo error model based on constrained global optimization neural dynamics is established. On this basis, a visual servo controller is constructed by combining quadratic sequential programming and performance constraints. This visual servo controller is more perfect and can avoid sudden changes in the control quantity, enabling the visual servo controller to more effectively drive the robot to the target point. At the same time, the pose compensation values generated by the rotational error and the translational error are used to adjust the control joints of each robot, greatly reducing the dependence on parameter calibration. At the same time, by using an approximation function and a fitting factor with equal ripple characteristics to linearly fit the pose compensation values, the adjustment amount of the robot control joints is simplified. At the same time, the linear fitting can eliminate the influence of the vibration of the robot on the calculation results, further improving the reliability and stability of the robot control. The data to be shared is sent to the robots that need data sharing through the preset data sharing logic, realizing the cooperation between the robots, reducing the data processing burden of the robots, improving the task operation efficiency and quality of the robots, and enabling the control of the robots to achieve a more ideal effect.

[0077] Embodiment 2

[0078] Please refer to Figure 2 , Figure 2 which is a schematic structural diagram of the robot cooperative control device based on visual servo in the embodiments of the present invention.

[0079] As Figure 2 shown, a robot cooperative control device based on visual servo, the device includes:

[0080] Task allocation module 21: used to obtain the task allocation plan, and allocate corresponding target tasks to each robot based on the task allocation plan. The target tasks include the task components allocated to each robot;

[0081] Moving trajectory layout module 22: used to determine the working range of each robot based on the target task, establish a task logic process, construct the moving trajectories of each robot within the working range based on the task logic process, and establish an overall moving trajectory layout based on the moving trajectories of each robot;

[0082] Mobile trajectory adjustment module 23: It is used to control each robot to execute the target task according to the overall movement trajectory layout. During the process of each robot executing the target task according to the overall movement trajectory layout, it collects real-time video streams, extracts the current movement trajectory of each robot based on the real-time video streams, calculates the error value between the current movement trajectory and the movement trajectories of each robot in the overall movement trajectory layout, constructs a visual servo controller, generates an optimized control quantity using the error value based on the visual servo controller, and adjusts the movement trajectory of the robot based on the optimized control quantity;

[0083] Processing pose adjustment module 24: It is used to obtain the current processing pose of each robot for the task component during the process of each robot executing the target task according to the overall movement trajectory layout, obtain the preset expected processing pose of each robot, generate corresponding rotational error and translational error based on the current processing pose and the preset expected processing pose, calculate the corresponding pose compensation value based on the rotational error and translational error, and adjust the control joints of each robot based on the pose compensation value;

[0084] Data sharing module 25: It is used to obtain the operation data uploaded by the robot based on the cloud device during the process of each robot executing the target task according to the overall movement trajectory layout, parse the operation data, obtain the data to be shared, and transmit the data to be shared to several robots that need to share the data to be shared based on the preset data sharing logic;

[0085] Cooperative cooperation module 26: It is used for the corresponding robot to adjust the working parameters in real time based on the data to be shared, and perform cooperative cooperation based on the working parameters adjusted in real time.

[0086] In the specific implementation process of the present invention, the specific implementation manner of the device item can refer to the implementation manner of the above method item, and will not be elaborated here.

[0087] In the embodiments of the present invention, the consideration of trajectory intersection points is added to the movement trajectory of each robot, and the calculation of time difference is introduced to avoid collisions during the execution of the target task by the robot. It is possible to achieve the overall movement trajectory layout of the robot. During the execution of the target task by the robot, a visual servo error model based on constrained global optimization neural dynamics is established. On this basis, a visual servo controller is constructed by combining quadratic sequential programming and performance constraints. This visual servo controller is more perfect and can avoid sudden changes in the control quantity, enabling the visual servo controller to more effectively drive the robot to the target point. At the same time, the pose compensation value generated by the rotational error and translational error is used to adjust the control joints of each robot, greatly reducing the dependence on parameter calibration. At the same time, by using an approximation function and fitting factor with equiripple characteristics to linearly fit the pose compensation value, the adjustment amount of the robot control joints is simplified. At the same time, linear fitting can eliminate the influence of robot vibration on the calculation results, further improving the reliability and stability of robot control. By presetting the data sharing logic, the data to be shared is sent to the robots that need data sharing, realizing the cooperation between robots, reducing the data processing burden of the robots, improving the task operation efficiency and quality of the robots, and enabling the control of the robots to achieve a more ideal effect.

[0088] A computer-readable storage medium provided by an embodiment of the present invention, on which a computer program is stored. When the program is executed by a processor, it implements the visual servo-based robot cooperative control method in any one of the above embodiments. Among them, the computer-readable storage medium includes, but is not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. That is, the storage device includes any medium that can store or transmit information in a readable form by a device (such as a computer, mobile phone), and can be a read-only memory, a magnetic disk or an optical disk, etc.

[0089] Embodiment III

[0090] Please refer to Figure 3 , Figure 3 which is a schematic diagram of the structural composition of the electronic device in the embodiment of the present invention.

[0091] An embodiment of the present invention further provides an electronic device, such as Figure 3 shown, the electronic device includes a memory 31, a processor 33, and a computer program 32 stored in the memory 31 and executable on the processor 33. Those skilled in the art can understand that Figure 3 the electronic device shown does not constitute a limitation on all devices, and may include more or fewer components than shown, or combine certain components. The memory 31 can be used to store the computer program 32 and each functional module. The processor 33 runs the computer program 32 stored in the memory 31, thereby performing various functional applications and data processing of the device. The memory can be an internal memory or an external memory, or include both an internal memory and an external memory. The internal memory can include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, or a random access memory. The external memory can include a hard disk, a floppy disk, a ZIP disk, a USB flash drive, a magnetic tape, etc. The processor 33 can be a central processing unit (CPU), or can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, a single-chip microcomputer, or the processor 33 can also be any conventional processor, etc. The processors and memories disclosed in the present invention include, but are not limited to, these types of processors and memories. The processors and memories disclosed in the present invention are only examples and not limitations.

[0092] As an embodiment, the electronic device includes: one or more processors 33, a memory 31, one or more computer programs 32, wherein the one or more computer programs 32 are stored in the memory 31 and configured to be executed by the one or more processors 33, and the one or more computer programs 32 are configured to execute the robot cooperative control method based on visual servo in any of the above embodiments. For the specific implementation process, please refer to the above embodiments and will not be elaborated here.

[0093] In the embodiments of the present invention, the consideration of trajectory intersection points is added to the movement trajectory of each robot, and the calculation of time difference is introduced, avoiding collisions of robots during the execution of target tasks, enabling the overall movement trajectory layout of the robots. During the execution of target tasks by the robots, a visual servo error model based on constrained global optimization neurodynamics is established. On this basis, a visual servo controller is constructed by combining quadratic sequential programming and performance constraints. This visual servo controller is more perfect and can avoid sudden changes in control quantities, enabling the visual servo controller to more effectively drive the robots to the target points. At the same time, the pose compensation values generated by rotational errors and translational errors are used to adjust the control joints of each robot, greatly reducing the dependence on parameter calibration. At the same time, by using an approximation function and fitting factor with equiripple characteristics to linearly fit the pose compensation values, the adjustment amount of the robot control joints is simplified. At the same time, linear fitting can eliminate the influence of robot vibration on the calculation results, further improving the reliability and stability of robot control. By presetting data sharing logic, the data to be shared is sent to the robots that need data sharing, realizing the cooperation between the robots, reducing the data processing burden of the robots, improving the task operation efficiency and quality of the robots, and enabling the control of the robots to achieve a more ideal effect.

[0094] In addition, the above has introduced in detail a method and related device for robot cooperative control based on visual servo provided by the embodiments of the present invention. Specific examples are used in this article to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A robot cooperative control method based on visual servo, characterized in that, The method includes: Obtaining a task assignment plan, and based on the task assignment plan, assigning corresponding target tasks to each robot, where the target tasks include task components assigned to each robot; Determining the working range of each robot based on the target tasks, establishing a task logic process, constructing the movement trajectories of each robot within the working range based on the task logic process, and establishing an overall movement trajectory layout based on the movement trajectories of each robot; Controlling each robot to execute the target tasks according to the overall movement trajectory layout. During the process of each robot executing the target tasks according to the overall movement trajectory layout, collecting real-time video streams, extracting the current movement trajectories of each robot based on the real-time video streams, calculating the error values between the current movement trajectories and the movement trajectories of each robot in the overall movement trajectory layout, constructing a visual servo controller, generating an optimized control quantity using the error values based on the visual servo controller, and adjusting the movement trajectories of the robots based on the optimized control quantity; During the process of each robot executing the target tasks according to the overall movement trajectory layout, obtaining the current processing poses of each robot for the task components, obtaining the preset expected processing poses of each robot, generating corresponding rotational errors and translational errors based on the current processing poses and the preset expected processing poses, calculating corresponding pose compensation values based on the rotational errors and translational errors, and adjusting the control joints of each robot based on the pose compensation values; During the process of each robot executing the target tasks according to the overall movement trajectory layout, obtaining the operation data uploaded by the robots based on cloud devices, parsing the operation data, obtaining the data to be shared, and transmitting the data to be shared to several robots that need to share the data to be shared based on a preset data sharing logic; The corresponding robots adjust the working parameters in real time based on the data to be shared and cooperate with each other based on the working parameters adjusted in real time; Among them, the establishment of the task logic process, the construction of the movement trajectories of each robot within the working range based on the task logic process, and the establishment of the overall movement trajectory layout based on the movement trajectories of each robot include: determining the target task process based on the task assignment plan, and establishing a task logic process based on the logical relationship between different target tasks and the positions of different target tasks in the task process; constructing a working model of each robot based on the task logic process, obtaining the current position points of each robot, and using Cartesian space to plan the movement trajectories of each robot within the working range based on the working model and the current position points; obtaining trajectory intersection points based on the movement trajectories of each robot, calculating the time differences for the corresponding several robots to move from the current position points to the trajectory intersection points, judging whether there will be collisions between the robots at the trajectory intersection points based on the time differences, and adjusting the movement trajectories of each robot based on the judgment results to obtain the optimized movement trajectories of each robot; establishing an overall movement trajectory layout based on the optimized movement trajectories of each robot; The construction of the visual servo controller, which generates an optimized control quantity by using the error value based on the visual servo controller, includes: constructing a visual servo error system based on the combination of neurodynamics with constraint global optimization and a preset robot kinematic model; performing Euler discretization on the visual servo error system to obtain a discretized visual servo error system; defining an objective function by using a positive definite weighting matrix based on sequential quadratic programming, and constructing a visual servo control model by using the discretized visual servo error system based on the objective function; generating a performance constraint function based on performance constraints, and constructing a visual servo controller by combining the visual servo control model with the performance constraint function; generating an adaptive control rate by using the error value based on the visual servo controller, and generating an optimized control quantity based on the adaptive control rate.

2. The robot cooperative control method based on visual servo according to claim 1, wherein The generation of the corresponding rotational error and translational error based on the current machining pose and the preset expected machining pose includes: Obtaining a first pixel coordinate corresponding to the current machining pose and a second pixel coordinate corresponding to the preset expected machining pose; Constructing a projection model, obtaining the conversion relationship between the first pixel coordinate and the second pixel coordinate based on the projection model, and generating a corresponding projection homography matrix based on the conversion relationship; Decomposing the projection homography matrix to obtain the corresponding rotational error and translational error.

3. The robot cooperative control method based on visual servo according to claim 1, characterized in that The calculation of the pose compensation value based on the rotational error and translational error, and the adjustment of the control joints of each robot based on the pose compensation value includes: Calculating a joint arm movement angle compensation value, a control torque compensation value, and a robot joint movement speed compensation value based on the rotational error and translational error, and the joint arm movement angle compensation value, the control torque compensation value, and the robot joint movement speed compensation value constitute the pose compensation value; Performing linear fitting on the pose compensation value to obtain a linear fitting result, converting the linear fitting result into a control signal, and transmitting the control signal to the corresponding robot, and the corresponding robot adjusts the control joint based on the control signal.

4. The robot cooperative control method based on visual servo according to claim 3, wherein The performing of linear fitting on the pose compensation value to obtain a linear fitting result includes: Performing a convergence process on the pose compensation value based on an approximation function with an equal ripple characteristic to obtain a pose compensation value after the convergence process; Performing a least squares linear fitting process on the pose compensation value after the convergence process based on a fitting factor to obtain a linear fitting result.

5. The robot cooperative control method based on visual servo according to claim 1, wherein The parsing of the operation data to obtain data to be shared, and transmitting the data to be shared to a plurality of robots that need to share the data to be shared based on a preset data sharing logic includes: Parsing the operation data based on a preset data parsing logic to obtain data to be shared; Obtaining the mapping relationship between the robot identifier and the data tag of the data to be shared based on a preset data sharing logic, and determining a plurality of robots that need to share the data to be shared based on the mapping relationship; Obtaining the network protocol addresses and ports of a plurality of robots that need to share the data to be shared, and transmitting the data to be shared to a plurality of robots that need to share the data to be shared based on the network protocol addresses and ports.

6. A robot cooperative control device based on visual servo, characterized in that, The device includes: Task Allocation Module: It is used to obtain a task allocation plan and allocate corresponding target tasks to each robot based on the task allocation plan. The target tasks include task components allocated to each robot. Moving Trajectory Layout Module: It is used to determine the working range of each robot based on the target tasks, establish a task logic process, construct the moving trajectories of each robot within the working range based on the task logic process, and establish an overall moving trajectory layout based on the moving trajectories of each robot. Moving Trajectory Adjustment Module: It is used to control each robot to execute the target tasks according to the overall moving trajectory layout. During the process of each robot executing the target tasks according to the overall moving trajectory layout, it collects real-time video streams, extracts the current moving trajectories of each robot based on the real-time video streams, calculates the error values between the current moving trajectories and the moving trajectories of each robot in the overall moving trajectory layout, constructs a visual servo controller, generates an optimized control quantity using the error values based on the visual servo controller, and adjusts the moving trajectories of the robots based on the optimized control quantity. Processing Pose Adjustment Module: It is used to obtain the current processing poses of each robot for the task components during the process of each robot executing the target tasks according to the overall moving trajectory layout, obtain the preset expected processing poses of each robot, generate corresponding rotational errors and translational errors based on the current processing poses and the preset expected processing poses, calculate corresponding pose compensation values based on the rotational errors and translational errors, and adjust the control joints of each robot based on the pose compensation values. Data Sharing Module: It is used to obtain the operation data uploaded by the robots based on the cloud device during the process of each robot executing the target tasks according to the overall moving trajectory layout, parse the operation data to obtain the data to be shared, and transmit the data to be shared to several robots that need to share the data to be shared based on the preset data sharing logic. Cooperative Coordination Module: It is used for the corresponding robot to adjust the working parameters in real time based on the data to be shared and perform cooperative coordination based on the working parameters adjusted in real time. Among them, establishing the task logic process, constructing the movement trajectories of each robot within the working range based on the task logic process, and establishing the overall movement trajectory layout based on the movement trajectories of each robot, includes: determining the target task process based on the task allocation scheme, and establishing the task logic process based on the logical relationships between different target tasks and the positions of different target tasks in the task process; constructing the working models of each robot based on the task logic process, obtaining the current position points of each robot, and using Cartesian space to plan the movement trajectories of each robot within the working range based on the working models and the current position points; obtaining the trajectory intersection points based on the movement trajectories of each robot, calculating the time differences for the corresponding several robots to move from the current position points to the trajectory intersection points, judging whether there will be collisions between each robot at the trajectory intersection points based on the time differences, and adjusting the movement trajectories of each robot based on the judgment results to obtain the optimized movement trajectories of each robot; establishing the overall movement trajectory layout based on the optimized movement trajectories of each robot; Constructing the visual servo controller, and generating an optimized control quantity using the error value based on the visual servo controller, includes: constructing a visual servo error system based on the combination of constrained global optimization neurodynamics and a preset robot kinematic model; performing Euler discretization on the visual servo error system to obtain a discretized visual servo error system; defining an objective function based on sequential quadratic programming using a positive definite weighting matrix, and constructing a visual servo control model based on the objective function using the discretized visual servo error system; generating a performance constraint function based on performance constraints, and constructing a visual servo controller based on the visual servo control model in combination with the performance constraint function; generating an adaptive control law based on the visual servo controller using the error value, and generating an optimized control quantity based on the adaptive control law.

7. An electronic device, the electronic device comprising a processor and a memory, characterized in that, The memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the electronic device executes the visual servo-based robot cooperative control method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, and when the computer instructions run on an electronic device, the electronic device is caused to execute the visual servo-based robot cooperative control method according to any one of claims 1 to 5.

Images