Dynamic vision-based intelligent control method and device for robot arm and brick laying robot

Abstract

The application discloses a kind of based on dynamic vision's mechanical arm intelligent control method, device and brick laying robot, mechanical arm is installed on brick laying robot, camera is installed on brick laying robot, the method includes: in the process that brick laying robot moves to first brick laying point, based on the posture information for brick laying robot collected to camera, determine the target compensation information of brick laying robot;According to target compensation information, first brick laying point is executed correction operation, obtain second brick laying point, and control brick laying robot moves to second brick laying point;When brick laying robot reaches second brick laying point, based on the image information of environment to be laid brick collected to camera, determine the movement control parameter of mechanical arm, and control mechanical arm executes operation matched with movement control parameter.It can be seen that the implementation of the application can realize the intelligent movement control of the mechanical arm of brick laying robot based on dynamic vision, which is beneficial to improve the efficiency and accuracy of brick laying operation of brick laying robot.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology, and in particular to an intelligent control method, device and brick-laying robot for a robotic arm based on dynamic vision. Background Technology

[0002] With the development of technology, in the construction industry, paving robots can now replace traditional manual paving, achieving automated paving and greatly improving efficiency. Most existing paving robots rely on vision detection, and then control the motor module based on the vision detection results to control the robot's robotic arm to perform the paving operation. This results in low efficiency during the paving process. Therefore, providing a new intelligent control method for robotic arm paving to improve the efficiency of paving robots is particularly important. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a method, device and paving robot for intelligent control of a robotic arm based on dynamic vision, which can realize intelligent movement control of the robotic arm of the paving robot based on dynamic vision, which is beneficial to improving the efficiency of the paving robot in paving operations and improving the accuracy of the paving robot in paving operations.

[0004] To address the aforementioned technical problems, the first aspect of this invention discloses an intelligent control method for a robotic arm based on dynamic vision. The robotic arm is mounted on a brick-laying robot, and the brick-laying robot is equipped with a camera. The method includes:

[0005] During the process of the paving robot moving to the first paving point, the target compensation information of the paving robot is determined based on the posture information of the paving robot collected by the camera; the posture information of the paving robot includes the paving robot's brick-grabbing posture information and / or the paving robot's chassis posture information, and the first paving point is the paving point corresponding to the target movement trajectory generated by the trajectory simulation model.

[0006] The first paving point is corrected according to the target compensation information to obtain the second paving point, and the paving robot is controlled to move to the second paving point.

[0007] When the paving robot reaches the second paving point, based on the image information of the environment to be paved captured by the camera, the robot determines the movement control parameters of the robotic arm and controls the robotic arm to perform operations that match the movement control parameters; the image information of the environment to be paved includes real-time image information of the target object.

[0008] As an optional implementation, in the first aspect of the invention, before the paving robot moves to the first paving point, the method further includes:

[0009] Collect target environment information corresponding to the location to be paved, and input the target environment information and the structural information of the paving robot into a preset trajectory simulation model to obtain the target movement trajectory of the paving robot. The target movement trajectory includes the target brick picking point and the first paving point.

[0010] Based on the target movement trajectory, the brick-laying robot is controlled to move to the target brick-picking point. When the brick-laying robot reaches the target brick-picking point, the brick-laying robot is controlled to perform a brick-grabbing operation.

[0011] Determine whether the brick-laying robot has completed the brick-grabbing operation;

[0012] When it is determined that the brick-laying robot has completed the brick-grabbing operation, control the brick-laying robot to move to the first brick-laying point;

[0013] When it is determined that the brick-laying robot has not completed the brick-grabbing operation, the steps of controlling the brick-laying robot to perform the brick-grabbing operation and determining whether the brick-laying robot has completed the brick-grabbing operation are re-triggered.

[0014] As an optional implementation, in the first aspect of the present invention, when the posture information of the paving robot includes the paving robot's brick-grabbing posture information, determining the target compensation information of the paving robot based on the posture information of the paving robot collected by the camera includes:

[0015] Based on the brick-grabbing posture information of the brick-laying robot collected by the camera, it is determined whether the brick-grabbing posture information meets the preset brick-grabbing posture conditions.

[0016] When it is determined that the brick-grabbing posture information does not meet the preset brick-grabbing posture conditions, the first environmental information of the first brick-laying point is obtained.

[0017] Based on the brick-grabbing posture information and the first environmental information, the first error information of the brick-laying robot is determined, and the first compensation information is generated based on the first error information. The first error information is the error information generated by the brick-laying robot during the brick-grabbing operation.

[0018] Based on the first compensation information, the target compensation information of the paving robot is determined.

[0019] As an optional implementation, in the first aspect of the present invention, when the posture information of the paving robot includes the chassis posture information of the paving robot, determining the target compensation information of the paving robot based on the posture information of the paving robot collected by the camera includes:

[0020] Based on the chassis posture information of the paving robot collected by the camera, it is determined whether the chassis posture information meets the preset chassis posture conditions.

[0021] When it is determined that the chassis posture information does not meet the preset chassis posture conditions, the chassis tilt coefficient of the paving robot is determined based on the chassis posture information of the paving robot.

[0022] Based on the chassis tilt coefficient of the paving robot, the second error information of the paving robot is determined, and the second compensation information is generated based on the second error information. The second error information is the chassis tilt error information of the paving robot.

[0023] Based on the second compensation information, the target compensation information of the paving robot is determined.

[0024] As an optional implementation, in the first aspect of the present invention, determining the control parameters of the robotic arm based on the image information of the environment to be paved acquired by the camera includes:

[0025] Based on the real-time image information of the target object captured by the camera, the distance between the first brick position corresponding to the brick to be laid and the pre-determined reference object is calculated, and it is determined whether the distance between the bricks meets the preset brick joint conditions.

[0026] When it is determined that the brick joint distance does not meet the preset brick joint conditions, the dynamic tracking parameters of the robotic arm are determined based on the brick joint distance and the first brick position.

[0027] The movement control parameters of the robotic arm are generated based on the dynamic tracking parameters; the dynamic tracking parameters are used to represent the dynamic relative positional relationship between the robotic arm and the first brick position.

[0028] And, controlling the robotic arm to perform operations matching the movement control parameters includes:

[0029] The robotic arm is moved based on the movement control parameters so that the distance between the first brick position corresponding to the brick to be laid and the pre-determined reference object meets the preset brick joint conditions.

[0030] As an optional implementation, in the first aspect of the invention, after the paving robot reaches the second paving point and before determining the control parameters of the robotic arm based on the image information of the paving environment captured by the camera, the method further includes:

[0031] The camera is controlled to acquire image information of the environment to be paved, and based on visual servoing technology and the image information of the environment to be paved, the paving position relationship between the robotic arm and the ground to be paved is determined, and it is determined whether the paving position relationship meets the preset position conditions; the paving position relationship is used to represent the relative position relationship between the robotic arm and the bricks included in the ground to be paved;

[0032] When it is determined that the brick laying position relationship does not meet the preset position conditions, the operation of determining the control parameters of the robotic arm based on the image information of the brick laying environment collected by the camera is triggered.

[0033] When it is determined that the brick-laying positional relationship meets the preset positional conditions, the brick-laying robot is controlled to perform the brick-laying operation.

[0034] As an optional implementation, in the first aspect of the present invention, the robotic arm is provided with a suction cup for gripping bricks;

[0035] The step of determining whether the brick-laying robot has completed the brick-grabbing operation includes:

[0036] Obtain the vacuum value of the suction cup and determine whether the vacuum value is greater than a preset vacuum threshold.

[0037] When it is determined that the vacuum value is greater than the preset vacuum threshold, it is determined that the brick-laying robot has completed the brick-grabbing operation;

[0038] When it is determined that the vacuum value is not greater than the preset vacuum threshold, it is determined that the brick-laying robot has not completed the brick-grabbing operation.

[0039] A second aspect of this invention discloses a robotic arm intelligent control device based on dynamic vision. The device is applied to a brick-laying robot, the robotic arm is mounted on the robot, and a camera is installed on the robot. The device includes:

[0040] The determination module is used to determine the target compensation information of the paving robot based on the posture information of the paving robot collected by the camera during the process of the paving robot moving to the first paving point; the posture information of the paving robot includes the paving robot's brick-grabbing posture information and / or the paving robot's chassis posture information, and the first paving point is the paving point corresponding to the target movement trajectory generated by the trajectory simulation model.

[0041] The correction module is used to perform a correction operation on the first paving point according to the target compensation information to obtain the second paving point;

[0042] The control module is used to control the brick-laying robot to move to the second brick-laying point;

[0043] The determining module is further configured to determine the movement control parameters of the robotic arm based on the image information of the environment to be paved collected by the camera when the paving robot reaches the second paving point.

[0044] The control module is also used to control the robotic arm to perform operations that match the movement control parameters; the image information of the environment to be paved includes real-time image information of the target object.

[0045] As an optional implementation, in a second aspect of the invention, the apparatus further includes:

[0046] The data acquisition module is used to collect target environmental information corresponding to the location to be paved before the paving robot moves to the first paving point;

[0047] The input module is used to input the target environment information and the structural information of the paving robot into a preset trajectory simulation model to obtain the target movement trajectory of the paving robot, the target movement trajectory including the target brick picking point and the first paving point;

[0048] The control module is also used to control the paving robot to move to the target brick-picking point based on the target movement trajectory, and to control the paving robot to perform a brick-grabbing operation when the paving robot reaches the target brick-picking point;

[0049] The judgment module is used to determine whether the paving robot has completed the brick-grabbing operation; when it is determined that the paving robot has not completed the brick-grabbing operation, the control module is re-triggered to execute the operation of controlling the paving robot to perform the brick-grabbing operation and the operation of determining whether the paving robot has completed the brick-grabbing operation.

[0050] The control module is further configured to control the paving robot to move to the first paving point when the judgment module determines that the paving robot has completed the brick-grabbing operation.

[0051] As an optional implementation, in a second aspect of the present invention, the specific method by which the determining module determines the target compensation information of the paving robot based on the posture information of the paving robot collected by the camera includes:

[0052] When the posture information of the paving robot includes the brick-grabbing posture information of the paving robot, based on the brick-grabbing posture information of the paving robot collected by the camera, it is determined whether the brick-grabbing posture information meets the preset brick-grabbing posture conditions.

[0053] When it is determined that the brick-grabbing posture information does not meet the preset brick-grabbing posture conditions, the first environmental information of the first brick-laying point is obtained.

[0054] Based on the brick-grabbing posture information and the first environmental information, the first error information of the brick-laying robot is determined, and the first compensation information is generated based on the first error information. The first error information is the error information generated by the brick-laying robot during the brick-grabbing operation.

[0055] Based on the first compensation information, the target compensation information of the paving robot is determined.

[0056] As an optional implementation, in a second aspect of the present invention, the specific method by which the determining module determines the target compensation information of the paving robot based on the posture information of the paving robot collected by the camera includes:

[0057] When the posture information of the paving robot includes the chassis posture information of the paving robot, based on the chassis posture information of the paving robot collected by the camera, it is determined whether the chassis posture information meets the preset chassis posture conditions.

[0058] When it is determined that the chassis posture information does not meet the preset chassis posture conditions, the chassis tilt coefficient of the paving robot is determined based on the chassis posture information of the paving robot.

[0059] Based on the chassis tilt coefficient of the paving robot, the second error information of the paving robot is determined, and the second compensation information is generated based on the second error information. The second error information is the chassis tilt error information of the paving robot.

[0060] Based on the second compensation information, the target compensation information of the paving robot is determined.

[0061] As an optional implementation, in a second aspect of the invention, the specific method by which the determining module determines the control parameters of the robotic arm based on the image information of the paving environment acquired by the camera includes:

[0062] Based on the real-time image information of the target object captured by the camera, the distance between the first brick position corresponding to the brick to be laid and the pre-determined reference object is calculated, and it is determined whether the distance between the bricks meets the preset brick joint conditions.

[0063] When it is determined that the brick joint distance does not meet the preset brick joint conditions, the dynamic tracking parameters of the robotic arm are determined based on the brick joint distance and the first brick position.

[0064] The movement control parameters of the robotic arm are generated based on the dynamic tracking parameters; the dynamic tracking parameters are used to represent the dynamic relative positional relationship between the robotic arm and the first brick position.

[0065] Furthermore, the specific methods by which the control module controls the robotic arm to perform operations matching the movement control parameters include:

[0066] The robotic arm is moved based on the movement control parameters so that the distance between the first brick position corresponding to the brick to be laid and the pre-determined reference object meets the preset brick joint conditions.

[0067] As an optional implementation, in a second aspect of the present invention, the control module is further configured to control the camera to collect the image information of the environment to be laid after the paving robot reaches the second paving point and before the determining module determines the control parameters of the robotic arm based on the image information of the environment to be laid collected by the camera.

[0068] The determining module is also used to determine the paving position relationship between the robotic arm and the ground to be paved based on visual servoing technology and the image information of the environment to be paved.

[0069] The judgment module is also used to determine whether the brick laying position relationship meets the preset position conditions; the brick laying position relationship is used to represent the relative position relationship between the robotic arm and the bricks included in the ground to be paved; when it is determined that the brick laying position relationship does not meet the preset position conditions, the determination module is triggered to perform the operation of determining the control parameters of the robotic arm based on the image information of the environment to be paved collected by the camera.

[0070] The control module is further configured to control the paving robot to perform paving operations when the judgment module determines that the paving position relationship meets the preset position conditions.

[0071] As an optional implementation, in a second aspect of the invention, the robotic arm is provided with a suction cup for gripping bricks;

[0072] The specific methods by which the judgment module determines whether the brick-laying robot has completed the brick-grabbing operation include:

[0073] Obtain the vacuum value of the suction cup and determine whether the vacuum value is greater than a preset vacuum threshold.

[0074] When it is determined that the vacuum value is greater than the preset vacuum threshold, it is determined that the brick-laying robot has completed the brick-grabbing operation;

[0075] When it is determined that the vacuum value is not greater than the preset vacuum threshold, it is determined that the brick-laying robot has not completed the brick-grabbing operation.

[0076] A third aspect of this invention discloses another intelligent control device for a robotic arm based on dynamic vision, the device being applied to a brick-laying robot, the device comprising:

[0077] Memory containing executable program code;

[0078] A processor coupled to the memory;

[0079] The processor calls the executable program code stored in the memory to execute the intelligent control method for a robotic arm based on dynamic vision disclosed in the first aspect of the present invention.

[0080] The fourth aspect of this invention discloses a brick-laying robot, wherein the computer storage medium stores computer instructions, and when the computer instructions are invoked, they are used to execute the intelligent control method for a robotic arm based on dynamic vision disclosed in the first aspect of this invention.

[0081] The fifth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute some or all of the steps in any of the intelligent paving methods based on depth cameras disclosed in the first aspect of the present invention.

[0082] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0083] In this embodiment of the invention, during the process of the paving robot moving to the first paving point, target compensation information for the paving robot is determined based on the posture information of the paving robot collected by the camera; a correction operation is performed on the first paving point according to the target compensation information to obtain the second paving point, and the paving robot is controlled to move to the second paving point; when the paving robot reaches the second paving point, the movement control parameters of the robotic arm are determined based on the image information of the environment to be paved collected by the camera, and the robotic arm is controlled to perform operations matching the movement control parameters. It is evident that implementing this invention enables intelligent movement control of the robotic arm of the paving robot based on dynamic vision, which is beneficial to improving the efficiency and accuracy of the paving robot in paving operations. Attached Figure Description

[0084] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0085] Figure 1 This is a flowchart illustrating an intelligent control method for a robotic arm based on dynamic vision, as disclosed in an embodiment of the present invention.

[0086] Figure 2 This is a flowchart illustrating another intelligent control method for a robotic arm based on dynamic vision disclosed in an embodiment of the present invention.

[0087] Figure 3 This is a schematic diagram of the structure of a robotic arm intelligent control device based on dynamic vision disclosed in an embodiment of the present invention;

[0088] Figure 4 This is a schematic diagram of another intelligent control device for a robotic arm based on dynamic vision disclosed in an embodiment of the present invention.

[0089] Figure 5 This is a schematic diagram of the structure of another intelligent control device for a robotic arm based on dynamic vision, as disclosed in an embodiment of the present invention. Detailed Implementation

[0090] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0091] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0092] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0093] This invention discloses a method, device, and paving robot for intelligent control of a robotic arm based on dynamic vision. It can realize intelligent movement control of the robotic arm of the paving robot based on dynamic vision, which is beneficial to improving the efficiency and accuracy of the paving robot in paving operations. The following are detailed descriptions of these methods.

[0094] Example 1

[0095] Please see Figure 1 , Figure 1 This is a flowchart illustrating an intelligent control method for a robotic arm based on dynamic vision, as disclosed in an embodiment of the present invention. Wherein, Figure 1 The described dynamic vision-based intelligent control method for robotic arms can be applied to brick-laying robots, to intelligent control devices for robotic arms based on dynamic vision, and to local or cloud servers for intelligent control of robotic arms based on dynamic vision. This invention does not limit the application of this method. Figure 1 As shown, the intelligent control method for a robotic arm based on dynamic vision can include the following operations:

[0096] 101. During the process of the paving robot moving to the first paving point, the target compensation information of the paving robot is determined based on the posture information of the paving robot collected by the camera.

[0097] In this embodiment of the invention, the posture information of the paving robot includes the paving robot's brick-grabbing posture information and / or the paving robot's chassis posture information, and the first paving point is the paving point corresponding to the target movement trajectory generated by the trajectory simulation model.

[0098] In this embodiment of the invention, optionally, the camera installed on the robot can be a three-dimensional depth camera, or one of a depth camera, a 3D camera, or a binocular camera. This embodiment of the invention does not limit the scope of the camera.

[0099] In this embodiment of the invention, optionally, the brick-grabbing posture information of the brick-laying robot is the posture information between the brick and the robotic arm when the robotic arm of the brick-laying robot grabs the brick; the chassis posture information of the brick-laying robot is the chassis tilt angle information and / or chassis offset information; wherein, the chassis tilt angle information is used to represent the tilt angle between the chassis of the brick-laying robot and the horizontal plane.

[0100] 102. Based on the target compensation information, perform a correction operation on the first paving point to obtain the second paving point, and control the paving robot to move to the second paving point.

[0101] In this embodiment of the invention, optionally, before performing a correction operation on the first paving point based on the target compensation information to obtain the second paving point, the method further includes:

[0102] Determine whether the target compensation information is used to indicate that the first paving point needs to be corrected;

[0103] When it is determined that the target compensation information indicates that the first paving point needs to be corrected, the operation of correcting the first paving point according to the target compensation information is triggered to obtain the second paving point.

[0104] When it is determined that the target compensation information indicates that no correction is needed for the first paving point, the paving robot is controlled to move to the first paving point. When the paving robot moves to the first paving point, the movement control parameters of the robotic arm are determined based on the image information of the environment to be paved captured by the camera, and the robotic arm is controlled to perform operations that match the movement control parameters.

[0105] This allows the paving robot to move directly to the first paving point without needing to correct the target compensation information, improving the efficiency of controlling the robotic arm and thus enhancing the efficiency of the paving robot in performing paving operations.

[0106] 103. When the paving robot reaches the second paving point, based on the image information of the environment to be paved collected by the camera, the robot determines the movement control parameters of the robotic arm and controls the robotic arm to perform operations that match the movement control parameters.

[0107] In this embodiment of the invention, the image information of the environment to be paved includes real-time image information of the target object. The target object includes one or more of the following: the ground to be constructed, and the surface of the bricks to be paved. Optionally, when the target object includes the ground to be constructed, the real-time image information of the target object includes real-time image information of the ground to be constructed; when the target object includes the surface of the bricks to be paved, the real-time image information of the target object includes real-time image information of the surface of the bricks to be paved. Further, the real-time image information of the ground to be constructed includes one or more of the following: real-time image information of the already paved bricks on the ground to be paved, and real-time image information of the corners corresponding to the ground to be paved.

[0108] In this embodiment of the invention, optionally, the movement control parameters of the robotic arm include parallel movement parameters and / or vertical movement parameters; wherein, the parallel movement parameters are used to represent control parameters for controlling the left and right movement of the robotic arm parallel to the ground; and the vertical movement parameters are used to represent control parameters for controlling the up and down movement of the robotic arm parallel to the ground.

[0109] It is evident that implementation Figure 1 The described dynamic vision-based intelligent control method for a robotic arm can determine the target compensation information of the paving robot based on the posture information of the paving robot collected by a camera during its movement to the first paving point. It can adjust the gripping posture of the robotic arm in the paving robot according to the target compensation information, compensating for errors caused by manual brick stacking and improving the accuracy of the paving robot's gripping operation. Furthermore, it can perform a correction operation on the first paving point based on the target compensation information to obtain the second paving point and control the paving robot to move to the second paving point. This method can dynamically adjust the endpoint of the paving robot's movement and improve... This technology improves the accuracy and reliability of determining the endpoint of the paving robot's movement, and enhances the efficiency of this determination. This, in turn, improves the efficiency of the paving robot in performing subsequent paving operations. When the paving robot reaches the second paving point, the movement control parameters of the robotic arm are determined based on the image information of the environment to be paved captured by the camera. The robotic arm is then controlled to perform operations matching these movement control parameters. This technology enables secondary positioning adjustments for the paving robot based on dynamic vision technology, which improves the accuracy and precision of the paving robot's paving operations.

[0110] Example 2

[0111] Please see Figure 2 , Figure 2 This is a flowchart illustrating an intelligent control method for a robotic arm based on dynamic vision, as disclosed in an embodiment of the present invention. Wherein, Figure 2 The described dynamic vision-based intelligent control method for robotic arms can be applied to dynamic vision-based intelligent control devices for robotic arms, or to local servers or cloud servers for dynamic vision-based intelligent control of robotic arms. This invention does not limit the application of this method.

[0112] like Figure 2 As shown, the intelligent control method for a robotic arm based on dynamic vision can include the following operations:

[0113] 201. Collect the target environment information corresponding to the location to be paved, and input the target environment information and the structural information of the paving robot into the preset trajectory simulation model to obtain the target movement trajectory of the paving robot.

[0114] In this embodiment of the invention, the target movement trajectory includes the target brick-picking point and the first brick-laying point.

[0115] In this embodiment of the invention, optionally, the target movement trajectory also includes the movement trajectory of the paving robot from its current position to the target brick-picking point and the movement trajectory of the paving robot from the target brick-picking point to the first paving point.

[0116] In this embodiment of the invention, the target movement trajectory may optionally be one of the following: the shortest path movement trajectory, the shortest movement time movement trajectory, or the movement trajectory with the fewest obstacles to avoid.

[0117] In this embodiment of the invention, optionally, the target environment information corresponding to the location to be paved includes one or more of the following: obstacle location information, obstacle volume information, and obstacle type information in the target environment. Optionally, the structural information of the paving robot includes the appearance model information of the paving robot, wherein the appearance model information of the paving robot includes one or more of the following: robotic arm information, end effector information, and load model information.

[0118] In this embodiment of the invention, optionally, the target brick-picking point is the position corresponding to the brick-laying robot when performing this brick-picking operation; the first brick-laying point is the position corresponding to the brick-laying robot when performing this brick-laying operation in the target movement trajectory generated according to the trajectory simulation model.

[0119] 202. Based on the target movement trajectory, control the paving robot to move to the target brick picking point. When the paving robot reaches the target brick picking point, control the paving robot to perform the brick grabbing operation.

[0120] In this embodiment of the invention, optionally, controlling the brick-laying robot to perform a brick-grabbing operation includes:

[0121] The suction cups in the robotic arm of the brick-laying robot are activated, and the suction cups in the robotic arm are used to pick up bricks.

[0122] In this embodiment of the invention, optionally, the moving distance of the paving robot is calculated by collecting laser data from the collection point; when the calculated moving distance of the paving robot meets the preset distance condition, it is determined that the paving robot has reached the target brick picking point.

[0123] 203. Determine whether the brick-laying robot has completed the brick-grabbing operation.

[0124] In this embodiment of the invention, when it is determined that the paving robot has completed the brick-grabbing operation, step 204 is triggered; when it is determined that the paving robot has not completed the brick-grabbing operation, step 202 is triggered to control the paving robot to perform the brick-grabbing operation and step 203 is triggered to determine whether the paving robot has completed the brick-grabbing operation.

[0125] 204. Control the paving robot to move to the first paving point.

[0126] In this embodiment of the invention, optionally, controlling the paving robot to move to the first paving point includes:

[0127] Based on the target movement trajectory, control the paving robot to perform movement operations that match the target movement trajectory, so that the paving robot moves to the first paving point.

[0128] This allows the paving robot to move to the first paving point based on the target movement trajectory, which improves the accuracy and reliability of the paving robot's movement to the first paving point.

[0129] 205. During the process of the paving robot moving to the first paving point, the target compensation information of the paving robot is determined based on the posture information of the paving robot collected by the camera.

[0130] 206. Based on the target compensation information, perform a correction operation on the first paving point to obtain the second paving point, and control the paving robot to move to the second paving point.

[0131] 207. When the paving robot reaches the second paving point, based on the image information of the environment to be paved collected by the camera, the robot determines the movement control parameters of the robotic arm and controls the robotic arm to perform operations that match the movement control parameters.

[0132] In this embodiment of the invention, for other descriptions of steps 205-207, please refer to the detailed description of steps 101-103 in Embodiment 1. These descriptions will not be repeated in this embodiment of the invention.

[0133] It is evident that implementation Figure 2 The described intelligent control method for a robotic arm based on dynamic vision can collect target environmental information corresponding to the location where bricks are to be laid. This target environmental information, along with the structural information of the brick-laying robot, is input into a preset trajectory simulation model to obtain the target movement trajectory of the brick-laying robot. This method can generate the target movement trajectory of the brick-laying robot based on the trajectory simulation model, improving the intelligence, accuracy, and reliability of the generated trajectory. Controlling the brick-laying robot to move to the target brick-grabbing point based on the target movement trajectory and then controlling the robot to perform the brick-grabbing operation improves the accuracy of the brick-laying operation. Furthermore, controlling the brick-laying robot to move to the first brick-laying point only after determining that the brick-grabbing operation has been completed further improves the precision and reliability of the brick-grabbing and movement operations, reducing the occurrence of brick-grabbing failures, and thus improving the efficiency and accuracy of the brick-laying robot's brick-laying operations.

[0134] In an optional embodiment, when the posture information of the paving robot includes the paving robot's brick-grabbing posture information, target compensation information for the paving robot is determined based on the posture information of the paving robot collected by the camera, including:

[0135] Based on the brick-grabbing posture information of the brick-laying robot collected by the camera, it is determined whether the brick-grabbing posture information meets the preset brick-grabbing posture conditions.

[0136] When it is determined that the brick-grabbing posture information does not meet the preset brick-grabbing posture conditions, the first environmental information of the first brick-laying point is obtained.

[0137] Based on the brick-grabbing posture information and the first environmental information, the first error information of the brick-laying robot is determined, and the first compensation information is generated based on the first error information. The first error information is the error information generated by the brick-laying robot during the brick-grabbing operation.

[0138] Based on the first compensation information, the target compensation information for the brick-laying robot is determined.

[0139] In this optional embodiment, the camera installed on the paving robot can be located on the chassis of the paving robot, and the number of cameras can be four. The specific number of cameras is not limited in this embodiment of the invention.

[0140] In this optional embodiment, optionally, determining whether the brick-grabbing posture information meets preset brick-grabbing posture conditions includes:

[0141] Based on the brick-grabbing posture information, the brick tilt angle information and the brick horizontal difference information of the brick-laying robot are determined. The brick tilt angle information is used to represent the tilt angle between the brick and the horizontal line, and the brick horizontal difference information is used to represent the difference distance between the brick and the preset standard grasping point of the brick-laying robot.

[0142] Determine whether the tilt angle information of the bricks picked up by the paving robot meets the preset brick tilt conditions and whether the horizontal difference information of the bricks picked up by the paving robot meets the preset brick difference distance conditions.

[0143] When it is determined that the tilt angle information of the bricks grasped by the paving robot meets the preset brick tilt condition and the horizontal difference information of the bricks grasped by the paving robot meets the preset brick difference distance condition, the brick grasping posture information is determined to meet the preset brick grasping posture condition.

[0144] When it is determined that the tilt angle information of the bricks grasped by the paving robot does not meet the preset brick tilt conditions and / or the horizontal difference information of the bricks grasped by the paving robot does not meet the preset brick difference distance conditions, it is determined that the brick grasping posture information does not meet the preset brick grasping posture conditions.

[0145] In this optional embodiment, optionally, the target compensation information for the paving robot is determined based on the first compensation information, including:

[0146] The first compensation information is determined as the target compensation information for the brick-laying robot.

[0147] In this optional embodiment, the first error information may include tilt angle error information and / or horizontal distance error information generated by the brick-laying robot during the brick-grabbing operation.

[0148] In this optional embodiment, the first environmental information of the first paving point may include the ground information corresponding to the first paving point, wherein the ground information corresponding to the first paving point includes information on the bricks already laid on the ground.

[0149] In this optional embodiment, the process can be terminated when it is determined that the brick-grabbing posture information meets the preset brick-grabbing posture conditions.

[0150] As can be seen, implementing this optional embodiment can determine whether the brick-grabbing posture information of the paving robot, collected by the camera, meets the preset brick-grabbing posture conditions. If it does, the first environmental information of the first paving point is obtained, and the first error information of the paving robot is determined based on the brick-grabbing posture information and the first environmental information. The first compensation information is then generated based on the first error information to determine the target compensation information of the paving robot. This method can determine the robot's first error information based on the brick-grabbing posture information and the first environmental information collected by the camera, which can improve the accuracy and reliability of determining the first error information and the intelligence of determining the first error information. This is beneficial to improving the accuracy and reliability of generating the first compensation information and the accuracy and reliability of determining the target compensation information of the paving robot. It can also achieve dynamic compensation based on the intelligent control of the robot using dynamic vision, which is beneficial to improving the accuracy and reliability of controlling the paving robot.

[0151] In another optional embodiment, when the posture information of the paving robot includes the chassis posture information of the paving robot, target compensation information for the paving robot is determined based on the posture information of the paving robot collected by the camera, including:

[0152] Based on the chassis posture information of the brick-laying robot collected by the camera, it is determined whether the chassis posture information meets the preset chassis posture conditions.

[0153] When it is determined that the chassis posture information does not meet the preset chassis posture conditions, the chassis tilt coefficient of the paving robot is determined based on the chassis posture information of the paving robot.

[0154] Based on the chassis tilt coefficient of the paving robot, the second error information of the paving robot is determined, and the second compensation information is generated based on the second error information. The second error information is the chassis tilt error information of the paving robot.

[0155] Based on the second compensation information, the target compensation information for the brick-laying robot is determined.

[0156] In this optional embodiment, optionally, the target compensation information for the paving robot is determined based on the second compensation information, including:

[0157] The second compensation information is determined as the target compensation information for the brick-laying robot.

[0158] In this optional embodiment, optionally, when the posture information of the paving robot includes the chassis posture information and the brick-grabbing posture information of the paving robot, the target compensation information of the paving robot is determined based on the posture information of the paving robot collected by the camera, including:

[0159] Based on the first compensation information and the second compensation information, the target compensation information for the brick-laying robot is determined.

[0160] Optionally, based on the first compensation information and the second compensation information, the target compensation information for the paving robot is determined, including:

[0161] The first compensation information and the second compensation information are determined as the target compensation information for the paving robot.

[0162] This allows the target compensation information of the paving robot to be determined based on the first and second compensation information when the robot's posture information includes both chassis posture information and brick-grabbing posture information. This improves the accuracy and comprehensiveness of the target compensation information, thereby enhancing the accuracy and intelligence of controlling the paving robot.

[0163] In this optional embodiment, the chassis tilt error information of the paving robot may optionally include one or more of the following: the angular tilt error information of the chassis of the paving robot relative to the horizontal plane, and the distance error information of the chassis of the paving robot relative to the ground.

[0164] In this optional embodiment, the process can be terminated when it is determined that the chassis attitude information meets the preset chassis attitude conditions.

[0165] In this optional embodiment, when the posture information of the tiling robot includes the tiling robot's gripping posture information and the chassis posture information, cameras are installed on both the robotic arm and the chassis of the tiling robot. The number of cameras installed on the chassis can be two sets, with one or two cameras in each set; the specific number of cameras on the chassis is not limited in this embodiment. Optionally, when there are two sets of cameras installed on the chassis, the two sets of cameras are respectively positioned in front and behind along the chassis's travel direction, and the cameras are used to detect the chassis posture information. That is, based on the real-time image information collected by the two sets of cameras installed on the chassis, the edge line of the laid tiles or a preset reference baseline can be identified. Based on the identification result, the travel trajectory of the tiling robot is determined, i.e., the travel trajectory of the tiling robot is determined according to the identification result, and the determined travel trajectory of the tiling robot is along the edge line of the laid tiles or the preset reference baseline. Furthermore, based on multiple cameras mounted on the robotic arm and chassis of the paving robot, real-time image information of both the robotic arm and chassis can be acquired. This combined real-time image information is then used to control the robot's movement and paving operations. In this way, the multiple cameras on the robot's chassis ensure that it moves in a straight line along the edge of the laid bricks or a preset reference line, improving the accuracy of controlling the robot's movement along its trajectory. This, in turn, enhances the precision of the paving robot's movement and, consequently, the accuracy of its paving operations.

[0166] As can be seen, implementing this optional embodiment can determine whether the chassis posture information of the paving robot meets the preset chassis posture conditions based on the chassis posture information collected by the camera. If it does not meet the conditions, the chassis tilt coefficient is determined based on the chassis posture information of the paving robot, and the second error information is determined based on the chassis tilt coefficient to generate the second compensation information. Then, the target compensation information of the paving robot is determined based on the second compensation information. It can determine the second error information of the robot based on the chassis posture information of the paving robot collected by the camera, which can improve the accuracy and reliability of determining the second error information and improve the intelligence of determining the second error information. This is conducive to improving the accuracy and reliability of generating the second compensation information and the accuracy and reliability of determining the target compensation information of the paving robot. It can achieve dynamic compensation based on dynamic vision for intelligent control of the robot, which is conducive to improving the accuracy and reliability of controlling the paving robot.

[0167] In another optional embodiment, the control parameters of the robotic arm are determined based on the image information of the environment to be paved captured by the camera, including:

[0168] Based on the real-time image information of the target object captured by the camera, the distance between the first brick position corresponding to the brick to be laid and the pre-determined reference object is calculated, and it is determined whether the distance between the bricks meets the preset brick joint conditions.

[0169] When it is determined that the brick joint distance does not meet the preset brick joint conditions, the dynamic tracking parameters of the robotic arm are determined based on the brick joint distance and the first brick position.

[0170] The movement control parameters of the robotic arm are generated based on the dynamic tracking parameters; the dynamic tracking parameters are used to represent the dynamic relative positional relationship between the robotic arm and the first brick position.

[0171] And, controlling the robotic arm to perform operations that match the movement control parameters, including:

[0172] The movement of the robotic arm is controlled based on the motion control parameters so that the distance between the first brick position corresponding to the brick to be laid and the pre-determined reference object meets the preset brick joint conditions.

[0173] In this optional embodiment, the predetermined reference object may be one of the following: bricks that have been laid in the ground to be constructed, reference bricks placed in advance in the ground to be constructed, standard lines used to indicate the edges of the laid bricks, or standard objects used to indicate the edges of the laid bricks.

[0174] In this optional embodiment, the brick joint distance can be used to represent the distance between the brick corresponding to the first brick position and a predetermined reference object.

[0175] In this optional embodiment, the predetermined reference point can be an adjacent brick adjacent to the brick to be paved. Furthermore, the number of adjacent bricks can be one or more. When the number of adjacent bricks is multiple, the adjacent bricks can be located at one of the positions above, below, to the left, or to the right of the brick to be paved. Optionally, when there are multiple adjacent bricks to be laid, each adjacent brick has a corresponding second brick position. The calculation of the brick joint distance between the first brick position corresponding to the brick to be laid and a pre-determined reference object, and the determination of whether the brick joint distance meets preset brick joint conditions, includes: calculating the target brick joint distance between the first brick position corresponding to the brick to be laid and each second brick position of each adjacent brick of the brick to be laid; determining whether all target brick joint distances meet preset brick joint conditions; when it is determined that all target brick joint distances meet preset brick joint conditions, the brick joint distance is determined to meet preset brick joint conditions; when it is determined that all target brick joint distances do not meet preset brick joint conditions, the brick joint distance is determined not to meet preset brick joint conditions. This method can calculate the target grout distance between the first grout point of the paving brick and the second grout point of each adjacent brick when there are multiple adjacent bricks to be paved. Based on all the target grout distances, it can determine whether the grout distance meets the preset grout conditions. This can improve the accuracy and reliability of determining whether the grout distance meets the preset grout conditions, and improve the intelligence and comprehensiveness of the determination.

[0176] In this optional embodiment, the dynamic tracking parameters can be used to represent the dynamic relative positional relationship between the robotic arm and the first brick point during the movement of the robotic arm.

[0177] In this optional embodiment, the method may further include:

[0178] When it is determined that the distance between the brick joints meets the preset brick joint conditions, the brick-laying robot is controlled to perform the brick-laying operation.

[0179] As can be seen, implementing this optional embodiment can calculate the distance between the first brick position corresponding to the brick to be laid and the pre-determined reference object based on real-time ground image information, and determine whether the distance between the bricks meets the preset brick joint conditions. If not, the dynamic tracking parameters of the robotic arm are determined and the movement control parameters of the robotic arm are further generated. This can improve the accuracy and reliability of determining whether the distance between the bricks meets the preset brick joint conditions, thereby improving the accuracy and reliability of determining the dynamic tracking parameters of the robotic arm, and further improving the accuracy and reliability of generating the movement control parameters of the robotic arm. Furthermore, controlling the movement of the robotic arm based on the movement control parameters can ensure that the distance between the first brick position corresponding to the brick to be laid and the pre-determined reference object meets the preset brick joint conditions, thereby improving the accuracy and stability of the brick-laying robot in performing the brick-laying operation. In addition, generating the movement control parameters of the robotic arm based on the dynamic tracking parameters can achieve rapid dynamic tracking movement, thereby improving the efficiency and accuracy of controlling the movement of the robotic arm, and further improving the efficiency and accuracy of the brick-laying robot in performing the brick-laying operation.

[0180] In yet another optional embodiment, after the paving robot reaches the second paving point and before determining the control parameters of the robotic arm based on the image information of the paving environment captured by the camera, the method further includes:

[0181] The system controls the camera to capture image information of the environment to be paved, and based on visual servoing technology and the image information of the environment to be paved, determines the paving position relationship between the robotic arm and the ground to be paved, and judges whether the paving position relationship meets the preset position conditions; the paving position relationship is used to represent the relative position relationship between the robotic arm and the bricks included in the ground to be paved.

[0182] When it is determined that the brick-laying position relationship does not meet the preset position conditions, the operation of determining the control parameters of the robotic arm is triggered based on the image information of the brick-laying environment captured by the camera.

[0183] When it is determined that the bricklaying positional relationship meets the preset positional conditions, the bricklaying robot is controlled to perform the bricklaying operation.

[0184] In this optional embodiment, the image information of the environment to be paved may include the image information of the ground to be paved; wherein, the image information of the ground to be paved may include one or more of the following: the information of the paved ground and the information of the corners of the ground to be paved.

[0185] In this optional embodiment, it should be noted that visual servoing technology is the behavior of automatically receiving and processing an image of a real object through optical devices and non-contact sensors, and using the information fed back from the image to enable the machine system to make further control or corresponding adaptive adjustments to the machine.

[0186] In this optional embodiment, the relative position between the robotic arm and the bricks included in the paved ground may include one or more of the following: the horizontal distance between the robotic arm and the bricks included in the paved ground, and the tilt angle between the robotic arm and the bricks included in the paved ground.

[0187] In this optional embodiment, determining whether the positional relationship of the paving tiles meets preset positional conditions may include:

[0188] Determine whether the brick-laying position relationship is used to indicate that the horizontal distance between the robotic arm and the bricks included in the ground to be paved is greater than a preset distance threshold and the tilt angle between the robotic arm and the bricks included in the ground to be paved is less than a preset angle threshold.

[0189] When it is determined that the horizontal distance between the robotic arm and the bricks in the ground to be paved is greater than a preset distance threshold and the tilt angle between the robotic arm and the bricks in the ground to be paved is less than a preset angle threshold, the paving position relationship is determined to meet the preset position conditions.

[0190] When it is determined that the horizontal distance between the robotic arm and the bricks in the paving area is not greater than a preset distance threshold and / or the tilt angle between the robotic arm and the bricks in the paving area is not less than a preset angle threshold, the paving position relationship is determined to not meet the preset position conditions.

[0191] In this optional embodiment, the visual servoing technology may include a visual servoing mode for the camera module and / or a visual servoing mode for the robotic arm module. When the robotic arm moves to the second paving point, the visual servoing mode of the camera module is activated. After the camera module's visual servoing mode is successfully activated, the visual servoing mode of the robotic arm module is activated. This allows the robotic arm to adjust its movement trajectory using real-time visual data acquired through the visual servoing mode of the robotic arm module. If the adjustment is successful, the visual servoing mode of the robotic arm module is deactivated, and then the visual servoing mode of the camera module is deactivated. This allows control of the activation and deactivation of the visual servoing mode of the robotic arm module based on the camera module's visual servoing mode, improving the accuracy and reliability of using the visual servoing mode, and consequently, enhancing the precision of trajectory adjustment for the paving robot based on the visual servoing mode.

[0192] As can be seen, implementing this optional embodiment can determine the brick-laying positional relationship between the robotic arm and the ground to be paved based on visual servoing technology and the image information of the environment to be paved, and determine whether the brick-laying positional relationship meets the preset positional conditions. If not, the control parameters of the robotic arm are determined based on the image information of the environment to be paved. If they are met, the brick-laying robot is controlled to perform the brick-laying operation. The ability to determine the brick-laying positional relationship based on visual servoing technology and the image information of the environment to be paved improves the accuracy of determining the brick-laying positional relationship, thereby improving the accuracy and reliability of determining whether the brick-laying positional relationship meets the preset positional conditions. This, in turn, helps to improve the accuracy of determining the control parameters of the robotic arm and improves the efficiency of controlling the brick-laying robot to perform the brick-laying operation.

[0193] In yet another alternative embodiment, the robotic arm is equipped with suction cups for gripping bricks;

[0194] Determining whether the brick-laying robot has completed the brick-grabbing operation includes:

[0195] Obtain the vacuum value of the suction cup and determine whether the vacuum value is greater than the preset vacuum threshold.

[0196] When the vacuum value is determined to be greater than the preset vacuum threshold, it is determined that the brick-laying robot has completed the brick-grabbing operation;

[0197] If the vacuum value is determined to be no greater than the preset vacuum threshold, it is determined that the brick-laying robot has not completed the brick-grabbing operation.

[0198] In this optional embodiment, optionally, after determining that the paving robot has not completed the brick-grabbing operation, the method further includes:

[0199] The suction cup on the robotic arm is controlled to re-execute the operation of grabbing the brick, and the step of judging whether the brick-laying robot has completed the brick-grabbing operation is triggered again, until it is determined that the vacuum value of the suction cup is greater than the preset vacuum threshold.

[0200] As can be seen, implementing this optional embodiment can determine whether the vacuum value obtained from the suction cup is greater than a preset vacuum threshold. If it is greater, it is determined that the paving robot has completed the brick-grabbing operation; if it is not greater, it is determined that the paving robot has not completed the brick-grabbing operation. This can improve the accuracy and reliability of determining whether the paving robot has completed the brick-grabbing operation, thereby reducing the occurrence of brick-grabbing failures and improving the efficiency and accuracy of the paving robot in performing paving operations.

[0201] Example 3

[0202] Please see Figure 3 , Figure 3This is a schematic diagram of another intelligent control device for a robotic arm based on dynamic vision disclosed in an embodiment of the present invention, wherein, Figure 3 The described device is used to control a brick-laying robot, and a robotic arm is mounted on the brick-laying robot. The brick-laying robot is equipped with a camera. Optionally, the device can be integrated into the brick-laying robot or exist independently of it, such as being located on a cloud control platform. This embodiment of the invention does not impose limitations. Figure 3 As shown, the intelligent control device for the robotic arm based on dynamic vision may include:

[0203] The determination module 301 is used to determine the target compensation information of the paving robot based on the posture information of the paving robot collected by the camera during the process of the paving robot moving to the first paving point. The posture information of the paving robot includes the paving robot's brick-grabbing posture information and / or the paving robot's chassis posture information. The first paving point is the paving point corresponding to the target movement trajectory generated by the trajectory simulation model.

[0204] Correction module 302 is used to perform a correction operation on the first paving point according to the target compensation information to obtain the second paving point;

[0205] Control module 303 is used to control the brick-laying robot to move to the second brick-laying point;

[0206] The determination module 301 is also used to determine the movement control parameters of the robotic arm based on the image information of the environment to be paved collected by the camera when the paving robot reaches the second paving point.

[0207] The control module 303 is also used to control the robotic arm to perform operations that match the movement control parameters; the image information of the environment to be paved includes real-time ground image information of the ground to be paved.

[0208] It is evident that implementation Figure 3The described device, during the process of a brick-laying robot moving to the first brick-laying point, determines the target compensation information of the brick-laying robot based on the posture information of the brick-laying robot collected by a camera. It can adjust the brick-grabbing posture of the robotic arm in the brick-laying robot according to the target compensation information, compensating for errors caused by manual brick stacking and improving the accuracy of the brick-grabbing operation. It also performs a correction operation on the first brick-laying point based on the target compensation information to obtain a second brick-laying point and controls the brick-laying robot to move to the second brick-laying point. This dynamic adjustment of the brick-laying robot's endpoint improves the accuracy and reliability of determining the corresponding endpoint, as well as the efficiency of determining the endpoint, thereby improving the efficiency of the subsequent brick-laying operation. When the brick-laying robot reaches the second brick-laying point, it determines the movement control parameters of the robotic arm based on the image information of the environment to be laid collected by the camera and controls the robotic arm to perform operations matching the movement control parameters. It can perform secondary positioning adjustments on the brick-laying robot based on dynamic vision technology, further improving the accuracy and precision of the brick-laying operation.

[0209] In an optional embodiment, such as Figure 4 As shown, the device also includes:

[0210] The data acquisition module 304 is used to collect target environment information corresponding to the location to be paved before the paving robot moves to the first paving point;

[0211] The input module 305 is used to input the target environment information and the structural information of the paving robot into the preset trajectory simulation model to obtain the target movement trajectory of the paving robot. The target movement trajectory includes the target brick picking point and the first brick laying point.

[0212] The control module 303 is also used to control the paving robot to move to the target brick picking point based on the target movement trajectory, and to control the paving robot to perform a brick grabbing operation when it reaches the target brick picking point.

[0213] The judgment module 306 is used to determine whether the brick-laying robot has completed the brick-grabbing operation; when it is determined that the brick-laying robot has not completed the brick-grabbing operation, the control module 303 is re-triggered to perform the brick-grabbing operation and the operation of determining whether the brick-laying robot has completed the brick-grabbing operation.

[0214] The control module 303 is also used to control the paving robot to move to the first paving point when the judgment module 306 determines that the paving robot has completed the brick-grabbing operation.

[0215] It is evident that implementation Figure 4The described device can collect target environmental information corresponding to the location to be paved, input the target environmental information and the structural information of the paving robot into a preset trajectory simulation model to obtain the target movement trajectory of the paving robot. It can generate the target movement trajectory of the paving robot based on the trajectory simulation model, improving the intelligence, accuracy, and reliability of the generated trajectory. Based on the target movement trajectory, it controls the paving robot to move to the target brick-grabbing point and performs the brick-grabbing operation, improving the accuracy of the paving robot's brick-laying operation. Furthermore, it only controls the paving robot to move to the first paving point after determining that the brick-grabbing operation has been completed, further improving the accuracy and reliability of the paving robot's brick-grabbing and movement operations, reducing the occurrence of brick-grabbing failures, and thus improving the efficiency and accuracy of the paving robot's brick-laying operation.

[0216] In another optional embodiment, the specific method by which the determining module 301 determines the target compensation information of the paving robot based on the posture information of the paving robot collected by the camera includes:

[0217] When the posture information of the paving robot includes the paving robot's brick-grabbing posture information, based on the brick-grabbing posture information of the paving robot collected by the camera, it is determined whether the brick-grabbing posture information meets the preset brick-grabbing posture conditions.

[0218] When it is determined that the brick-grabbing posture information does not meet the preset brick-grabbing posture conditions, the first environmental information of the first brick-laying point is obtained.

[0219] Based on the brick-grabbing posture information and the first environmental information, the first error information of the brick-laying robot is determined, and the first compensation information is generated based on the first error information. The first error information is the error information generated by the brick-laying robot during the brick-grabbing operation.

[0220] Based on the first compensation information, the target compensation information for the brick-laying robot is determined.

[0221] It is evident that implementation Figure 4The described device can determine whether the brick-grabbing posture information of the paving robot, acquired by a camera, meets preset brick-grabbing posture conditions. If it does, it acquires the first environmental information of the first paving point, determines the first error information of the paving robot based on the brick-grabbing posture information and the first environmental information, and generates first compensation information based on the first error information to determine the target compensation information of the paving robot. This device can determine the robot's first error information based on the brick-grabbing posture information and the first environmental information acquired by the camera, improving the accuracy and reliability of determining the first error information, as well as its intelligence. This, in turn, improves the accuracy and reliability of generating the first compensation information and the accuracy and reliability of determining the target compensation information of the paving robot. Furthermore, it enables dynamic compensation based on dynamic vision for intelligent control of the robot, thereby improving the accuracy and reliability of controlling the paving robot.

[0222] In yet another optional embodiment, the specific method by which the determining module 301 determines the target compensation information of the paving robot based on the posture information of the paving robot collected by the camera includes:

[0223] When the posture information of the paving robot includes the posture information of the chassis of the paving robot, the chassis posture information collected by the camera is used to determine whether the chassis posture information meets the preset chassis posture conditions.

[0224] When it is determined that the chassis posture information does not meet the preset chassis posture conditions, the chassis tilt coefficient of the paving robot is determined based on the chassis posture information of the paving robot.

[0225] Based on the chassis tilt coefficient of the paving robot, the second error information of the paving robot is determined, and the second compensation information is generated based on the second error information. The second error information is the chassis tilt error information of the paving robot.

[0226] Based on the second compensation information, the target compensation information for the brick-laying robot is determined.

[0227] It is evident that implementation Figure 4The described device can determine whether the chassis posture information of a paving robot meets preset chassis posture conditions based on the chassis posture information collected by a camera. If not, it determines the chassis tilt coefficient based on the chassis posture information of the paving robot, and determines second error information based on the chassis tilt coefficient to generate second compensation information. Then, it determines the target compensation information of the paving robot based on the second compensation information. It can determine the second error information of the robot based on the chassis posture information of the paving robot collected by the camera, which can improve the accuracy and reliability of determining the second error information, as well as the intelligence of determining the second error information. This is conducive to improving the accuracy and reliability of generating the second compensation information and the accuracy and reliability of determining the target compensation information of the paving robot. It can achieve dynamic compensation based on dynamic vision for intelligent control of the robot, which is conducive to improving the accuracy and reliability of controlling the paving robot.

[0228] In yet another optional embodiment, the specific method by which the determining module 301 determines the control parameters of the robotic arm based on the image information of the environment to be paved captured by the camera includes:

[0229] Based on the real-time ground image information of the ground to be paved captured by the camera, the distance between the first brick position corresponding to the brick to be paved and the pre-determined reference object is calculated, and it is determined whether the distance between the bricks meets the preset brick joint conditions.

[0230] When it is determined that the brick joint distance does not meet the preset brick joint conditions, the dynamic tracking parameters of the robotic arm are determined based on the brick joint distance and the first brick position.

[0231] The movement control parameters of the robotic arm are generated based on the dynamic tracking parameters; the dynamic tracking parameters are used to represent the dynamic relative positional relationship between the robotic arm and the first brick position.

[0232] Furthermore, the specific methods by which the control module 303 controls the robotic arm to perform operations matching the movement control parameters include:

[0233] The movement of the robotic arm is controlled based on the motion control parameters so that the distance between the first brick position corresponding to the brick to be laid and the pre-determined reference object meets the preset brick joint conditions.

[0234] It is evident that implementation Figure 4The described device can calculate the distance between the first brick position corresponding to the brick to be laid and a pre-determined reference object based on real-time ground image information, and determine whether the distance between the bricks meets the preset brick joint conditions. If not, it determines the dynamic tracking parameters of the robotic arm and further generates the movement control parameters of the robotic arm. This improves the accuracy and reliability of determining whether the distance between the bricks meets the preset brick joint conditions, thereby improving the accuracy and reliability of determining the dynamic tracking parameters of the robotic arm, and further improving the accuracy and reliability of generating the movement control parameters of the robotic arm. Furthermore, controlling the movement of the robotic arm based on the movement control parameters ensures that the distance between the first brick position corresponding to the brick to be laid and the pre-determined reference object meets the preset brick joint conditions, improving the accuracy and stability of the brick-laying robot in performing the brick-laying operation. Additionally, generating the movement control parameters of the robotic arm based on the dynamic tracking parameters enables rapid dynamic tracking movement, thereby improving the efficiency and accuracy of controlling the movement of the robotic arm, and further improving the efficiency and accuracy of the brick-laying robot in performing the brick-laying operation.

[0235] In another optional embodiment, the control module 303 is further configured to control the camera to collect the image information of the environment to be laid after the paving robot reaches the second paving point and before the determining module 301 determines the control parameters of the robotic arm based on the image information of the environment to be laid collected by the camera.

[0236] The determination module 301 is also used to determine the paving position relationship between the robotic arm and the ground to be paved based on visual servoing technology and image information of the environment to be paved.

[0237] The judgment module 306 is also used to judge whether the brick laying position relationship meets the preset position conditions; the brick laying position relationship is used to represent the relative position relationship between the robotic arm and the bricks included in the ground to be paved; when it is judged that the brick laying position relationship does not meet the preset position conditions, the determination module 301 is triggered to perform the operation of determining the control parameters of the robotic arm based on the image information of the brick laying environment collected by the camera.

[0238] The control module 303 is also used to control the paving robot to perform paving operations when the judgment module 306 determines that the paving position relationship meets the preset position conditions.

[0239] It is evident that implementation Figure 4The described device can determine the brick-laying positional relationship between the robotic arm and the ground to be paved based on visual servoing technology and image information of the environment to be paved. It can also determine whether the brick-laying positional relationship meets preset positional conditions. If not, it determines the control parameters of the robotic arm based on the image information of the environment to be paved. If the conditions are met, it controls the brick-laying robot to perform the brick-laying operation. The ability to determine the brick-laying positional relationship based on visual servoing technology and image information of the environment to be paved improves the accuracy of determining the brick-laying positional relationship, thereby improving the accuracy and reliability of judging whether the brick-laying positional relationship meets the preset positional conditions. This, in turn, helps to improve the accuracy of determining the control parameters of the robotic arm and improves the efficiency of controlling the brick-laying robot to perform the brick-laying operation.

[0240] In yet another alternative embodiment, the robotic arm is equipped with suction cups for gripping bricks;

[0241] The specific methods by which the judgment module 306 determines whether the brick-laying robot has completed the brick-grabbing operation include:

[0242] Obtain the vacuum value of the suction cup and determine whether the vacuum value is greater than the preset vacuum threshold.

[0243] When the vacuum value is determined to be greater than the preset vacuum threshold, it is determined that the brick-laying robot has completed the brick-grabbing operation;

[0244] If the vacuum value is determined to be no greater than the preset vacuum threshold, it is determined that the brick-laying robot has not completed the brick-grabbing operation.

[0245] It is evident that implementation Figure 4 The described device can determine whether the vacuum value of the suction cup is greater than a preset vacuum threshold based on the obtained vacuum value. If it is greater, it determines that the paving robot has completed the brick-grabbing operation; if it is not greater, it determines that the paving robot has not completed the brick-grabbing operation. This improves the accuracy and reliability of determining whether the paving robot has completed the brick-grabbing operation, thereby reducing the occurrence of brick-grabbing failures and improving the efficiency and accuracy of the paving robot in performing paving operations.

[0246] Example 4

[0247] Please see Figure 5 , Figure 5 This is a schematic diagram of another intelligent control device for a robotic arm based on dynamic vision, as disclosed in an embodiment of the present invention. Figure 5 The described device is used to control a brick-laying robot, and a robotic arm is mounted on the brick-laying robot. The brick-laying robot is equipped with a camera. Optionally, the device can be integrated into the brick-laying robot or exist independently of it, such as being located on a cloud control platform. This embodiment of the invention does not impose limitations. Figure 5As shown, the intelligent control device for the robotic arm based on dynamic vision may include:

[0248] Memory 401 storing executable program code;

[0249] Processor 402 coupled to memory 401;

[0250] The processor 402 calls the executable program code stored in the memory 401 to execute the steps in the intelligent control method for a robotic arm based on dynamic vision described in Embodiment 1 or Embodiment 2 of the present invention.

[0251] Example 5

[0252] This invention discloses a computer-storable medium storing computer instructions. When these computer instructions are invoked, they are used to execute the steps in the dynamic vision-based intelligent control method for robotic arms described in Embodiment 1 or Embodiment 2 of this invention.

[0253] Example 6

[0254] This invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps in the dynamic vision-based intelligent control method for robotic arms described in Embodiment 1 or Embodiment 2.

[0255] Example 7

[0256] This invention discloses a brick-laying robot equipped with a robotic arm and a camera. The brick-laying robot performs some or all of the steps in any of the dynamic vision-based intelligent control methods for robotic arms disclosed in Embodiments 1-2 of this invention; alternatively, the brick-laying robot may include any of the dynamic vision-based intelligent control devices for robotic arms described in Embodiment 3.

[0257] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0258] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0259] Finally, it should be noted that the intelligent control method, device, and paving robot for a robotic arm based on dynamic vision disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent control of a robotic arm based on dynamic vision, characterized in that, The robotic arm is mounted on a brick-laying robot, and the brick-laying robot is equipped with a camera. The method includes: During the process of the paving robot moving to the first paving point, the target compensation information of the paving robot is determined based on the posture information of the paving robot collected by the camera; the posture information of the paving robot includes the paving robot's brick-grabbing posture information and / or the paving robot's chassis posture information, and the first paving point is the paving point corresponding to the target movement trajectory generated by the trajectory simulation model. The first paving point is corrected according to the target compensation information to obtain the second paving point, and the paving robot is controlled to move to the second paving point. When the paving robot reaches the second paving point, based on the image information of the environment to be paved captured by the camera, the robot determines the movement control parameters of the robotic arm and controls the robotic arm to perform operations that match the movement control parameters; the image information of the environment to be paved includes real-time image information of the target object; Before the paving robot moves to the first paving point, the method further includes: Collect target environment information corresponding to the location to be paved, and input the target environment information and the structural information of the paving robot into a preset trajectory simulation model to obtain the target movement trajectory of the paving robot. The target movement trajectory includes the target brick picking point and the first paving point. Based on the target movement trajectory, the brick-laying robot is controlled to move to the target brick-picking point. When the brick-laying robot reaches the target brick-picking point, the brick-laying robot is controlled to perform a brick-grabbing operation. Determine whether the brick-laying robot has completed the brick-grabbing operation; When it is determined that the brick-laying robot has completed the brick-grabbing operation, control the brick-laying robot to move to the first brick-laying point; When it is determined that the brick-laying robot has not completed the brick-grabbing operation, the steps of controlling the brick-laying robot to perform the brick-grabbing operation and determining whether the brick-laying robot has completed the brick-grabbing operation are re-triggered. The robotic arm is equipped with a suction cup, which is used to grip bricks. The step of determining whether the brick-laying robot has completed the brick-grabbing operation includes: Obtain the vacuum value of the suction cup and determine whether the vacuum value is greater than a preset vacuum threshold. When it is determined that the vacuum value is greater than the preset vacuum threshold, it is determined that the brick-laying robot has completed the brick-grabbing operation; When it is determined that the vacuum value is not greater than the preset vacuum threshold, it is determined that the brick-laying robot has not completed the brick-grabbing operation.

2. The intelligent control method for a robotic arm based on dynamic vision according to claim 1, characterized in that, When the posture information of the paving robot includes the paving robot's brick-grabbing posture information, determining the target compensation information of the paving robot based on the posture information of the paving robot collected by the camera includes: Based on the brick-grabbing posture information of the brick-laying robot collected by the camera, it is determined whether the brick-grabbing posture information meets the preset brick-grabbing posture conditions. When it is determined that the brick-grabbing posture information does not meet the preset brick-grabbing posture conditions, the first environmental information of the first brick-laying point is obtained. Based on the brick-grabbing posture information and the first environmental information, the first error information of the brick-laying robot is determined, and the first compensation information is generated based on the first error information. The first error information is the error information generated by the brick-laying robot during the brick-grabbing operation. Based on the first compensation information, the target compensation information of the paving robot is determined.

3. The intelligent control method for a robotic arm based on dynamic vision according to claim 1, characterized in that, When the posture information of the paving robot includes the chassis posture information of the paving robot, determining the target compensation information of the paving robot based on the posture information of the paving robot collected by the camera includes: Based on the chassis posture information of the paving robot collected by the camera, it is determined whether the chassis posture information meets the preset chassis posture conditions. When it is determined that the chassis posture information does not meet the preset chassis posture conditions, the chassis tilt coefficient of the paving robot is determined based on the chassis posture information of the paving robot. Based on the chassis tilt coefficient of the paving robot, the second error information of the paving robot is determined, and the second compensation information is generated based on the second error information. The second error information is the chassis tilt error information of the paving robot. Based on the second compensation information, the target compensation information of the paving robot is determined.

4. The intelligent control method for a robotic arm based on dynamic vision according to any one of claims 1-3, characterized in that, The process of determining the control parameters of the robotic arm based on the image information of the paving environment captured by the camera includes: Based on the real-time image information of the target object captured by the camera, the distance between the first brick position corresponding to the brick to be laid and the pre-determined reference object is calculated, and it is determined whether the distance between the bricks meets the preset brick joint conditions. When it is determined that the brick joint distance does not meet the preset brick joint conditions, the dynamic tracking parameters of the robotic arm are determined based on the brick joint distance and the first brick position. The movement control parameters of the robotic arm are generated based on the dynamic tracking parameters; the dynamic tracking parameters are used to represent the dynamic relative positional relationship between the robotic arm and the first brick position. And, controlling the robotic arm to perform operations matching the movement control parameters includes: The robotic arm is moved based on the movement control parameters so that the distance between the first brick position corresponding to the brick to be laid and the pre-determined reference object meets the preset brick joint conditions.

5. The intelligent control method for a robotic arm based on dynamic vision according to claim 4, characterized in that, After the paving robot reaches the second paving point, and before determining the control parameters of the robotic arm based on the image information of the paving environment captured by the camera, the method further includes: The camera is controlled to acquire image information of the environment to be paved, and based on visual servoing technology and the image information of the environment to be paved, the paving position relationship between the robotic arm and the ground to be paved is determined, and it is determined whether the paving position relationship meets the preset position conditions; the paving position relationship is used to represent the relative position relationship between the robotic arm and the bricks included in the ground to be paved; When it is determined that the brick laying position relationship does not meet the preset position conditions, the operation of determining the control parameters of the robotic arm based on the image information of the brick laying environment collected by the camera is triggered. When it is determined that the brick-laying positional relationship meets the preset positional conditions, the brick-laying robot is controlled to perform the brick-laying operation.

6. A robotic arm intelligent control device based on dynamic vision, characterized in that, The device is applied to a brick-laying robot, the robotic arm is mounted on the brick-laying robot, and a camera is installed on the brick-laying robot. The device includes: The determination module is used to determine the target compensation information of the paving robot based on the posture information of the paving robot collected by the camera during the process of the paving robot moving to the first paving point; the posture information of the paving robot includes the paving robot's brick-grabbing posture information and / or the paving robot's chassis posture information, and the first paving point is the paving point corresponding to the target movement trajectory generated by the trajectory simulation model. The correction module is used to perform a correction operation on the first paving point according to the target compensation information to obtain the second paving point; The control module is used to control the brick-laying robot to move to the second brick-laying point; The determining module is further configured to determine the movement control parameters of the robotic arm based on the image information of the environment to be paved collected by the camera when the paving robot reaches the second paving point. The control module is also used to control the robotic arm to perform operations that match the movement control parameters; the image information of the environment to be paved includes real-time image information of the target object; The data acquisition module is used to collect target environmental information corresponding to the location to be paved before the paving robot moves to the first paving point; The input module is used to input the target environment information and the structural information of the paving robot into a preset trajectory simulation model to obtain the target movement trajectory of the paving robot, the target movement trajectory including the target brick picking point and the first paving point; The control module is also used to control the paving robot to move to the target brick-picking point based on the target movement trajectory, and to control the paving robot to perform a brick-grabbing operation when the paving robot reaches the target brick-picking point; The judgment module is used to determine whether the paving robot has completed the brick-grabbing operation; when it is determined that the paving robot has not completed the brick-grabbing operation, the control module is re-triggered to execute the operation of controlling the paving robot to perform the brick-grabbing operation and the operation of determining whether the paving robot has completed the brick-grabbing operation. The control module is also used to control the paving robot to move to the first paving point when the judgment module determines that the paving robot has completed the brick-grabbing operation; The robotic arm is equipped with a suction cup, which is used to grip bricks. The specific methods by which the judgment module determines whether the brick-laying robot has completed the brick-grabbing operation include: Obtain the vacuum value of the suction cup and determine whether the vacuum value is greater than a preset vacuum threshold. When it is determined that the vacuum value is greater than the preset vacuum threshold, it is determined that the brick-laying robot has completed the brick-grabbing operation; When it is determined that the vacuum value is not greater than the preset vacuum threshold, it is determined that the brick-laying robot has not completed the brick-grabbing operation.

7. A robotic arm intelligent control device based on dynamic vision, characterized in that, The device is used in a brick-laying robot, and the device includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the intelligent control method for a robotic arm based on dynamic vision as described in any one of claims 1-5.

8. A brick-laying robot, characterized in that, The robotic arm is mounted on the brick-laying robot, and the brick-laying robot is equipped with a camera; The paving robot is used to execute the intelligent control method for a robotic arm based on dynamic vision as described in any one of claims 1-5.

Images