Method and apparatus for controlling manipulator poses, electronic device and storage medium

Abstract

The present invention provides a method and a device for controlling manipulator poses, an electronic device, and a storage medium. The method for controlling manipulator poses is applicable to an electronic device, wherein a communication link exists between the electronic device and a manipulator, a camera device is attached to the manipulator, a communication link exists between the camera device and the electronic device, and the camera device serves to photograph a target object located in a target area, wherein the method comprises: controlling the camera device to photograph a target object and obtaining the target point cloud data of the target object;Determining the first difference between the target point cloud data and the pre-stored target point cloud template data, wherein the pre-stored target point cloud template data is point cloud data obtained by photographing the target object using the camera device when the manipulator is in the target pose; and controlling the manipulator to move it into the target pose according to the first difference. The present invention can improve the accuracy of manipulator poses.

Description

Field of the invention

[0001] The present invention relates to the technical field of intelligent control technology and, in particular, to a method and apparatus for controlling manipulator poses, an electronic device and a storage medium. State of the art

[0002] With the continuous advancement of the manufacturing industry, more and more manufacturers are opting to combine automated production lines and manipulators to improve production efficiency. Currently, automated production lines typically use manipulators instead of manual labor to perform monotonous, complex, and high-risk operations. For a manipulator, personnel must manually configure the points in advance before it can autonomously move to the specified pose. This type of manipulator control is less efficient and has low accuracy in controlling manipulators to move them to the specified pose. Object of the invention

[0003] It is an object of the present invention to provide a method and apparatus for controlling manipulator poses, an electronic device and a storage medium to solve the technical problem of low accuracy in controlling manipulator poses. Technical solution

[0004] The present invention provides a method for controlling manipulator poses usable for an electronic device, wherein a communication connection exists between the electronic device and a manipulator, a camera device is attached to the manipulator, a communication connection exists between the camera device and the electronic device, and the camera device is for photographing a target object located in a target area, the method comprising: controlling the camera device to photograph a target object and obtaining the target point cloud data of the target object;Determining the first difference between the target point cloud data and the pre-stored template data for the target point cloud, wherein the pre-stored template data for the target point cloud is point cloud data obtained by photographing the target object with the camera device when the manipulator is in the target pose; and controlling the manipulator to move it to the target pose according to the first difference;

[0005] In some embodiments, the method for obtaining the pre-stored template data for the target point cloud comprises: receiving a first point input by the user; controlling the manipulator to move it to the target pose according to the first point; and controlling the camera device to photograph the target object located in the target area to obtain the pre-stored template data for the target point cloud.

[0006] In some embodiments, determining the first difference between the target point cloud data and the pre-stored target point cloud template data comprises: determining the first coordinate of each first point in the target point cloud data and the second coordinate of each second point in the target point cloud template data in the camera coordinate system; associating all second points with the corresponding first points according to the first coordinates to determine the second coordinates of the second points closest to the corresponding first coordinates, and determining a plurality of point pairs according to the first coordinates of all first points and the second coordinates of the corresponding second points; and calculating the difference between the first coordinate of a first point and the second coordinate of the corresponding second point in each point pair to obtain the first difference.

[0007] In some embodiments, controlling the manipulator to move to the target pose according to the first difference includes: transferring the first difference from the camera coordinate system to the world coordinate system to obtain the second difference; determining the second point according to the first point and the second difference; and controlling the manipulator according to the second point to move the manipulator to the target pose.

[0008] In some embodiments, the method further comprises: If the first difference is greater than the predetermined threshold, the adjustment of manipulator poses continues.

[0009] In some embodiments, continuing the adjustment of manipulator poses comprises: continuing to control the camera device to photograph the target object to obtain the target point cloud data; and determining the first difference between the target point cloud data and the pre-stored template data for the target point cloud and continuing the adjustment of manipulator poses according to the first difference until the first difference is less than or equal to the predetermined threshold.

[0010] In some embodiments, when a communication connection exists between the electronic device and a plurality of manipulators, the method further comprises: sending the first point to a plurality of manipulators and controlling the camera device attached to each manipulator to photograph the target object to obtain the target point cloud data for each manipulator; determining the first difference between the target point cloud data and the pre-stored template data for the target point cloud; and controlling each manipulator to move it to the target pose according to the corresponding first difference.

[0011] An embodiment of the present invention further provides a manipulator pose control device comprising: a control module for controlling the camera device to photograph the target object to obtain the target point cloud data; a determination module for determining the first difference between the target point cloud data and the pre-stored template data for the target point cloud, wherein the pre-stored template data for the target point cloud is point cloud data obtained by photographing the target object with the camera device at the time the manipulator is in the target pose; the control module further for controlling the manipulator to move it into the target pose according to the first difference.

[0012] An embodiment of the present invention further provides an electronic device comprising: a memory storing at least one instruction; and a processor executing the instruction stored in the memory to implement the method for controlling manipulator poses.

[0013] An embodiment of the present invention further provides a computer-readable storage medium storing at least one instruction, wherein the at least one instruction is executed by the processor of the electronic device to implement the method for controlling manipulator poses.

[0014] From the above-mentioned technical solutions, in the embodiments of the present invention, the camera device located on the manipulator for obtaining the target point cloud data can photograph a target object when the manipulator is in an arbitrary pose, and the difference between the current manipulator pose and the target pose is determined according to the target point cloud data and the pre-stored template data for the target point cloud, so as to move the manipulator to the target pose according to the first difference. In this way, the manipulator poses can be adjusted according to the principles of computer vision, and thus the accuracy of manipulator poses can be improved without the need for manual adjustment, thus improving the adjustment efficiency of manipulator poses. Brief description of the drawings Fig. 1 shows an illustration of an application scenario of the method for controlling manipulator poses according to an embodiment of the present invention; Fig. 2 shows a flowchart of the method for controlling manipulator poses according to an embodiment of the present invention; Fig. 3 shows a flowchart of the method for obtaining the template data for the target point cloud according to an embodiment of the present invention; Fig. 4 shows a flowchart of the method for determining the first difference according to an embodiment of the present invention; Fig. 5 shows a flowchart of the method for controlling the manipulator to move it to the target pose according to an embodiment of the present invention; Fig. 6 shows a flowchart of the method for continuing control of the manipulator to move it to the target pose, according to an embodiment of the present invention; Fig. 7 shows a flowchart of the method for controlling a plurality of manipulators to move them to the target pose according to an embodiment of the present invention; Fig. 8 shows a diagram of the functional modules of the manipulator pose control device according to an embodiment of the present invention; Fig. 9 shows a schematic diagram of the structure of the electronic device according to an embodiment of the present invention. Detailed description of the implementation examples

[0015] To better understand the objects, features, and advantages of the present invention, specific embodiments will be described in detail below with reference to the accompanying drawings. It should be noted that, unless there is a conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other. In the following description, numerous specific details are set forth to provide a thorough understanding of the present invention. The described embodiments represent only some of the embodiments of the present invention, but not all embodiments.

[0016] Furthermore, the terms "first" and "second" are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or the set of specified technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of these features. In the description of the present invention, "multiple" means two or more, unless expressly and specifically limited otherwise.

[0017] Unless otherwise stated or defined, all technical and scientific terms used herein have common-sense meanings to one of ordinary skill in the art to which this invention pertains. The terms used in the description of the present invention are for the purpose of describing specific embodiments only and are not intended to be limiting of the present invention. As used herein, the term "and / or" includes all combinations of one or more of the associated listed elements.

[0018] An embodiment of the present invention provides a method for controlling manipulator poses that can be used for one or more electronic devices. An electronic device is a device that can automatically perform numerical calculations and / or information processing according to predetermined or stored instructions. The hardware of such a device includes, but is not limited to, a microprocessor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), digital signal processor (DSP), embedded devices, etc.

[0019] An electronic device can be any electronic product that can interact with users, such as a PC, tablet, smartphone, personal digital assistant (PDA), game console, Internet Protocol Television (IPTV), and smart wearable device.

[0020] The electronic device may also include a network device. The network device may include a single network server, a server group consisting of multiple network servers, or a cloud computing-based cloud consisting of a large number of hosts or network servers.

[0021] The network on which an electronic device resides includes, but is not limited to, the Internet, a wide area network (WAN), a metropolitan area network (MAN), a local area network (LAN), a virtual private network (VPN), etc.

[0022] It will be Fig. 1. The inventive method for controlling manipulator poses can be used for an electronic device 100. The electronic device 100 is communicatively connected to a manipulator 200 and serves to receive the point information input by the user in order to control the manipulator 200 such that it is moved into a pose corresponding to the point information according to the point information, so that the manipulator can carry out a predetermined operation (e.g., gripping workpieces located in a predetermined area). A camera device 300 is attached to the manipulator 200 and serves toa product or a workpiece on a production line) and obtain the point cloud data of the target object 400 and adjust the pose of the manipulator 200 according to the point cloud data and the pre-stored template data for the target point cloud to achieve a more accurate manipulator pose.

[0023] In one embodiment of the present invention, the electronic device 100 may be an external electronic device communicatively connected to the manipulator 200, or it may be a central control device of the manipulator 200, but the present invention is not limited in this regard.

[0024] It will be Fig. 2, which shows a flowchart of the method for controlling manipulator poses according to an embodiment of the present invention. Depending on the need, the order of the steps in this flowchart can be changed, and some steps can be omitted. The method for controlling manipulator poses according to the embodiment of the present invention includes the following steps: S20: Control the camera setup to photograph a target object and obtain the target point cloud data of the target object.

[0025] In one embodiment of the present invention, when the manipulator is in any pose, the electronic device sends a photography instruction to the camera device to control the camera device to photograph a target object located in the target area and obtain the target point cloud data corresponding to the target object.

[0026] In one embodiment of the present invention, the target point cloud data comprises a plurality of first points, each first point corresponding to a respective first coordinate. A respective first coordinate may be a three-dimensional coordinate representing the position of the corresponding first point in the camera coordinate system of the camera device. For example, the coordinates of a particular first point may be represented by (x1, y1, z1).

[0027] S21: Determining the first difference between the target point cloud data and the pre-stored template data for the target point cloud, wherein the pre-stored template data for the target point cloud is point cloud data obtained by photographing the target object using the camera device when the manipulator is in the target pose.

[0028] In one embodiment of the present invention, the target point cloud data can represent the positional relationship between the camera device and the target object, since the target point cloud data is a collection of initial points in the camera coordinate system. To adjust the manipulator poses, a first difference between the target point cloud data and the pre-stored template data for the target point cloud can first be determined in order to adjust the manipulator pose according to the first difference.

[0029] In one embodiment of the present invention, the pre-stored template data for the target point cloud is the point cloud data obtained when the manipulator is in the target pose and the camera device photographs the target object. This point cloud data can represent the positional relationship between the camera device and the target object when the manipulator is in the target pose. The smaller the first difference between the target point cloud data and the pre-stored template data for the target point cloud, the closer the manipulator pose is to the target pose.

[0030] Specifically, the procedure for obtaining the pre-stored template data for the target point cloud refers to the explanations given in Fig. 3; the procedure for determining the first difference is described in the relevant notes to Fig. 4 described.

[0031] S22: Control the manipulator to move it to the target pose according to the first difference.

[0032] In one embodiment of the present invention, the first difference serves to represent the difference between the manipulator pose and the target pose when the target point cloud is obtained. Consequently, the manipulator pose can be corrected and calculated according to the first difference, so that the electronic device controls the manipulator to move it to the target pose to ensure that the manipulator can complete the specified operation in the target pose. For example, a specified operation can be grasping a target object located in the target area. The specific method for controlling the manipulator to move it to the target pose is described in the corresponding explanations of Fig. 5 described.

[0033] From the above-mentioned technical solutions, in the embodiments of the present invention, the camera device located on the manipulator for obtaining the target point cloud data can photograph a target object when the manipulator is in an arbitrary pose, and the difference between the current manipulator pose and the target pose is determined according to the target point cloud data and the pre-stored template data for the target point cloud, so as to move the manipulator to the target pose according to the first difference. In this way, the manipulator poses can be adjusted according to the principles of computer vision, and thus the accuracy of manipulator poses can be improved without the need for manual adjustment, thus improving the adjustment efficiency of the manipulator poses.

[0034] It will be Fig. 3, which shows a flowchart of the method for obtaining the pre-stored template data for the target point cloud. Depending on the need, the order of the steps in this flowchart can be changed, and some steps can be omitted. The method for obtaining the pre-stored template data for the target point cloud according to the embodiment of the present invention includes the following steps: S30: Receiving a first point entered by the user.

[0035] In one embodiment of the present invention, the first point may be a multidimensional vector input by the user into the electronic device, which serves to represent a manipulator pose desired by the user in three-dimensional space. For example, the multidimensional vector corresponding to the first point may be in the form of (x, y, z, ϕ x , ϕ y , ϕ z), where x, y, z represent the coordinates in three-dimensional space for the manipulator pose corresponding to the first point, ϕ x represents the angle between the line from the manipulator end to the manipulator base center and the x-axis, ϕ y represents the angle between the line from the manipulator end to the manipulator base center and the y-axis and ϕ z represents the angle between the line from the manipulator end to the manipulator base center and the z-axis.

[0036] S31: Control the manipulator to move it to the target pose according to the first point.

[0037] In one embodiment of the present invention, the electronic device controls the manipulator according to the first point to move it into a target pose. The target pose represents the manipulator pose desired by the user according to the first point. The manipulator pose includes various categories, with the manipulator having different functions in different pose categories. For example, if the pose corresponding to the first point is a grasping pose, the manipulator is moved into the target pose to grasp the target object located in the target area.

[0038] S32: Control the camera device to photograph the target object located in the target area to obtain the pre-stored template data for the target point cloud.

[0039] In one embodiment of the present invention, after the manipulator has been moved into the target pose, the camera device can be controlled to photograph the target object located in the target area to obtain template data for the target point cloud. The template data for the target point cloud comprises a plurality of second points, each second point corresponding to a corresponding second coordinate. Each second coordinate serves to represent the position of the second point in the camera coordinate system corresponding to the camera device. For example, the second coordinate can be represented in the form of (x2, y2, z2) coordinates.

[0040] In one embodiment of the present invention, in the subsequent manipulator pose control process, the first difference between the target point cloud data and the target point cloud template data can be calculated according to the first coordinate and the second coordinate. This allows the electronic device to control the manipulator to move to the target pose, thereby improving the accuracy of manipulator pose control.

[0041] It will be Fig. 4, which shows a flowchart of the method for determining the first difference according to an embodiment of the present invention. Depending on the need, the order of the steps in this flowchart can be changed, and some steps can be omitted. The method for determining the first difference according to the embodiment of the present invention includes the following steps: S40: Determine the first coordinate of each first point in the target point cloud data and the second coordinate of each second point in the template data for the target point cloud in the camera coordinate system.

[0042] To improve the accuracy of manipulator pose adjustment, in one embodiment of the present invention, the first coordinate of each first point in the target point cloud data and the second coordinate of each second point in the template data for the target point cloud can first be determined. Subsequently, the respective first differences can be determined according to all first and second coordinates.

[0043] S41: Assigning all the second points to the corresponding first points according to the first coordinates to determine the second coordinates of the second points closest to the corresponding first coordinates, and determining a plurality of point pairs according to the first coordinates of all the first points and the second coordinates of the corresponding second points.

[0044] In order to comprehensively compare the global features of the target point cloud data and the template data for the target point cloud, in one embodiment of the present invention, multiple point pairs are obtained by assigning the first and second coordinates to the first points and second points, respectively. Each point pair comprises a first point and a second point, wherein the distance between the point pair and the first coordinate of the corresponding first point or the second coordinate of the corresponding second point is minimal.

[0045] For example, if the first coordinate of a first point is (1, 2, 3) and the second coordinate of a second point is (1, 1, 2) and the distance between the first coordinate and the second coordinate is minimal, this first point and this second point are associated with each other and together form a point pair.

[0046] S42: Calculate the difference between the first coordinate of a first point and the second coordinate of the corresponding second point in each pair of points to obtain the first difference.

[0047] In one embodiment of the present invention, the first and second coordinates comprise multiple dimensions. Determining a first coordinate difference by determining the difference between the first coordinate of a first point and the second coordinate of the corresponding second point in each point pair is performed as follows: calculating a difference value between the first coordinate and the second coordinate in the same dimension to obtain the difference value of each point pair in each dimension; calculating the average of the difference values ​​of all point pairs in each dimension to obtain the mean difference value in each dimension; setting this mean difference value as the first coordinate difference for all dimensions.

[0048] For example, if a first point is associated with a second point and there are two pairs of points and the first coordinate and the second coordinate of one pair of points are (1, 2, 3) and (1, 1, 2), respectively, and the first coordinate and the second coordinate of the other pair of points are (4, 5, 6) and (4, 5, 7), respectively, the mean difference value in the x-coordinate dimension is 0, the mean difference value in the y-coordinate dimension is -0.5, and the mean difference value in the z-coordinate dimension is 0. Therefore, the first difference is (0, -0.5, 0).

[0049] It will be Fig. 5, which shows a flowchart of the method for controlling the manipulator to move it to the target pose according to an embodiment of the present invention. Depending on the need, the order of the steps in this flowchart can be changed, and some steps can be omitted. The method for controlling the manipulator to move it to the target pose according to the embodiment of the present invention includes the following steps: S50: Transfer the first difference from the camera coordinate system to the world coordinate system to obtain the second difference.

[0050] Since, in one embodiment of the present invention, the first difference is determined according to the first coordinate and the second coordinate in the camera coordinate system, the coordinate difference represented by the first difference is the difference between the target point cloud and the point cloud template in the camera coordinate system. To move the manipulator to the target pose according to the first difference, the first difference can first be transferred to the world coordinate system to obtain the second difference. The world coordinate system can be a three-dimensional coordinate system constructed around the manipulator base center.

[0051] S51: Determine the second point according to the first point and the second difference.

[0052] In one embodiment of the present invention, the first three dimensions in the vector corresponding to the first point and the coordinate corresponding to the second difference may be added to obtain the second point.

[0053] For example, if the first point is (1, 2, 3, 45°, 20°, 60°) and the second difference is (0, -0.5, 0), the second point is (1, 1.5, 3, 45°, 20°, 60°).

[0054] S52: Control the manipulator according to the second point to move the manipulator to the target pose.

[0055] In one embodiment of the present invention, the second point represents the corresponding position in the world coordinate system of the pose obtained after correcting the current pose.

[0056] In one embodiment of the present invention, after determining the first difference, the electronic device further determines whether the manipulator is in the target pose. If the first difference is less than or equal to the predetermined threshold, it is determined that the manipulator is in the target pose. If the first difference is greater than the predetermined threshold, it is determined that the difference between the current manipulator pose and the target pose is large, and the adjustment of the manipulator is continued. Details on the continuation of the adjustment of manipulator poses are described in the corresponding description in Fig. 6 can be found.

[0057] It will be Fig. 6, which shows a flowchart of the method for continuing the adjustment of manipulator poses according to an embodiment of the present invention. Depending on the need, the order of the steps in this flowchart can be changed and some steps can be omitted. The method for continuing the adjustment of manipulator poses according to the embodiment of the present invention includes the following steps: S60: Continue controlling the camera device to photograph the target object to obtain the target point cloud data.

[0058] In one embodiment of the present invention, to determine whether the manipulator is in the target pose, after setting the manipulator pose according to the second point, the electronic device further controls the camera device to photograph the target object to obtain the target point cloud data.

[0059] S61: Determine the first difference between the target point cloud data and the pre-stored template data for the target point cloud.

[0060] To determine whether a large difference still exists between the target point cloud data and the pre-stored target point cloud template data, the difference between the target point cloud and the point cloud template is determined again. The procedure for redetermining the first difference is identical to steps S40 to S42 and will therefore not be explained again here.

[0061] S62: Compare the first difference with the predefined threshold. If the first difference is greater than the predefined threshold, step S63 is executed; if the first difference is less than or equal to the predefined threshold, step S64 is executed.

[0062] In one embodiment of the present invention, if the first difference is greater than the predetermined threshold, it means that there is still a large difference between the target point cloud data and the template data for the target point cloud. Consequently, the difference between the current manipulator pose and the target pose is still large. Further adjustment of the manipulator pose is necessary.

[0063] S63: Continue setting manipulator poses according to the first difference and return to step S60.

[0064] In one embodiment of the present invention, the method for continuing the adjustment of manipulator poses according to the first difference is the same as the method described in steps S50 to S52, and therefore will not be described again here.

[0065] S64: Determine the current manipulator pose as the target pose.

[0066] In one embodiment of the present invention, if the first difference is less than or equal to the predetermined threshold, this means that the difference between the target point cloud data and the template data for the target point cloud is small, so that the agreement between the current manipulator pose and the target pose is high and it can be determined that the current manipulator pose is the target pose.

[0067] In one embodiment of the present invention, when a communication link exists between the electronic device and multiple manipulators, the electronic device further copies the first point to the multiple manipulators, thereby improving the efficiency in controlling the setting of multiple manipulator poses. Fig. 7, which shows a flowchart of the method for controlling multiple manipulators to move them to the target pose according to an embodiment of the present invention. Depending on the need, the order of the steps in this flowchart can be changed, and some steps can be omitted. The method for controlling multiple manipulators to move them to the target pose according to the embodiment of the present invention includes the following steps: S70: Send the first point to multiple manipulators and control the camera device attached to each manipulator to photograph the target object to obtain the target point cloud data for each manipulator.

[0068] In one embodiment of the present invention, when a communication link is established between the electronic device and multiple manipulators, it further serves to control the multiple manipulators to move them to the target pose. To improve the efficiency of controlling the setting of multiple manipulator poses, the electronic device can send the first point to multiple manipulators to directly copy the first point to each manipulator, eliminating the need for personnel to set the point of each individual manipulator one by one, thereby improving the efficiency of controlling the setting of multiple manipulator poses.

[0069] In one embodiment of the present invention, after all manipulator poses have been adjusted according to the first point, it is further necessary to determine whether all manipulator poses are correctly adjusted, thereby improving the accuracy of all manipulator poses. In this way, the camera device attached to each manipulator can be controlled to photograph the target object to obtain the target point cloud data for each manipulator.

[0070] S71: Determine the first differences between the target point cloud data and the pre-stored template data for the target point cloud.

[0071] To determine whether all manipulator poses are correctly adjusted, in one embodiment of the present invention, the first difference between the target point cloud data for each manipulator and the pre-stored template data for the target point cloud can be determined separately. Each manipulator pose can then be adjusted separately according to the corresponding first difference. The smaller this first difference, the smaller the difference between the current manipulator pose and the target pose after the manipulator poses have been adjusted according to the first point. The larger this first difference, the greater the difference between the current manipulator poses and the target pose.

[0072] S72: Control each manipulator to move it to the target pose according to the corresponding first difference.

[0073] In one embodiment of the present invention, the current manipulator poses can be corrected according to the corresponding first differences to reduce the difference between the current manipulator poses and the target pose, thereby improving the accuracy of controlling the setting of multiple manipulator poses. The method for controlling all manipulators to move them to the target pose according to the corresponding first differences is the same as that described in the detailed description of steps S50 to S52, and therefore will not be described again here.

[0074] In this way, by copying the first point to all manipulators in the production environment, control of multiple manipulator poses can be achieved, thus avoiding the inefficiency caused by manually adjusting manipulator poses. The first differences for the corresponding manipulators are determined according to the target point cloud and the pre-stored point cloud template. Then, all manipulator poses are adjusted separately according to the corresponding first differences, thereby improving the accuracy of all manipulator poses.

[0075] It will be Fig. 8, which shows a diagram of the functional modules of a manipulator pose control device according to an embodiment of the present invention. The manipulator pose control device 81 comprises a control module 810 and a determination module 811. In the present invention, a module / unit is understood to mean a series of computer-readable instruction sections executed by a processor 13 (see Fig. 9) can be executed and can perform a specific function and stored in a memory 12 (see Fig. 9). In the present embodiment, the functions of each module / unit are described in detail below.

[0076] The control module 810 is used to control the camera device for photographing the target object in order to obtain the target point cloud data.

[0077] The determination module 811 is for determining the first difference between the target point cloud data and the pre-stored template data for the target point cloud, wherein the pre-stored template data for the target point cloud is point cloud data obtained by photographing the target object using the camera device at the time when the manipulator is in the target pose.

[0078] The control module 810 further serves to control the manipulator to move it into the target pose according to the first difference.

[0079] In one embodiment of the present invention, the control module 810 is specifically provided for: receiving a first point input by the user; controlling the manipulator to move it to the target pose according to the first point; and controlling the camera device to photograph the target object located in the target area to obtain the pre-stored template data for the target point cloud.

[0080] In one embodiment of the present invention, the determination module 811 is specifically configured to: determine the first coordinate of each first point in the target point cloud data and the second coordinate of each second point in the template data for the target point cloud in the camera coordinate system; associate all second points with the corresponding first points according to the first coordinates to determine the second coordinates of the second points closest to the corresponding first coordinates, and determine a plurality of point pairs according to the first coordinates of all first points and the second coordinates of the corresponding second points; and calculate the difference between the first coordinate of a first point and the second coordinate of the corresponding second point in each point pair to obtain the first difference.

[0081] In one embodiment of the present invention, the control module 810 is further configured to: transfer the first difference from the camera coordinate system to the world coordinate system to obtain the second difference; determine the second point according to the first point and the second difference; and control the manipulator according to the second point to move the manipulator to the target pose.

[0082] In one embodiment of the present invention, the control module 810 is further provided for the following: If the first difference is greater than the predetermined threshold, the adjustment of manipulator poses is continued.

[0083] In one embodiment of the present invention, the control module 810 is further configured to: continue controlling the camera device to photograph the target object to obtain the target point cloud data; and determine the first difference between the target point cloud data and the prestored template data for the target point cloud and continue adjusting manipulator poses according to the first difference until the first difference is less than or equal to the predetermined threshold.

[0084] In one embodiment of the present invention, the control module 810 is further configured to: send the first point to a plurality of manipulators and control the camera device attached to each manipulator to photograph the target object to obtain the target point cloud data for each manipulator; determine the first difference between the target point cloud data and the pre-stored template data for the target point cloud; and control each manipulator to move it to the target pose according to the first corresponding difference.

[0085] From the above-mentioned technical solutions, in the embodiments of the present invention, the camera device located on the manipulator for obtaining the target point cloud data can photograph a target object when the manipulator is in an arbitrary pose, and the difference between the current manipulator pose and the target pose is determined according to the target point cloud data and the pre-stored template data for the target point cloud, so as to move the manipulator to the target pose according to the first difference. In this way, the manipulator poses can be adjusted according to the principles of computer vision, and thus the accuracy of manipulator poses can be improved without the need for manual adjustment, thus improving the adjustment efficiency of manipulator poses.

[0086] It will be Fig. 9, which shows a schematic diagram of the structure of the electronic device according to an embodiment of the present invention. The electronic device 1 includes a memory 12 and a processor 13. The memory 12 is used to store computer-readable instructions. The processor 13 is used to execute the computer-readable instructions stored in the memory to implement the method for controlling manipulator poses described in one of the above embodiments.

[0087] In one embodiment of the present invention, the electronic device 1 further comprises a bus and a computer program stored in the memory 12 and executable on the processor 13, for example a program for controlling manipulator poses.

[0088] Fig. 9 shows only the electronic device 1 with the memory 12 and the processor 13. It will be clear to the person skilled in the art that the electronic device 1 is not limited to the Fig. 9 and may be equipped with, combined with, or differently arranged with fewer or more components than shown.

[0089] As in Fig. 2, the memory 12 of the electronic device 1 stores computer-readable instructions for implementing the method for controlling manipulator poses, wherein the processor 13 can execute a plurality of instructions to achieve the following: controlling the camera device to photograph a target object and obtain the target point cloud data of the target object; determining the first difference between the target point cloud data and the pre-stored template data for the target point cloud, wherein the pre-stored template data for the target point cloud is point cloud data obtained by photographing the target object using the camera device when the manipulator is in the target pose; and controlling the manipulator to move it into the target pose according to the first difference.

[0090] For the concrete implementation method of the above instructions by means of the processor 13, reference is made to the description of the associated steps in the corresponding embodiment in Fig. 2, which is why it will not be discussed here.

[0091] Those skilled in the art will appreciate that the schematic diagram shown above is only an example of the electronic device 1 and does not represent a limitation of the electronic device 1. The electronic device 1 may have a bus structure or a star structure and may further include more or less different hardware or software than shown in the figure, or may include different component arrangements. For example, the electronic device 1 may further include an input and output device and a network access device.

[0092] It should be noted that electronic device 1 is merely exemplary. Other existing or possible future electronic products suitable for the present invention also fall within the scope of the present invention and are incorporated herein by reference.

[0093] The memory 12 comprises at least one type of readable storage medium that is non-volatile or volatile. Readable storage media include flash memory, removable hard drives, multimedia cards, card memories (e.g., SD or DX memory), magnetic storage, hard disks, optical data storage, etc. In some embodiments, the memory 12 may be an internal storage unit of the electronic device 1, such as a removable hard drive of the electronic device 1. In other embodiments, the memory 12 may also be an external storage device attached to the electronic device 1, such as a removable hard drive, a smart media card (SMC), an SD card (secure digital memory card), or a flash card. The memory 12 is not only used to store application software and various data in the electronic device 1, such asCode for the program to control manipulator poses, but also to temporarily store output data or data to be output.

[0094] In some embodiments, the processor 13 may consist of integrated circuits. For example, it may consist either of a single packaged integrated circuit or of multiple integrated circuits with the same or different functions and may include one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 13 is the control unit of the electronic device 1, which is connected to various components of the electronic device 1 via various interfaces and lines and performs various functions of the electronic device 1 and processes data by executing programs or modules stored in the memory 12 (e.g., programs for controlling manipulator poses) and by retrieving data stored in the memory 12.

[0095] The processor 13 executes the operating system of the electronic device 1 and various installed application programs. The processor 13 executes the application programs to Fig. 2 shown steps of the method for controlling manipulator poses described in all embodiments.

[0096] As an example, a computer program may be divided into one or more modules / units stored in memory 12 and executed by processor 13 to complete the present invention. The one or more modules / units may be a series of computer-readable instruction sections capable of performing a specific function and used to describe the execution process of the computer program in electronic device 1. For example, a computer program may be divided into a control module 810 and a determination module 811.

[0097] The above-mentioned integrated unit, implemented in the form of a software functional module, can be stored in a computer-readable storage medium. This software functional module is stored in a storage medium and comprises a plurality of instructions for causing a computing device (e.g., personal computer, computer, or network device) or a processor to execute the method for controlling manipulator poses described in various embodiments of the present invention.

[0098] If the integrated modules / units of the electronic device 1 are implemented in the form of software functional units and are sold or used as standalone products, they can be stored in a computer-readable storage medium. Based on this understanding, the implementation of all or part of the processes of the method according to the embodiments of the present invention can also be achieved by instructing corresponding hardware devices using a computer program, wherein the computer program can be stored on a computer-readable storage medium. When the computer program is executed by the processor, the steps of the method described in the above embodiments can be implemented.

[0099] The computer program includes computer program code, which may be in the form of source code, object code, executable files, or various intermediate forms. The computer-readable medium may include any entity or device capable of transmitting the computer program code, a recording medium, a USB disk, a removable hard disk, a magnetic disk, an optical storage device, a computer memory, a read-only memory (ROM), a random access memory, and other memories, etc.

[0100] Furthermore, the computer-readable storage medium may primarily comprise a storage program area and a storage data area, wherein the storage program area may store an operating system and at least one application program required for its operation, and the storage data area may store data created by the use of blockchain nodes and the like.

[0101] The bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be classified as an address bus, a data bus, and a control bus. To simplify the illustration, Fig. 9, only one arrow is used to represent a bus. However, this does not mean that there is only one bus or one bus type. The bus is configured to enable a communication link between the memory 12 and at least one processor 13.

[0102] Embodiments of the present invention further provide a computer-readable storage medium (not shown). The computer-readable storage medium stores computer-readable instructions that are executed by the processor located in the electronic device to implement the method for controlling manipulator poses described in any of the above-mentioned embodiments.

[0103] In the various embodiments of the present invention, it should be understood that the disclosed systems, devices, and methods may be implemented in other ways. For example, the device embodiments described above are for illustrative purposes only. For example, the division of the modules is only a logical division of functions. In the actual implementation, other division methods may also exist.

[0104] The modules described as separate components may or may not be physically separated. The components depicted as modules may or may not be physical units. They may be located in one location or distributed across multiple network units. Depending on the actual needs, some or all of the modules may be used to solve the tasks of the embodiments.

[0105] Furthermore, the functional modules in the various embodiments of the present invention may be integrated into a processing unit or implemented as separate physical units. Two or more units may be integrated into a single unit. These integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0106] Furthermore, it is clear that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. Multiple units or devices mentioned in the description may also be implemented by one unit or device using software or hardware. The words "first," "second," etc., are used to denote names and do not imply a particular order.

[0107] Finally, it should be noted that the above embodiments are only illustrative of the technical solutions of the present invention and are not limiting. Numerous modifications to the disclosed embodiments may be made by one skilled in the art without departing from the spirit or scope of this invention. All equivalent modifications and equivalent changes that can be made in accordance with the technical concepts of the present invention are within the scope of the present invention.

Claims

[1] Method for controlling manipulator poses usable for an electronic device, wherein a communication connection exists between the electronic device and a manipulator, characterized by that a camera device is attached to the manipulator, a communication connection exists between the camera device and the electronic device and the camera device serves to photograph a target object located in a target area, the method comprising the following: Controlling the camera device to photograph a target object and obtain the target point cloud data of the target object; Determining the first difference between the target point cloud data and the pre-stored template data for the target point cloud, wherein the pre-stored template data for the target point cloud is point cloud data obtained by photographing the target object using the camera device when the manipulator is in the target pose; and Control the manipulator to move it to the target pose according to the first difference. [2] Method for controlling manipulator poses according to claim 1, characterized by that the method for obtaining the pre-stored template data for the target point cloud comprises: Receiving a first point entered by the user; Controlling the manipulator to move it to the target pose according to the first point; and Control the camera setup to photograph the target object located in the target area to obtain the pre-stored template data for the target point cloud. [3] Method for controlling manipulator poses according to claim 1, characterized by that determining the first difference between the target point cloud data and the pre-stored template data for the target point cloud comprises: Determining the first coordinate of each first point in the target point cloud data and the second coordinate of each second point in the template data for the target point cloud in the camera coordinate system; Assigning all second points to the corresponding first points according to the first coordinates to determine the second coordinates of the second points that are closest to the corresponding first coordinates, and determining a plurality of point pairs according to the first coordinates of all first points and the second coordinates of the corresponding second points; and Calculate the difference between the first coordinate of a first point and the second coordinate of the corresponding second point in each pair of points to obtain the first difference. [4] Method for controlling manipulator poses according to claim 2, characterized by that controlling the manipulator to move it to the target pose according to the first difference comprises: Transferring the first difference from the camera coordinate system to the world coordinate system to obtain the second difference; Determining the second point according to the first point and the second difference; and Control the manipulator according to the second point to move the manipulator to the target pose. [5] Method for controlling manipulator poses according to claim 1 or 4, characterized by that the method further comprises: If the first difference is greater than the predetermined threshold, the adjustment of manipulator poses is continued. [6] Method for controlling manipulator poses according to claim 5, characterized by that continuing to adjust manipulator poses involves: Continue controlling the camera device to photograph the target object to obtain the target point cloud data; and Determining the first difference between the target point cloud data and the pre-stored template data for the target point cloud and continuing to adjust manipulator poses according to the first difference until the first difference is less than or equal to the predetermined threshold. [7] Method for controlling manipulator poses according to claim 2, characterized by that, when a communication link exists between the electronic device and a plurality of manipulators, the method further comprises: Sending the first point to multiple manipulators and controlling the camera device attached to each manipulator to photograph the target object to obtain the target point cloud data for each manipulator; Determining the first difference between the target point cloud data and the pre-stored template data for the target point cloud; and Control each manipulator to move it to the target pose according to the corresponding first difference. [8] A device for controlling manipulator poses, characterized by that the device comprises a module implementing the method for controlling manipulator poses according to one of claims 1 to 7, and that the device comprises: a control module used to control the camera device for photographing the target object to obtain the target point cloud data; a determination module for determining the first difference between the target point cloud data and the pre-stored template data for the target point cloud, wherein the pre-stored template data for the target point cloud is point cloud data obtained by photographing the target object using the camera device at the time when the manipulator is in the target pose; wherein the control module is further for controlling the manipulator to move it into the target pose according to the first difference. [9] An electronic device, characterized by that the electronic device includes: a memory that stores at least one instruction; and a processor that executes the instruction stored in the memory to implement the method for controlling manipulator poses according to any one of claims 1 to 7. [10] A computer-readable storage medium, characterized by in that the computer-readable storage medium stores a computer-readable instruction, the computer-readable instruction being executed by the processor to implement the method for controlling manipulator poses according to any one of claims 1 to 7.

Images