A motion mechanism control method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a motion mechanism control method and device, electronic equipment and storage medium, and relate to the technical field of robots. The specific solutions are as follows: a current image pose of an object to be grabbed by a motion mechanism in an image collected by an image collection device is obtained; a rotation angle difference between the current image pose and a reference image pose of the object is determined; a translation deviation of the motion mechanism from a first physical position of the reference image pose in a physical coordinate system to a second physical position of the current image pose in the physical coordinate system is determined according to the current image pose, the reference image pose, a calibration matrix, a skew matrix and the rotation angle difference, the calibration matrix representing a transformation relationship between a pixel coordinate system and the physical coordinate system, and the skew matrix representing a transformation relationship between the physical coordinate system and an orthogonal coordinate system; and the motion mechanism is controlled to move according to the translation deviation. The application of the solutions provided in the embodiments of the present application can reduce the error when the motion mechanism is controlled to move.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, and in particular to a motion mechanism control method and device, electronic equipment and a storage medium. BACKGROUND

[0002] With the maturity of robot technology, more and more industrial production scenarios introduce motion mechanisms such as mechanical arms and mechanical hands. Based on a visual positioning result, these motion mechanisms move to a position where an object such as a workpiece is located, then grasp the object, and install or place the object to a specified position.

[0003] To ensure that the motion mechanism can accurately grasp the object, in related technologies, after obtaining an image pose of the object in an image collected by an image collection device, a pose of the object in a physical coordinate system of the motion mechanism is determined based on the image pose, as a running pose of the motion mechanism. Then, on the premise that the motion mechanism moves as a rigid body, the motion mechanism is controlled to move from a current pose to the running pose.

[0004] However, in the case where the xy axes of the motion mechanism are not orthogonal, the motion mechanism is difficult to move as a rigid body, so there is often a large error when the motion mechanism is controlled to move in the above manner, and it is difficult to accurately move to the position where the object is located, and further difficult to accurately grasp the object. SUMMARY

[0005] Embodiments of the present application aim to provide a motion mechanism control method and device, electronic equipment and a storage medium to reduce the error when the motion mechanism is controlled to move. The specific technical solutions are as follows:

[0006] According to an aspect of an embodiment of the present application, a motion mechanism control method is provided, and the method comprises:

[0007] obtaining a current image pose of an object to be grasped by a motion mechanism in an image collected by an image collection device;

[0008] determining a rotation angle difference between the current image pose and a reference image pose of the object;

[0009] determining a translation deviation of the motion mechanism from a first physical position to a second physical position according to the current image pose, the reference image pose, a calibration matrix, a skew matrix and the rotation angle difference, wherein the calibration matrix represents a transformation relationship between a pixel coordinate system of the image collection device and a physical coordinate system of the motion mechanism, the skew matrix represents a transformation relationship between the physical coordinate system and an orthogonal coordinate system, the first physical position is a position corresponding to the reference image pose in the physical coordinate system, and the second physical position is a position corresponding to the current image pose in the physical coordinate system.

[0010] According to the translation deviation, the motion mechanism is controlled to move from the first physical position to the second physical position.

[0011] In one embodiment of the present application, the translation deviation of the motion mechanism from the first physical position to the second physical position is determined according to the current image pose, the reference image pose, a calibration matrix, a skew matrix and the rotation angle difference, comprising:

[0012] According to the position information in the calibration matrix and the reference image pose, the first physical position is determined;

[0013] According to the position information in the calibration matrix, the skew matrix, the rotation angle difference and the current image pose, the second physical position is determined;

[0014] The difference between the first physical position and the second physical position is calculated as the translation deviation of the motion mechanism from the first physical position to the second physical position.

[0015] In one embodiment of the present application, the second physical position is determined according to the calibration matrix, the skew matrix, the rotation angle difference and the position information in the current image pose, comprising:

[0016] The second physical position is determined according to the following expression:

[0017]

[0018] wherein, is the inverse matrix of the skew matrix, and δ is the skew angle, is a rotation transformation matrix, Δθ is the rotation angle difference, and M HAE is the calibration matrix, and (x cur , y cur ) is the coordinate of the position information in the current image pose in the pixel coordinate system.

[0019] In one embodiment of the present application, the skew matrix is obtained in the following manner:

[0020] A calibration result of hand-eye calibration of the physical coordinate system and the pixel coordinate system is obtained;

[0021] According to the calibration result, a skew angle of the physical coordinate system relative to an orthogonal coordinate system is generated;

[0022] Based on the obtained skew angle, the skew matrix is constructed.

[0023] In one embodiment of the present application, if the motion mechanism is a two-axis separated mechanism, the calibration result includes a single-axis calibration matrix corresponding to each single axis, wherein the single-axis calibration matrix corresponding to each single axis represents the transformation relationship between the orthogonal coordinate system corresponding to the single axis and the pixel coordinate system, and the orthogonal coordinate system corresponding to each single axis is an orthogonal coordinate system constructed with the single axis as the coordinate axis; and the generation of the skew angle of the physical coordinate system relative to the orthogonal coordinate system according to the calibration result includes: obtaining coordinates of the position information in the reference image pose in the two single-axis corresponding orthogonal coordinate systems according to the two single-axis calibration matrices; generating the rotation angle between the two single-axis corresponding orthogonal coordinate systems based on the obtained coordinates; and determining the skew angle of the physical coordinate system relative to the orthogonal coordinate system according to the rotation angle.

[0024] Or

[0025] If the motion mechanism is a two-axis integrated mechanism, the calibration result is the calibration matrix; and the generation of the skew angle of the physical coordinate system relative to the orthogonal coordinate system according to the calibration result includes: determining an x-axis scaling coefficient using a first coefficient in the calibration matrix, wherein the first coefficient represents the scaling and rotation relationship of the x-axis direction between the pixel coordinate system and the physical coordinate system; determining a y-axis scaling coefficient using a second coefficient in the calibration matrix, wherein the second coefficient represents the scaling, rotation and skew relationship of the y-axis direction between the pixel coordinate system and the physical coordinate system; and generating the skew angle of the physical coordinate system relative to the orthogonal coordinate system according to the first coefficient, the second coefficient, the x-axis scaling coefficient and the y-axis scaling coefficient.

[0026] In one embodiment of the present application, the generation of the rotation angle between the two single-axis corresponding orthogonal coordinate systems based on the obtained coordinates includes: generating the rotation angle between the two single-axis corresponding orthogonal coordinate systems according to the following expression: wherein α is the rotation angle between the two single-axis corresponding orthogonal coordinate systems, and (x1, y1) and (x2, y2) are the coordinates of the position information in the reference image pose in the two single-axis corresponding orthogonal coordinate systems, respectively.

[0027] Or

[0028] The generation of the skew angle of the physical coordinate system relative to the orthogonal coordinate system according to the first coefficient, the second coefficient, the x-axis scaling coefficient and the y-axis scaling coefficient includes: generating the skew angle of the physical coordinate system relative to the orthogonal coordinate system according to the following expression: wherein δ is the skew angle, D2 is the second coefficient, D1 is the first coefficient, sx is the x-axis scaling coefficient, and sy is the y-axis scaling coefficient.

[0029] In one embodiment of the present application, if the motion mechanism is a two-axis separated mechanism, the calibration matrix is obtained in the following manner:

[0030] obtaining single-axis calibration matrices obtained by respectively performing hand-eye calibration on two single axes in the motion mechanism, wherein each single-axis calibration matrix corresponding to a single axis represents a transformation relationship between an orthogonal coordinate system corresponding to the single axis and the pixel coordinate system, and each single-axis orthogonal coordinate system is an orthogonal coordinate system constructed with the single axis as a coordinate axis; collecting calibration data under the condition that each single axis is moved and the other single axis is fixed when performing hand-eye calibration on each single axis; obtaining coordinates of position information in the reference image pose in the orthogonal coordinate systems corresponding to the two single axes respectively according to the two single-axis calibration matrices; generating a rotation angle between the orthogonal coordinate systems corresponding to the two single axes based on the obtained coordinates; obtaining an included angle between the two single axes based on the rotation angle; generating a base on the two single axes based on the included angle, and determining the calibration matrix using the generated base and the obtained single-axis calibration matrices;

[0031] or

[0032] if the motion mechanism is a two-axis integrated mechanism, the calibration matrix is obtained in the following manner:

[0033] obtaining image coordinates and physical coordinates of a reference object corresponding to the motion mechanism at different positions, wherein the different positions are not collinear; constructing a calibration matrix based on the obtained image coordinates and physical coordinates, wherein the image coordinates are coordinates of the reference object in an image collected by an image collection device.

[0034] According to another aspect of the embodiments of the present application, a motion mechanism control device is provided, and the device comprises:

[0035] an image pose obtaining module, configured to obtain a current image pose of an object to be grabbed by a motion mechanism in an image collected by an image collection device;

[0036] an angle difference determining module, configured to determine a rotation angle difference between the current image pose and a reference image pose of the object;

[0037] a translation deviation determination module, configured to determine a translation deviation of the motion mechanism from a first physical position to a second physical position according to the current image pose, the reference image pose, a calibration matrix, a skew matrix and the rotation angle difference, wherein the calibration matrix represents a transformation relationship between a pixel coordinate system of the image acquisition device and a physical coordinate system of the motion mechanism, the skew matrix represents a transformation relationship between the physical coordinate system and an orthogonal coordinate system, the first physical position is a position corresponding to the reference image pose in the physical coordinate system, and the second physical position is a position corresponding to the current image pose in the physical coordinate system;

[0038] a motion mechanism control module, configured to control the motion mechanism to move from the first physical position to the second physical position according to the translation deviation.

[0039] In an embodiment of the present application, the translation deviation determination module comprises: a first physical position determination unit, configured to determine the first physical position according to the calibration matrix and position information in the reference image pose; a second physical position determination unit, configured to determine the second physical position according to the calibration matrix, the skew matrix, the rotation angle difference and position information in the current image pose; and a difference calculation unit, configured to calculate a difference between the first physical position and the second physical position as the translation deviation of the motion mechanism from the first physical position to the second physical position.

[0040] In an embodiment of the present application, the second physical position determination unit is specifically configured to determine the second physical position according to the following expression: wherein, is an inverse matrix of the skew matrix, and δ is a skew angle, is a rotation transformation matrix, Δθ is the rotation angle difference, and M HAE is the calibration matrix, and (x cur , y cur ) is a coordinate of the position information in the current image pose in the pixel coordinate system.

[0041] In an embodiment of the present application, the skew matrix is obtained in the following manner: obtaining a calibration result of hand-eye calibration of the physical coordinate system and the pixel coordinate system; generating a skew angle of the physical coordinate system relative to the orthogonal coordinate system according to the calibration result; and constructing the skew matrix based on the obtained skew angle.

[0042] In one embodiment of the present application, if the motion mechanism is a two-axis separated mechanism, the calibration result comprises a single-axis calibration matrix corresponding to each single axis, wherein the single-axis calibration matrix corresponding to each single axis represents the transformation relationship between the orthogonal coordinate system corresponding to the single axis and the pixel coordinate system, and the orthogonal coordinate system corresponding to each single axis is an orthogonal coordinate system constructed with the single axis as the coordinate axis; the skew angle of the physical coordinate system relative to the orthogonal coordinate system is generated according to the calibration result, comprising: obtaining the coordinates of the position information in the reference image pose in the two single-axis corresponding orthogonal coordinate systems according to the two single-axis calibration matrices; generating the rotation angle between the two single-axis corresponding orthogonal coordinate systems based on the obtained coordinates; determining the skew angle of the physical coordinate system relative to the orthogonal coordinate system according to the rotation angle; or, if the motion mechanism is a two-axis integrated mechanism, the calibration result is the calibration matrix; the skew angle of the physical coordinate system relative to the orthogonal coordinate system is generated according to the calibration result, comprising: determining the x-axis scaling coefficient using the first coefficient in the calibration matrix, wherein the first coefficient represents the scaling and rotation relationship of the x-axis direction between the pixel coordinate system and the physical coordinate system; determining the y-axis scaling coefficient using the second coefficient in the calibration matrix, wherein the second coefficient represents the scaling, rotation and skew relationship of the y-axis direction between the pixel coordinate system and the physical coordinate system; and generating the skew angle of the physical coordinate system relative to the orthogonal coordinate system according to the first coefficient, the second coefficient, the x-axis scaling coefficient and the y-axis scaling coefficient.

[0043] In one embodiment of the present application, the rotation angle between the two single-axis corresponding orthogonal coordinate systems is generated based on the obtained coordinates, comprising: generating the rotation angle between the two single-axis corresponding orthogonal coordinate systems according to the following expression: wherein a is the rotation angle between the two single-axis corresponding orthogonal coordinate systems, and (x1, y1) and (x2, y2) are the coordinates of the position information in the reference image pose in the two single-axis corresponding orthogonal coordinate systems, respectively; or, the skew angle of the physical coordinate system relative to the orthogonal coordinate system is generated according to the first coefficient, the second coefficient, the x-axis scaling coefficient and the y-axis scaling coefficient, comprising: generating the skew angle of the physical coordinate system relative to the orthogonal coordinate system according to the following expression: wherein δ is the skew angle, D2 is the second coefficient, D1 is the first coefficient, sx is the x-axis scaling coefficient, and sy is the y-axis scaling coefficient.

[0044] In one embodiment of the present application, if the motion mechanism is a two-axis separated mechanism, the calibration matrix is obtained in the following manner: single-axis calibration matrices obtained by respectively performing hand-eye calibration on two single axes of the motion mechanism are obtained, wherein each single-axis calibration matrix represents the transformation relationship between the orthogonal coordinate system corresponding to the single axis and the pixel coordinate system, and each single-axis calibration matrix is obtained by collecting calibration data under the condition that the single axis is moved and the other single axis is fixed when performing hand-eye calibration on each single axis; coordinates of the position information in the reference image pose in the orthogonal coordinate systems corresponding to the two single axes are obtained according to the two single-axis calibration matrices; a rotation angle between the orthogonal coordinate systems corresponding to the two single axes is generated based on the obtained coordinates; an included angle between the two single axes is obtained based on the rotation angle; a base on the two single axes is generated based on the included angle, and the calibration matrix is determined by using the generated base and the obtained single-axis calibration matrices; or, if the motion mechanism is a two-axis integrated mechanism, the calibration matrix is obtained in the following manner: image coordinates and physical coordinates of a reference object corresponding to the motion mechanism at different positions are obtained, wherein the different positions are not collinear; the calibration matrix is constructed based on the obtained image coordinates and physical coordinates, wherein the image coordinates are coordinates of the reference object in an image collected by an image collection device.

[0045] According to still another aspect of the embodiments of the present application, an electronic device is provided, comprising:

[0046] a memory for storing a computer program;

[0047] a processor for executing the program stored on the memory, thereby implementing the motion mechanism control method described above.

[0048] According to still another aspect of the embodiments of the present application, a computer readable storage medium is provided, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the motion mechanism control method described above.

[0049] According to still another aspect of the embodiments of the present application, a computer program product containing instructions is provided, which, when executed on a computer, causes the computer to perform the motion mechanism control method described above.

[0050] The embodiments of the present application have the following beneficial effects:

[0051] From the above, in the scheme provided by the embodiment of the application, the current image pose, the reference image pose, the calibration matrix, the skew matrix and the rotation angle difference are used to determine the translation deviation of the motion mechanism from the first physical position to the second physical position. Since the skew matrix represents the transformation relationship between the physical coordinate system and the orthogonal coordinate, in the case that the two coordinate axes of the physical coordinate system are not orthogonal, the translation deviation between the first physical position corresponding to the reference image pose and the second physical position corresponding to the current image pose in the physical coordinate system can be accurately determined. In this way, the motion mechanism can be more accurately controlled from the first physical position to the second physical position according to the more accurate translation deviation in the physical coordinate system, and the error in controlling the motion mechanism to move is reduced.

[0052] Of course, implementing any product or method of the application does not necessarily require all the advantages described above to be achieved at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only constitute some embodiments of the application, and other embodiments can also be obtained according to these drawings for those skilled in the art.

[0054] Figure 1a A schematic diagram of a non-orthogonal coordinate system and an orthogonal coordinate system provided by the embodiment of the application;

[0055] Figure 1b A schematic diagram of a two-axis separated mechanism provided by the embodiment of the application;

[0056] Figure 1c A schematic diagram of a two-axis integrated mechanism provided by the embodiment of the application;

[0057] Figure 2 A flowchart of a motion mechanism control method provided by the embodiment of the application;

[0058] Figure 3 A flowchart of a skew matrix obtaining method provided by the embodiment of the application;

[0059] Figure 4 Another schematic diagram of a non-orthogonal coordinate system and an orthogonal coordinate system provided by the embodiment of the application;

[0060] Figure 5 A flowchart of a calibration matrix obtaining method provided by the embodiment of the application;

[0061] Figure 6 A structural schematic diagram of a motion mechanism control device provided by the embodiment of the application;

[0062] Figure 7 A structural schematic diagram of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0063] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art based on the present application belong to the scope of protection of the present application.

[0064] The background of the present application will be described below.

[0065] Referring to Figure 1a A schematic diagram of a non-orthogonal coordinate system and an orthogonal coordinate system is provided. Figure 1a The xoy' coordinate system in the above formula is a non-orthogonal coordinate system, and the xoy coordinate system is an orthogonal coordinate system established with the x axis of the xoy' coordinate system, and the coordinate system origin o and the x axis of the xoy' coordinate system and the xoy coordinate system are coincident. It is assumed that the angle between the two axes of the xoy' coordinate system is r, and in the xoy' coordinate system, is the x axis unit vector, is the y' axis unit vector, and the lengths of the two unit vectors are 1,

[0066] Then, in the non-orthogonal xoy' coordinate system, The coordinates of the F point can be expressed as The coordinates of the G point can be expressed as

[0067] In the orthogonal xoy coordinate system, the coordinates of the F point can be expressed as The coordinates of the G point can be expressed as

[0068] Thus, it can be concluded that in the non-orthogonal xoy' coordinate system, the coordinate deviation from the F point to the G point is: In the orthogonal xoy coordinate system, the coordinate deviation from the F point to the G point is:

[0069] Obviously, and The coordinate deviation of the F point to the G point in the orthogonal xoy coordinate system is not the same as the coordinate deviation in the non-orthogonal xoy' coordinate system, that is, the coordinate deviation of the F point to the G point in the orthogonal xoy coordinate system is different from the coordinate deviation in the non-orthogonal xoy' coordinate system. That is, in the case where the xy axes of the motion mechanism are non-orthogonal, if the motion mechanism is controlled to move on the premise of rigid body motion, there is often a large error when the motion mechanism is controlled to move, and the error is larger when the bevel angle is larger. Therefore, it is difficult to accurately move to the position of the object based on rigid body motion transformation, and it is further difficult to accurately grasp the object.

[0070] The execution subject of the embodiment of the present application is described below.

[0071] The scheme provided by the embodiment of the present application can be applied to electronic devices such as desktop computers, notebook computers, tablet computers, and servers, and the above-mentioned electronic devices can control the movement of the mechanical arm, or the execution subject of the scheme provided by the embodiment of the present application can also be a mechanical arm. For ease of description, the execution subject of the motion mechanism control method provided by the embodiment of the present application is collectively referred to as a control device.

[0072] The concepts related to the embodiment of the present application are described below.

[0073] I. Bevel angle

[0074] The bevel angle is the angle between an axis in the non-orthogonal coordinate system and the corresponding axis in the orthogonal coordinate system. For example, the bevel angle can be the angle between the y axis and the y' axis. Figure 1a

[0075] II. Bevel matrix

[0076] The bevel matrix represents the transformation relationship between the physical coordinate system and the orthogonal coordinate system. The transformation relationship between the physical coordinate system and the orthogonal coordinate system can include translation transformation, rotation transformation, rigid body transformation, non-uniform scaling transformation, and bevel transformation, etc.

[0077] III. Calibration matrix

[0078] The calibration matrix represents the transformation relationship between the pixel coordinate system of the image acquisition device and the physical coordinate system of the motion mechanism.

[0079] IV. Motion mechanism as a two-axis separated mechanism and motion mechanism as a two-axis integrated mechanism

[0080] (1) Motion mechanism as a two-axis separated mechanism

[0081] The structure in the motion mechanism that moves along the x axis direction of the physical coordinate system is separated from the structure in the motion mechanism that moves along the y axis direction of the physical coordinate system, and the motion mechanism can carry the object to move in the x axis direction, and the end effector and the camera in the motion mechanism for grasping the object can move in the y axis direction. For example, refer to​Figure 1b The provided motion mechanism is a schematic diagram of a two-axis separated mechanism. The black block on the right side represents a first movable structure of the motion mechanism, which can be constrained by a first motion rod and move the object to be grabbed on the left side of the first movable structure along the positive and negative directions of the x-axis. The black block on the top side represents a second movable structure of the motion mechanism, which can be constrained by a second motion rod and move the end effector and the image acquisition device along the positive and negative directions of the y-axis.

[0082] (2) The motion mechanism is a two-axis integrated mechanism

[0083] The structure that moves along the x-axis direction of the physical coordinate system in the motion mechanism is integrated with the structure that moves along the y-axis direction of the physical coordinate system. The end effector for grabbing the object in the motion mechanism can move in the x-axis direction or in the y-axis direction. For example, refer to Figure 1c The provided motion mechanism is a schematic diagram of a two-axis integrated mechanism. The image acquisition device can be installed at a fixed position in the scene. The black block on the right side represents a third movable structure of the motion mechanism, which can be constrained by a third motion rod and move the fourth movable structure and the fourth motion rod on the left side of the third movable structure along the positive and negative directions of the x-axis. The black block on the left side represents a fourth movable structure of the motion mechanism, which can be constrained by a fourth motion rod and move the end effector along the positive and negative directions of the y-axis.

[0084] The motion mechanism control method provided by the embodiments of the present application will be described below in combination with specific embodiments.

[0085] In an embodiment of the present application, refer to Figure 2 The provided is a flowchart of a motion mechanism control method. The above method comprises the following steps S201-S204.

[0086] Step S201: Obtain the current image pose of the object to be grabbed by the motion mechanism in the image acquired by the image acquisition device.

[0087] The image acquisition device can be installed at a fixed position in the surrounding environment where the motion mechanism is located. The image acquisition device can also be installed on the mechanical arm. For example, for the case of a two-axis separated motion mechanism, the image acquisition device can be installed on the x-axis of the motion mechanism and moved by the y-axis of the motion mechanism to grab the object. The object to be grabbed by the motion mechanism can be a workpiece to be grabbed. The current image pose is the image pose of the object to be grabbed in the pixel coordinate system. The pose in the pixel coordinate system is obtained according to the image acquired by the image acquisition device. The motion mechanism needs to move from the physical position corresponding to the reference image pose to the physical position corresponding to the current image pose to grab the object.

[0088] The way of obtaining the current image pose in step S201 is described below.

[0089] In an implementation, after the image acquisition device acquires the image of the current pose of the object to be grabbed, the control device performs target recognition on the image of the object to be grabbed based on a preset template of the object to be grabbed, recognizes the position and angle of the target object in the image, and takes the obtained position and angle as the current image pose of the object to be grabbed.

[0090] Step S202: Determine the rotation angle difference between the current image pose and the reference image pose of the object.

[0091] The reference image pose is the image pose of the object in the pixel coordinate system when the object is in the reference pose.

[0092] The way of obtaining the reference image pose of the object is described below.

[0093] Specifically, the motion mechanism can be controlled to grab the object in the reference pose, and then the image acquisition device installed thereon can be used to obtain the image of the object grabbed by the motion mechanism, and then the pose of the object on the motion mechanism in the image can be recognized to obtain the reference image pose.

[0094] The way of determining the rotation angle difference in step S202 is described below.

[0095] In an implementation, the control device can calculate the difference between the angle information in the current image pose and the angle information in the reference image pose as the rotation angle difference.

[0096] Step S203: Determine the translation deviation of the motion mechanism from the first physical position to the second physical position according to the current image pose, the reference image pose, the calibration matrix, the skew matrix, and the rotation angle difference.

[0097] The first physical position is the position corresponding to the reference image pose in the physical coordinate system, and the second physical position is the position corresponding to the current image pose in the physical coordinate system. The physical coordinate system is the coordinate system referred to by the motion mechanism during movement. In the case where the x-axis and y-axis of the motion mechanism are not orthogonal, the x-axis and y-axis of the physical coordinate system are also not orthogonal. In addition, the directions of the x-axis and y-axis of the physical coordinate system can be the same as the directions of the first motion rod, the second motion rod, the third motion rod, and the fourth motion rod of the motion mechanism. Figure 1b and Figure 1c In the case where the motion mechanism is a multi-axis robot arm, the multi-axis robot arm can also perform a composite motion to realize movement in the x-axis and y-axis directions.

[0098] The way of determining the translation deviation in step S203 is described below.

[0099] In an implementation, the control device determines the first physical position according to the calibration matrix and the position information in the reference image pose; determines the second physical position according to the calibration matrix, the skew matrix, the rotation angle difference and the position information in the current image pose; and calculates the difference between the first physical position and the second physical position as the translation deviation of the motion mechanism moving from the first physical position to the second physical position.

[0100] Specifically, the control device left multiplies the calibration matrix by the position coordinates in the reference image pose to obtain the first physical position coordinates, for example, the calibration matrix Then the obtained first physical position coordinates are

[0101] wherein (x base , y base ) are the coordinates of the position information in the reference image pose in the pixel coordinate system.

[0102] Then, the control device determines the second physical position according to the following expression:

[0103]

[0104] wherein is the inverse matrix of the skew matrix, and δ is the skew angle, is the rotation transformation matrix, Δθ is the rotation angle difference, and M HAE is the calibration matrix, and (x cur , y cur ) are the coordinates of the position information in the current image pose in the pixel coordinate system.

[0105] Thus, the difference between the obtained first physical position and the second physical position is:

[0106]

[0107] wherein (x offset , y offset ) is the translation deviation, x offset is the translation deviation in the x-axis direction in the physical coordinate system, and y offset is the translation deviation in the y-axis direction in the physical coordinate system.

[0108] It can be seen that the second physical position can be determined more accurately according to the above expression, and thus the accuracy of the determined translation deviation can be improved.

[0109] Therefore, according to the position information in the calibration matrix, the skew matrix, the rotation angle difference and the current image pose, the second position information can be determined more accurately, and the difference between the first physical position and the second physical position can be determined accurately, so that the translation deviation of moving from the first physical position to the second physical position is obtained, and the accuracy of controlling the motion mechanism to move is improved.

[0110] The obtaining manner of the skew angle, the skew matrix and the calibration matrix will be described in the embodiments below, which will not be described here.

[0111] Step S204: controlling the motion mechanism to move from the first physical position to the second physical position according to the translation deviation.

[0112] Specifically, the control device can control the motion mechanism to move from the first physical position to the second physical position by the distance of the translation deviation. For example, in the case that the translation deviation includes the x-axis deviation and the y-axis deviation, the control device controls the motion mechanism to move in the x-axis direction by the distance of the x-axis deviation and to move in the y-axis direction by the distance of the y-axis deviation.

[0113] As can be seen from the above, in the scheme provided by the embodiments of the present application, the current image pose, the reference image pose, the calibration matrix, the skew matrix and the rotation angle difference are used to determine the translation deviation of the motion mechanism moving from the first physical position to the second physical position. Since the skew matrix represents the transformation relationship between the physical coordinate system and the orthogonal coordinate, in the case that the two coordinate axes of the physical coordinate system are not orthogonal, the translation deviation between the first physical position corresponding to the reference image pose and the second physical position corresponding to the current image pose in the physical coordinate system can be accurately determined. Therefore, according to the more accurate translation deviation in the physical coordinate system, the motion mechanism can be more accurately controlled to move from the first physical position to the second physical position, and the error of controlling the motion mechanism to move is reduced.

[0114] Therefore, in the case that the two axes of the motion mechanism are not orthogonal, the motion mechanism can also be accurately controlled to move. For the motion mechanism whose x-axis and y-axis are theoretically orthogonal, it is difficult to ensure that the x-axis and y-axis of the motion mechanism are completely orthogonal in actual use. Therefore, the motion mechanism can also be controlled to move by using the scheme provided by the embodiments of the present application, and the error of controlling the motion mechanism to move is reduced.

[0115] The manner of obtaining the skew matrix will be described below.

[0116] In an embodiment of the present application, referring to Figure 3 A flowchart of a skew matrix obtaining method is provided. The above method includes the following steps S301-S303.

[0117] Step S301: obtaining a calibration result of hand-eye calibration of the physical coordinate system and the pixel coordinate system. Step S302: obtaining a skew matrix according to the calibration result. Step S303: obtaining a rotation angle difference between the physical coordinate system and the pixel coordinate system according to the skew matrix.

[0118] Specifically, after the hand-eye calibration of the physical coordinate system and the pixel coordinate system is completed, a calibration coordinate pair including a physical coordinate in the calibrated physical coordinate system and an image coordinate in the pixel coordinate system can be obtained, and a calibration result can be solved according to the calibration coordinate pair, for example, the calibration result can be calculated by a least square method or the like.

[0119] However, in the case of different structures of the two-axis motion mechanism, the manner of hand-eye calibration and the obtained calibration result are different, and the manner of obtaining the calibration matrix and the obtained calibration result are described in the embodiments below, which are not described in detail here.

[0120] Step S302: generating a skew angle of the physical coordinate system relative to the orthogonal coordinate system according to the calibration result.

[0121] The implementation manner of step S302 is described in the embodiments below, which is not described in detail here.

[0122] Step S303: constructing a skew matrix based on the obtained skew angle.

[0123] Specifically, the skew matrix can be constructed according to the preset skew matrix and the obtained skew angle. For example, the obtained skew matrix is:

[0124] In this way, the skew angle used for constructing the skew matrix is generated according to the calibration result, so that the skew angle can be obtained using the existing calibration result, the skew matrix can be easily obtained, the calculation efficiency is improved. Moreover, the skew angle obtained based on the calibration result is more accurate, which can avoid the error caused by measurement, improve the accuracy of the obtained skew angle, improve the accuracy of the determined translation deviation, and reduce the error when controlling the motion mechanism to move.

[0125] The obtained calibration result of step S301 is described below.

[0126] In the first case, in the case of a two-axis separated mechanism, the obtained calibration result is a single-axis calibration matrix corresponding to each single axis. Each single-axis calibration matrix corresponding to each single axis represents the transformation relationship between the orthogonal coordinate system corresponding to the single axis and the pixel coordinate system. The orthogonal coordinate system corresponding to each single axis is an orthogonal coordinate system constructed with the single axis as the coordinate axis. For example, if the first axis of the two-axis motion mechanism is calibrated, the obtained calibration result is M1, and if the second axis of the two-axis motion mechanism is calibrated, the obtained calibration result is M2, M1 and M2 represent two single-axis calibration matrices respectively. M1 represents the transformation relationship between the first orthogonal coordinate system established with the first axis as the reference and the pixel coordinate system, and M2 represents the transformation relationship between the second orthogonal coordinate system established with the second axis as the reference and the pixel coordinate system.

[0127] In the second case, when the motion mechanism is a two-axis integrated mechanism, the calibration result obtained is a calibration matrix.

[0128] The following describes the manner of generating the skew angle in step S302 when the motion mechanism is a two-axis separated mechanism.

[0129] In an implementation, according to the two single-axis calibration matrices, coordinates of the position information in the reference image pose in the two single-axis corresponding orthogonal coordinate systems are obtained; based on the obtained coordinates, a rotation angle between the two single-axis corresponding orthogonal coordinate systems is generated; and according to the rotation angle, the skew angle of the physical coordinate system relative to the orthogonal coordinate system is determined.

[0130] The position information in the reference image pose is the position of the object carried by the motion mechanism during movement,

[0131] Specifically, the single-axis calibration matrix obtained by left multiplying the position information in the reference image pose can be used to obtain the coordinates of the position information in the two reference image poses in the two single-axis corresponding orthogonal coordinate systems.

[0132] For example, referring to Figure 4 Another schematic diagram of a non-orthogonal coordinate system and an orthogonal coordinate system is provided, Figure 4 In the schematic diagram, xoy is a non-orthogonal coordinate system, xoy' is an orthogonal coordinate system corresponding to the x-axis of the non-orthogonal coordinate system, and x'oy is an orthogonal coordinate system corresponding to the y-axis of the non-orthogonal coordinate system.

[0133] If the two single-axis calibration matrices obtained are: the single-axis calibration matrix of the xoy' coordinate system and the xoy coordinate system is M1, and the single-axis calibration matrix of the x'oy coordinate system and the xoy coordinate system is M2. The position information in the reference image pose is P(x base , y base ).

[0134] Then, the coordinates of the position information in the reference image pose in the two single-axis corresponding orthogonal coordinate systems are: in the xoy' coordinate system, in the x'oy coordinate system,

[0135] Then, the rotation angle between the two single-axis corresponding orthogonal coordinate systems is generated according to the following expression: Wherein, α is the rotation angle between the two single-axis corresponding orthogonal coordinate systems, (x1, y1) and (x2, y2) are the coordinates of the position information in the reference image pose in the two single-axis corresponding orthogonal coordinate systems, respectively.

[0136] The expression of the rotation angle is described as follows.

[0137] According to the rotation transformation, it can be determined that the coordinate rotation transformation between the two orthogonal coordinate systems of the xoy' coordinate system and the x'oy coordinate system of the rotation transformation can be performed according to the following expression:

[0138]

[0139] It can be obtained that (x1, y1) and (x2, y2) are brought into the above expression, and the conversion of the above expression is:

[0140]

[0141] Further simplifying the above formula, the following expression is obtained:

[0142]

[0143] According to the above expression, it can be determined that the expression of α is It can be seen that in the case of a two-axis separation mechanism, the rotation angle between the orthogonal coordinate systems can be accurately obtained, and the accurate bevel angle can be obtained, further improving the accuracy of determining the translation deviation and reducing the error when controlling the movement of the motion mechanism.

[0144] After obtaining the rotation angle α between the two single-axis corresponding orthogonal coordinate systems, it can be obtained that the included angle between the x-axis and the y-axis of the xoy coordinate system is α+90°, and further it can be obtained that the bevel angle is the included angle between the x-axis and the y-axis minus the included angle between the x-axis and the y' axis of the xoy' coordinate system, that is, the bevel angle δ is α+90°-90°.

[0145] Next, the way of generating the bevel angle in the case of a two-axis integrated mechanism is described.

[0146] In an implementation manner, a first coefficient in the calibration matrix is used to determine an x-axis scaling coefficient; a second coefficient in the calibration matrix is used to determine a y-axis scaling coefficient; and the bevel angle of the physical coordinate system relative to the orthogonal coordinate system is generated according to the first coefficient, the second coefficient, the x-axis scaling coefficient and the y-axis scaling coefficient.

[0147] The first coefficient represents the scaling and rotation relationship of the x-axis direction between the pixel coordinate system and the physical coordinate system, and the second coefficient represents the scaling, rotation and bevel relationship of the y-axis direction between the pixel coordinate system and the physical coordinate system.

[0148] The obtaining manner of the x-axis scaling coefficient and the y-axis scaling coefficient, and the concept of the first coefficient and the second coefficient are described in the following embodiments, which are not described here in detail.

[0149] Specifically, the skew angle of the physical coordinate system relative to the orthogonal coordinate system is generated according to the following expression:

[0150]

[0151] wherein, δ is the skew angle, D2 is the second coefficient, D1 is the first coefficient, sx is the x-axis scaling coefficient, and sy is the y-axis scaling coefficient.

[0152] The expression of the skew angle is described below.

[0153] In the case of a two-axis integrated mechanism, the pixel coordinate system and the physical coordinate system have an affine transformation relationship, which includes translation transformation, rotation transformation, rigid body transformation, non-uniform scaling transformation, and skew transformation.

[0154] The expression of the translation transformation matrix is as follows:

[0155]

[0156] The expression of the rotation transformation matrix is as follows:

[0157]

[0158] The expression of the non-uniform scaling transformation matrix is as follows:

[0159]

[0160] The expression of the skew transformation matrix is as follows:

[0161]

[0162] When two coordinate systems have translation transformation, rotation transformation, non-uniform scaling transformation, and skew transformation, it is called affine transformation. Affine transformation can be regarded as the composition of non-singular transformation and translation transformation, and the affine transformation matrix can be expressed as:

[0163]

[0164] Simplifying the above expression, the following expression can be obtained:

[0165]

[0166] In fact, in the case of a two-axis integrated mechanism, the pixel coordinate system and the physical coordinate system have an affine transformation, and the obtained calibration matrix is an affine transformation matrix. If the calibration matrix M is expressed according to the following expression: HAE :

[0167]

[0168] wherein, Mij To calibrate the element in the i-th row and j-th column of the matrix.

[0169] Then the following expression of M HAE = M offine can be obtained:

[0170]

[0171] Further, it can be seen that M 11 and M 21 characterize the scaling and rotation relationship of the x-axis direction between the pixel coordinate system and the physical coordinate system, and the first coefficient can include M 11 and M 21 . M 12 and M 22 characterize the scaling, rotation and skew relationship of the y-axis direction between the pixel coordinate system and the physical coordinate system, and the second coefficient can include M 12 and M 22 . Then, the x-axis scaling coefficient and the y-axis scaling coefficient can be determined according to the following expression:

[0172]

[0173] According to the expression of M HAE = M offine , the following can be obtained:

[0174]

[0175] Therefore, the skew angle can be obtained according to the following expression:

[0176]

[0177] Similarly, the skew angle can also be obtained according to the following expression:

[0178]

[0179] θ+δ can also be expressed by the following expression:

[0180]

[0181] Then, the skew angle can also be obtained according to the following expression:

[0182]

[0183] It can be seen that in the case of a two-axis integrated mechanism, the rotation angle between the orthogonal coordinate systems can be accurately obtained, and the skew angle can be accurately obtained, thereby further improving the accuracy of determining the translation deviation and reducing the error when controlling the movement of the motion mechanism.

[0184] In this way, the beveling angle is generated for two cases of the two-axis integrated mechanism and the two-axis separated mechanism respectively, different ways are adopted to generate the beveling angle for different cases, the accuracy of generating the beveling angle is further improved, the accuracy of determining the translation deviation is improved, and the error in controlling the movement of the motion mechanism is reduced.

[0185] The way of obtaining the calibration matrix is described below.

[0186] In an embodiment of the present application, in the case of the two-axis separated mechanism, the calibration matrix is obtained in the following way.

[0187] Referring to Figure 5 A flowchart of a calibration matrix obtaining method is provided, and the method includes the following steps S501-S505.

[0188] Step S501: Obtain single-axis calibration matrices obtained by respectively performing hand-eye calibration on two single axes in the motion mechanism.

[0189] Each single-axis calibration matrix corresponding to each single axis represents the transformation relationship between the orthogonal coordinate system corresponding to the single axis and the pixel coordinate system. The orthogonal coordinate system corresponding to each single axis is an orthogonal coordinate system constructed with the single axis as the coordinate axis. When performing hand-eye calibration on each single axis, the calibration data is collected under the condition that the single axis is moved and the other single axis is fixed. The calibration data includes multiple sets of calibration coordinate pairs of image coordinates in the pixel coordinate system and physical coordinates in the physical coordinate system. The image coordinates and the physical coordinates in the calibration coordinate pairs are coordinates collected synchronously after the motion mechanism moves.

[0190] Specifically, assuming that the two single axes in the motion mechanism are x-axis and y-axis, when performing hand-eye calibration on the x-axis, the x-axis of the motion mechanism is controlled to move at least twice. After each movement of the x-axis of the motion mechanism, the image acquisition device collects the image of the reference object on the motion mechanism, identifies the image coordinates of the reference object on the motion mechanism in the pixel coordinate system, and records the physical coordinates of the reference object on the motion mechanism at that time. The obtained image coordinates and the physical coordinates corresponding to the image coordinates are taken as the calibration data of the x-axis. Similarly, the calibration data of the y-axis is obtained in the above manner.

[0191] Then, the single-axis calibration matrix of the x-axis is generated according to the calibration data of the x-axis, and the single-axis calibration matrix of the y-axis is generated according to the calibration data of the y-axis.

[0192] Step S502: Obtain the coordinates of the position information in the reference image pose in the orthogonal coordinate systems corresponding to the two single axes respectively according to the two single-axis calibration matrices.

[0193] The implementation of step S502 is the same as the way of obtaining the position information in the coordinate of the two single-axis corresponding orthogonal coordinate systems in the above-mentioned way of generating the bevel angle in step S302, and will not be described in detail here.

[0194] Step S503: Based on the obtained coordinates, the rotation angle between the two single-axis corresponding orthogonal coordinate systems is generated.

[0195] The implementation of step S503 is the same as the way of generating the rotation angle between the two single-axis corresponding orthogonal coordinate systems based on the obtained coordinates in the above-mentioned way of generating the bevel angle in step S302, and will not be described in detail here.

[0196] Step S504: Based on the rotation angle, the included angle between the two single axes is obtained.

[0197] Specifically, taking Figure 4 for example, after obtaining the rotation angle α, the included angle between the x-axis and the y-axis of the xoy coordinate system can be α+90°.

[0198] Step S505: Based on the included angle, the bases on the two single axes are generated, and the generated bases are used to determine the calibration matrix.

[0199] Wherein, the two bases on the single axis are unit vectors in the direction of the two single axes.

[0200] For example, taking Figure 4 for example, xoy' is the orthogonal coordinate system corresponding to the x-axis in the non-orthogonal coordinate system, and the included angle is: , the generated base on the x-axis can be:

[0201]

[0202] The generated base on the y-axis can be:

[0203]

[0204] Then, for the coordinates between the xoy' coordinate system and the xoy coordinate system, there is the following coordinate transformation relationship:

[0205]

[0206] Wherein, P xoy’ (x xoy’ ,y xoy’ ) represents the coordinates in the xoy' coordinate system, and P xoy (x xoy ,y xoy ) represents the coordinates in the xoy coordinate system.

[0207] Assuming that the single-axis calibration matrix corresponding to the x-axis is M1, and there is a coordinate P(x, y) in any pixel coordinate system i (x i ,y i ), then P xoy’ (x xoy’ ,y xoy’ ) = M1*P i (x i’ ,y i ).

[0208] Further, the following expression can be obtained:

[0209]

[0210] The transformation relationship expression between the coordinate in the pixel coordinate system and the coordinate in the xoy coordinate system is obtained:

[0211]

[0212] Therefore, the M1 in the above expression can be used as the calibration matrix.

[0213] In an embodiment of the present application, in the case of a two-axis integrated mechanism, the calibration matrix is obtained in the following manner: obtaining the image coordinates of the reference object corresponding to the motion mechanism at different positions and the physical coordinates of the reference object corresponding to the motion mechanism, and constructing the calibration matrix based on the obtained image coordinates and physical coordinates.

[0214] Wherein, the different positions are not collinear. That is, at least three or more positions in the different positions are not on the same straight line. The image coordinates are the coordinates of the reference object in the image collected by the image collection device. The reference object can be the object gripped by the motion mechanism, or a mark arranged on the motion mechanism, or a calibration plate, etc.

[0215] The obtained image coordinates of the reference object corresponding to the motion mechanism and the physical coordinates of the reference object corresponding to the motion mechanism are the image coordinates and physical coordinates of the reference object at the position after the motion mechanism moves to the position. The obtained image coordinates and physical coordinates are the calibration coordinate pairs at the position.

[0216] For example, after controlling the motion to a position, the image collection device collects the image of the reference object of the motion mechanism, and then identifies the image coordinates of the reference object, and records the physical coordinates of the reference object as a calibration coordinate pair. Then, after controlling the motion to another position, the image coordinates and physical coordinates are collected again as a calibration coordinate pair. Until at least three groups of calibration coordinate pairs corresponding to different positions and not on the same straight line are obtained. Then, the calibration matrix between the image coordinates and the physical coordinates is solved based on the calibration coordinate pairs.​

[0217] It can be seen that, for the two-axis integrated mechanism and the two-axis separated mechanism, the calibration matrix is generated in different ways, and the calibration matrix can be accurately obtained, and the accuracy of the determined translation deviation is improved, and the error of the motion mechanism during control is reduced.

[0218] Corresponding to the motion mechanism control method, the embodiment of the application provides a motion mechanism control device.

[0219] Referring to Figure 6 A structure diagram of a motion mechanism control device is provided, and the device comprises:

[0220] An image pose obtaining module 601 is configured to obtain a current image pose of an object to be grabbed by the motion mechanism in an image collected by an image collection device;

[0221] An angle difference determining module 602 is configured to determine a rotation angle difference between the current image pose and a reference image pose of the object;

[0222] A translation deviation determining module 603 is configured to determine a translation deviation of the motion mechanism from a first physical position to a second physical position according to the current image pose, the reference image pose, a calibration matrix, a skew matrix, and the rotation angle difference, wherein the calibration matrix represents a transformation relationship between a pixel coordinate system of the image collection device and a physical coordinate system of the motion mechanism, the skew matrix represents a transformation relationship between the physical coordinate system and an orthogonal coordinate system, the first physical position is a position corresponding to the reference image pose in the physical coordinate system, and the second physical position is a position corresponding to the current image pose in the physical coordinate system;

[0223] A motion mechanism control module 604 is configured to control the motion mechanism to move from the first physical position to the second physical position according to the translation deviation.

[0224] As can be seen from the above, in the scheme provided by the embodiment of the application, the current image pose, the reference image pose, the calibration matrix, the skew matrix, and the rotation angle difference are used to determine the translation deviation of the motion mechanism from the first physical position to the second physical position. Since the skew matrix represents the transformation relationship between the physical coordinate system and the orthogonal coordinate, in the case that the two coordinate axes of the physical coordinate system are not orthogonal, the translation deviation between the first physical position corresponding to the reference image pose and the second physical position corresponding to the current image pose in the physical coordinate system can be accurately determined. In this way, the motion mechanism can be more accurately controlled to move from the first physical position to the second physical position according to the more accurate translation deviation in the physical coordinate system, and the error of the motion mechanism during control is reduced.

[0225] In one embodiment of the present application, the translation deviation determination module comprises: a first physical position determination unit configured to determine the first physical position according to the position information in the calibration matrix and the reference image pose; a second physical position determination unit configured to determine the second physical position according to the position information in the calibration matrix, the skew matrix, the rotation angle difference and the current image pose; and a difference calculation unit configured to calculate the difference between the first physical position and the second physical position as the translation deviation of the motion mechanism moving from the first physical position to the second physical position.

[0226] In this way, the second position information can be determined more accurately according to the position information in the calibration matrix, the skew matrix, the rotation angle difference and the current image pose, and the difference between the first physical position and the second physical position can be determined accurately, so as to obtain accurate translation deviation of the motion mechanism moving from the first physical position to the second physical position, and improve the accuracy of controlling the motion mechanism to move.

[0227] In one embodiment of the present application, the second physical position determination unit is specifically configured to determine the second physical position according to the following expression: wherein, is the inverse matrix of the skew matrix, and δ is the skew angle, is the rotation transformation matrix, Δθ is the rotation angle difference, and M HAE is the calibration matrix, (x cur , y cur ) is the coordinate of the position information in the current image pose in the pixel coordinate system.

[0228] As can be seen, the second physical position can be determined more accurately according to the above expression, and the accuracy of the determined translation deviation can be improved.

[0229] In one embodiment of the present application, the skew matrix is obtained in the following manner: obtaining a calibration result of hand-eye calibration of the physical coordinate system and the pixel coordinate system; generating a skew angle of the physical coordinate system relative to the orthogonal coordinate system according to the calibration result; and constructing the skew matrix based on the obtained skew angle.

[0230] In this way, the skew angle used for constructing the skew matrix is generated according to the calibration result, so that the skew angle can be obtained using the existing calibration result, the skew matrix can be easily obtained, and the calculation efficiency is improved. Moreover, the skew angle obtained based on the calibration result is more accurate, the error caused by measurement can be avoided, the accuracy of the obtained skew angle is improved, the accuracy of the determined translation deviation is improved, and the error of controlling the motion mechanism to move is reduced.

[0231] In one embodiment of this application, if the motion mechanism is a two-axis split mechanism, the calibration result includes a single-axis calibration matrix corresponding to each single axis. Each single-axis calibration matrix represents the transformation relationship between the orthogonal coordinate system and the pixel coordinate system corresponding to that single axis. The orthogonal coordinate system corresponding to each single axis is an orthogonal coordinate system constructed with that single axis as the coordinate axis. Based on the calibration result, generating the tangent angle of the physical coordinate system relative to the orthogonal coordinate system includes: obtaining the coordinates of the position information in the reference image pose in the two orthogonal coordinate systems corresponding to the single axis based on the two single-axis calibration matrices; generating the rotation angle between the two orthogonal coordinate systems corresponding to the single axis based on the obtained coordinates; and determining the physical coordinate system based on the rotation angle. The tangent angle of the physical coordinate system relative to the orthogonal coordinate system; or, if the motion mechanism is a two-axis integrated mechanism, the calibration result is a calibration matrix; based on the calibration result, the tangent angle of the physical coordinate system relative to the orthogonal coordinate system is generated, including: using the first coefficient in the calibration matrix to determine the x-axis scaling factor, wherein the first coefficient characterizes the scaling and rotation relationship between the pixel coordinate system and the x-axis direction in the physical coordinate system; using the second coefficient in the calibration matrix to determine the y-axis scaling factor, wherein the second coefficient characterizes the scaling, rotation, and tangent relationship between the pixel coordinate system and the y-axis direction in the physical coordinate system; based on the first coefficient, the second coefficient, the x-axis scaling factor, and the y-axis scaling factor, the tangent angle of the physical coordinate system relative to the orthogonal coordinate system is generated.

[0232] In this way, oblique angles are generated for both the two-axis integrated mechanism and the two-axis separate mechanism. Different methods are used to generate oblique angles for different situations, which further improves the accuracy of generating oblique angles, improves the accuracy of determining translation deviation, and reduces the error when controlling the movement of the motion mechanism.

[0233] In one embodiment of this application, generating a rotation angle between two orthogonal coordinate systems corresponding to two single axes based on the obtained coordinates includes: generating the rotation angle between the two orthogonal coordinate systems corresponding to two single axes according to the following expression: Where α is the rotation angle between the two orthogonal coordinate systems corresponding to the two single axes, and (x1, y1) and (x2, y2) are the coordinates of the position information in the pose of the reference image in the two orthogonal coordinate systems corresponding to the two single axes, respectively; or, based on the first coefficient, the second coefficient, the x-axis scaling coefficient, and the y-axis scaling coefficient, the oblique angle of the physical coordinate system relative to the orthogonal coordinate system is generated, including: generating the oblique angle of the physical coordinate system relative to the orthogonal coordinate system according to the following expression: Where δ is the chamfer angle, D2 is the second coefficient, D1 is the first coefficient, sx is the x-axis scaling factor, and sy is the y-axis scaling factor.

[0234] It can be seen that, in the case of the two-axis separated mechanism, the rotation angle between the orthogonal coordinate systems can be accurately obtained, and then the accurate beveling angle can be obtained, and the accuracy of determining the translation deviation is further improved, and the error of controlling the movement of the motion mechanism is reduced. In the case of the two-axis integrated mechanism, the rotation angle between the orthogonal coordinate systems can be accurately obtained, and then the accurate beveling angle can be obtained, and the accuracy of determining the translation deviation is further improved, and the error of controlling the movement of the motion mechanism is reduced.

[0235] In an embodiment of the present application, if the motion mechanism is a two-axis separated mechanism, the calibration matrix is obtained in the following manner: obtaining single-axis calibration matrices obtained by respectively performing hand-eye calibration on two single axes in the motion mechanism, wherein each single-axis calibration matrix corresponding to a single axis represents the transformation relationship between the orthogonal coordinate system corresponding to the single axis and the pixel coordinate system, and each single-axis calibration matrix corresponding to a single axis is obtained by moving the single axis and fixing the other single axis when collecting calibration data during hand-eye calibration of each single axis; obtaining coordinates of the position information in the reference image pose in the orthogonal coordinate systems corresponding to the two single axes according to the two single-axis calibration matrices; generating the rotation angle between the orthogonal coordinate systems corresponding to the two single axes based on the obtained coordinates; obtaining the included angle between the two single axes based on the rotation angle; generating the base on the two single axes based on the included angle, and determining the calibration matrix using the generated base and the obtained single-axis calibration matrices; or, if the motion mechanism is a two-axis integrated mechanism, the calibration matrix is obtained in the following manner: obtaining image coordinates and physical coordinates of a reference object corresponding to the motion mechanism when the reference object is located at different positions, wherein the different positions are not collinear; constructing the calibration matrix based on the obtained image coordinates and physical coordinates, wherein the image coordinates are the coordinates of the reference object in the image collected by the image collection device.

[0236] It can be seen that, for the two-axis integrated mechanism and the two-axis separated mechanism, the calibration matrix is generated in different ways, which can accurately obtain the calibration matrix, and then improve the accuracy of determining the translation deviation and reduce the error of controlling the movement of the motion mechanism.

[0237] The present application also provides an electronic device, as shown in the accompanying drawings, comprising: Figure 7

[0238] The memory 701 is used to store computer programs.

[0239] The processor 702 is used to execute the programs stored in the memory 701, and implement the motion mechanism control method in any of the above embodiments.

[0240] ​The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0241] The communication interface is used for communication between the above electronic device and other devices.

[0242] The memory can include a Random Access Memory (RAM) and can also include a Non-Volatile Memory (NVM), for example, at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0243] The processor mentioned above can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0244] In another embodiment provided in the application, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of any of the above motion mechanism control methods are implemented.

[0245] In another embodiment provided in the application, a computer program product containing instructions is also provided, and when the computer program product is run on a computer, the computer is caused to execute any of the motion mechanism control methods in the above embodiments.

[0246] In the embodiments described above, all or some of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or some of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded into and executed by a computer, all or some of the processes or functions according to the embodiments described in the specification are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), an optical medium (for example, DVD), or a solid state disk (SSD) and the like.

[0247] It should be noted that, in this document, the terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0248] Each of the embodiments in the specification is described in a related manner, and the same or similar parts between each of the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the method, device, electronic device, storage medium and computer program product, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0249] The above merely provides the preferred embodiment of the present application, and not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for controlling a motion mechanism, characterized in that, The method includes: Obtain the current image pose of the object to be grasped by the motion mechanism in the image captured by the image acquisition device; Determine the rotation angle difference between the current image pose and the reference image pose of the object; Based on the current image pose, the reference image pose, the calibration matrix, the slant matrix, and the rotation angle difference, the translational deviation of the motion mechanism from the first physical position to the second physical position is determined. The calibration matrix represents the transformation relationship between the pixel coordinate system of the image acquisition device and the physical coordinate system of the motion mechanism. The slant matrix represents the transformation relationship between the physical coordinate system and the orthogonal coordinate system. The first physical position is the position corresponding to the reference image pose in the physical coordinate system, and the second physical position is the position corresponding to the current image pose in the physical coordinate system. According to the translational deviation, the motion mechanism is controlled to move from the first physical position to the second physical position.

2. The method according to claim 1, characterized in that, The step of determining the translational deviation of the motion mechanism from the first physical position to the second physical position based on the current image pose, the reference image pose, the calibration matrix, the skewing matrix, and the rotation angle difference includes: The first physical position is determined based on the calibration matrix and the position information in the pose of the reference image; The second physical position is determined based on the calibration matrix, the skewing matrix, the rotation angle difference, and the position information in the current image pose. The difference between the first physical position and the second physical position is calculated as the translational deviation of the motion mechanism from the first physical position to the second physical position.

3. The method according to claim 2, characterized in that, Determining the second physical position based on the calibration matrix, the skewing matrix, the rotation angle difference, and the position information in the current image pose includes: The second physical location is determined according to the following expression: in, Let be the inverse of the tangent matrix, and δ be the tangent angle. Let M be the rotation transformation matrix, Δθ be the rotation angle difference, and M be the rotation transformation matrix. HAE For the calibration matrix, (x cur y cur ) represents the coordinates of the position information in the current image pose in the pixel coordinate system.

4. The method according to any one of claims 1-3, characterized in that, The tangent matrix is ​​obtained as follows: Obtain the calibration results of hand-eye calibration of the physical coordinate system and the pixel coordinate system; Based on the calibration results, the oblique angle of the physical coordinate system relative to the orthogonal coordinate system is generated; Based on the obtained tangent angle, construct the tangent matrix.

5. The method according to claim 4, characterized in that, If the motion mechanism is a two-axis split mechanism, the calibration result includes a single-axis calibration matrix corresponding to each single axis, wherein the single-axis calibration matrix corresponding to each single axis represents the transformation relationship between the orthogonal coordinate system corresponding to that single axis and the pixel coordinate system, and the orthogonal coordinate system corresponding to each single axis is: an orthogonal coordinate system constructed with that single axis as the coordinate axis; the step of generating the tangent angle of the physical coordinate system relative to the orthogonal coordinate system according to the calibration result includes: obtaining the coordinates of the position information in the pose of the reference image in the two orthogonal coordinate systems corresponding to the single axis according to the two single-axis calibration matrices; generating the rotation angle between the two orthogonal coordinate systems corresponding to the single axis based on the obtained coordinates; and determining the tangent angle of the physical coordinate system relative to the orthogonal coordinate system according to the rotation angle; If the motion mechanism is a two-axis integrated mechanism, the calibration result is the calibration matrix; generating the tangent angle of the physical coordinate system relative to the orthogonal coordinate system based on the calibration result includes: using a first coefficient in the calibration matrix to determine the x-axis scaling factor, wherein the first coefficient characterizes the scaling and rotation relationship between the pixel coordinate system and the x-axis direction in the physical coordinate system; using a second coefficient in the calibration matrix to determine the y-axis scaling factor, wherein the second coefficient characterizes the scaling, rotation, and tangent relationship between the pixel coordinate system and the y-axis direction in the physical coordinate system; and generating the tangent angle of the physical coordinate system relative to the orthogonal coordinate system based on the first coefficient, the second coefficient, the x-axis scaling factor, and the y-axis scaling factor.

6. The method according to claim 5, characterized in that, In the case that the motion mechanism is a two-axis split mechanism, generating the rotation angle between the two orthogonal coordinate systems corresponding to the two single axes based on the obtained coordinates includes: generating the rotation angle between the two orthogonal coordinate systems corresponding to the two single axes according to the following expression: Where α is the rotation angle between the two orthogonal coordinate systems corresponding to the two single axes, and (x1, y1) and (x2, y2) are the coordinates of the position information in the pose of the reference image in the two orthogonal coordinate systems corresponding to the two single axes, respectively. When the motion mechanism is a two-axis integrated mechanism, generating the tangent angle of the physical coordinate system relative to the orthogonal coordinate system based on the first coefficient, the second coefficient, the x-axis scaling coefficient, and the y-axis scaling coefficient includes: generating the tangent angle of the physical coordinate system relative to the orthogonal coordinate system according to the following expression: Wherein, δ is the chamfer angle, D2 is the second coefficient, D1 is the first coefficient, sx is the x-axis scaling factor, and sy is the y-axis scaling factor.

7. The method according to any one of claims 1-3, characterized in that, If the motion mechanism is a two-axis split mechanism, the calibration matrix is ​​obtained as follows: A single-axis calibration matrix is ​​obtained by performing hand-eye calibration on each of the two single axes of the motion mechanism. Each single-axis calibration matrix represents the transformation relationship between the orthogonal coordinate system corresponding to that single axis and the pixel coordinate system. Each single-axis orthogonal coordinate system is an orthogonal coordinate system constructed with that single axis as the coordinate axis. During hand-eye calibration of each single axis, calibration data is collected while moving that single axis and fixing the other single axis. Based on the two single-axis calibration matrices, the coordinates of the position information in the pose of the reference image in the orthogonal coordinate systems corresponding to the two single axes are obtained. Based on the obtained coordinates, a rotation angle between the two orthogonal coordinate systems corresponding to the two single axes is generated. Based on the rotation angle, the included angle between the two single axes is obtained. Based on the included angle, a basis is generated on the two single axes, and the generated basis and the obtained single-axis calibration matrix are used to determine the calibration matrix. If the motion mechanism is a two-axis integrated mechanism, the calibration matrix is ​​obtained in the following manner: obtain the image coordinates of the reference object corresponding to the motion mechanism when it is located at different positions and the physical coordinates of the reference object corresponding to the motion mechanism, wherein the different positions are not collinear; construct a calibration matrix based on the obtained image coordinates and physical coordinates, wherein the image coordinates are: the coordinates of the reference object in the image acquired by the image acquisition device.

8. A motion mechanism control device, characterized in that, The device includes: The image pose acquisition module is used to obtain the current image pose of the object to be grasped by the motion mechanism in the image acquired by the image acquisition device; Angle difference determination module is used to determine the rotation angle difference between the current image pose and the reference image pose of the object; The translation deviation determination module is used to determine the translation deviation of the motion mechanism from a first physical position to a second physical position based on the current image pose, the reference image pose, the calibration matrix, the slant matrix, and the rotation angle difference. The calibration matrix represents the transformation relationship between the pixel coordinate system of the image acquisition device and the physical coordinate system of the motion mechanism, and the slant matrix represents the transformation relationship between the physical coordinate system and the orthogonal coordinate system. The first physical position is the position corresponding to the reference image pose in the physical coordinate system, and the second physical position is the position corresponding to the current image pose in the physical coordinate system. The motion mechanism control module is used to control the motion mechanism to move from the first physical position to the second physical position according to the translational deviation.

9. The apparatus according to claim 8, characterized in that, The translation deviation determination module includes: a first physical position determination unit, used to determine a first physical position based on the calibration matrix and the position information in the reference image pose; a second physical position determination unit, used to determine a second physical position based on the calibration matrix, the slant matrix, the rotation angle difference, and the position information in the current image pose; and a difference calculation unit, used to calculate the difference between the first physical position and the second physical position as the translation deviation of the motion mechanism moving from the first physical position to the second physical position.

10. The apparatus according to claim 9, characterized in that, The second physical location determination unit is specifically used to determine the second physical location according to the following expression: in, Let be the inverse of the tangent matrix, and δ be the tangent angle. Let M be the rotation transformation matrix, Δθ be the rotation angle difference, and M be the rotation transformation matrix. HAE For the calibration matrix, (x cur y cur ) represents the coordinates of the position information in the current image pose in the pixel coordinate system.

11. The apparatus according to any one of claims 8-10, characterized in that, The slant matrix is ​​obtained as follows: the calibration result of hand-eye calibration of the physical coordinate system and the pixel coordinate system is obtained; based on the calibration result, the slant angle of the physical coordinate system relative to the orthogonal coordinate system is generated. Based on the obtained tangent angle, construct the tangent matrix.

12. The apparatus according to claim 11, characterized in that, If the motion mechanism is a two-axis split mechanism, the calibration result includes a single-axis calibration matrix corresponding to each single axis, wherein the single-axis calibration matrix corresponding to each single axis represents the transformation relationship between the orthogonal coordinate system corresponding to that single axis and the pixel coordinate system, and the orthogonal coordinate system corresponding to each single axis is: an orthogonal coordinate system constructed with that single axis as the coordinate axis; the step of generating the tangent angle of the physical coordinate system relative to the orthogonal coordinate system according to the calibration result includes: obtaining the coordinates of the position information in the pose of the reference image in the two orthogonal coordinate systems corresponding to the single axis according to the two single-axis calibration matrices; generating the rotation angle between the two orthogonal coordinate systems corresponding to the single axis based on the obtained coordinates; and determining the tangent angle of the physical coordinate system relative to the orthogonal coordinate system according to the rotation angle; If the motion mechanism is a two-axis integrated mechanism, the calibration result is the calibration matrix; generating the tangent angle of the physical coordinate system relative to the orthogonal coordinate system based on the calibration result includes: using a first coefficient in the calibration matrix to determine the x-axis scaling factor, wherein the first coefficient characterizes the scaling and rotation relationship between the pixel coordinate system and the x-axis direction in the physical coordinate system; using a second coefficient in the calibration matrix to determine the y-axis scaling factor, wherein the second coefficient characterizes the scaling, rotation, and tangent relationship between the pixel coordinate system and the y-axis direction in the physical coordinate system; and generating the tangent angle of the physical coordinate system relative to the orthogonal coordinate system based on the first coefficient, the second coefficient, the x-axis scaling factor, and the y-axis scaling factor.

13. The apparatus according to claim 12, characterized in that, In the case that the motion mechanism is a two-axis split mechanism, generating the rotation angle between the two orthogonal coordinate systems corresponding to the two single axes based on the obtained coordinates includes: generating the rotation angle between the two orthogonal coordinate systems corresponding to the two single axes according to the following expression: Where α is the rotation angle between the two orthogonal coordinate systems corresponding to the two single axes, and (x1, y1) and (x2, y2) are the coordinates of the position information in the pose of the reference image in the two orthogonal coordinate systems corresponding to the two single axes, respectively. When the motion mechanism is a two-axis integrated mechanism, generating the tangent angle of the physical coordinate system relative to the orthogonal coordinate system based on the first coefficient, the second coefficient, the x-axis scaling coefficient, and the y-axis scaling coefficient includes: generating the tangent angle of the physical coordinate system relative to the orthogonal coordinate system according to the following expression: Wherein, δ is the chamfer angle, D2 is the second coefficient, D1 is the first coefficient, sx is the x-axis scaling factor, and sy is the y-axis scaling factor.

14. The apparatus according to any one of claims 8-10, characterized in that, If the motion mechanism is a two-axis split mechanism, the calibration matrix is ​​obtained as follows: A single-axis calibration matrix is ​​obtained by performing hand-eye calibration on each of the two single axes of the motion mechanism. Each single-axis calibration matrix represents the transformation relationship between the orthogonal coordinate system corresponding to that single axis and the pixel coordinate system. Each single-axis orthogonal coordinate system is an orthogonal coordinate system constructed with that single axis as the coordinate axis. During hand-eye calibration of each single axis, calibration data is collected while moving that single axis and fixing the other single axis. Based on the two single-axis calibration matrices, the coordinates of the position information in the pose of the reference image in the orthogonal coordinate systems corresponding to the two single axes are obtained. Based on the obtained coordinates, a rotation angle between the two orthogonal coordinate systems corresponding to the two single axes is generated. Based on the rotation angle, the included angle between the two single axes is obtained. Based on the included angle, a basis is generated on the two single axes, and the generated basis and the obtained single-axis calibration matrix are used to determine the calibration matrix. If the motion mechanism is a two-axis integrated mechanism, the calibration matrix is ​​obtained in the following manner: obtain the image coordinates of the reference object corresponding to the motion mechanism when it is located at different positions and the physical coordinates of the reference object corresponding to the motion mechanism, wherein the different positions are not collinear; construct a calibration matrix based on the obtained image coordinates and physical coordinates, wherein the image coordinates are: the coordinates of the reference object in the image acquired by the image acquisition device.

15. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-7.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

Images