Composite mark-based execution arm detection method and robot system

Abstract

The present disclosure relates to the technical field of detection, and discloses an execution arm detection method. The execution arm detection method comprises: determining a driving signal for controlling a target pose of a terminal end of an execution arm, the driving signal corresponding to the target pose; acquiring a positioning image; in the positioning image, identifying a plurality of marks located on the terminal end of the execution arm, the plurality of marks comprising a plurality of pose marks for identifying a pose and at least one composite mark for identifying a pose and an angle; determining an actual pose of the terminal end of the execution arm based on the at least one composite mark and the plurality of pose marks; and determining a performance of the execution arm based on the target pose and the actual pose.

Description

Technical Field

[0001] This disclosure relates to the field of detection technology, and in particular to an actuator arm detection method and robot system based on composite marking. Background Technology

[0002] Before the robot starts working, the performance of the actuator arm needs to be tested. Among them, the accuracy of the actuator arm's pose is an important standard for measuring the performance of the actuator arm.

[0003] The pose of an actuator arm includes the position and orientation of its end effector. Typically, the pose of the actuator arm can be measured using devices such as laser trackers. Based on the obtained pose, it can be determined whether the end effector has moved to the desired position and orientation, thereby determining the actuator arm's performance. However, devices such as laser trackers have technical drawbacks, including high cost, inability to measure pose in a single measurement, and the need for periodic calibration.

[0004] Therefore, there is a need to provide a simple and effective method for detecting the performance of the actuator arm in order to assess its quality. Summary of the Invention

[0005] In some embodiments, this disclosure provides an actuator arm detection method. The method may include: determining a drive signal for controlling a target pose of the actuator arm's end effector, the drive signal corresponding to the target pose; acquiring a positioning image; identifying multiple markers located on the end effector of the actuator arm in the positioning image, the multiple markers including multiple pose markers for identifying pose and at least one composite marker for identifying pose and angle; determining the actual pose of the actuator arm's end effector based on at least one composite marker and the multiple pose markers; and determining the performance of the actuator arm based on the target pose and the actual pose.

[0006] In some embodiments, this disclosure provides a computer device including: a memory for storing at least one instruction; and a processor coupled to the memory and configured to execute at least one instruction to perform a method according to any of some embodiments of this disclosure.

[0007] In some embodiments, this disclosure provides a computer-readable storage medium for storing at least one instruction, which, when executed by a computer, causes the computer to perform any of the methods described in some embodiments of this disclosure.

[0008] In some embodiments, this disclosure provides a robot system, including: an actuator arm, the end of which is provided with a plurality of identifiers, the plurality of identifiers including a plurality of pose identifiers and at least one composite identifier; at least one drive device for driving the actuator arm; an image acquisition device for acquiring positioning images of the actuator arm; and a control device configured to be connected to the at least one drive device and the image acquisition device to perform the method of any one of the embodiments of this disclosure. Attached Figure Description

[0009] Figure 1 A schematic diagram of an actuator detection system according to some embodiments of the present disclosure is shown;

[0010] Figure 2 A schematic diagram of the structure of an actuator arm according to some embodiments of the present disclosure is shown;

[0011] Figure 3 This diagram illustrates the structure of an actuator arm according to some embodiments of the present disclosure;

[0012] Figure 4 A schematic diagram illustrating a label including multiple identifiers according to some embodiments of the present disclosure;

[0013] Figure 5 A schematic diagram showing a label disposed on the periphery of the end of an actuator arm and formed into a cylindrical shape according to some embodiments of the present disclosure;

[0014] Figure 6 Schematic diagrams illustrating implementation scenarios according to some embodiments of the present disclosure;

[0015] Figure 7 A flowchart illustrating a detection method for an actuator detection system according to some embodiments of the present disclosure is shown.

[0016] Figure 8 A flowchart illustrating a method for determining a drive signal for controlling the target pose of the end effector arm according to some embodiments of the present disclosure;

[0017] Figure 9 A flowchart illustrating a method for determining a drive signal for controlling the target pose of the end effector arm according to other embodiments of the present disclosure;

[0018] Figure 10 A flowchart illustrating a method for updating a target joint parameter set of a joint of an actuator arm according to some embodiments of the present disclosure;

[0019] Figure 11 A flowchart illustrating a method for determining the actual pose of the end effector of an actuator arm according to some embodiments of the present disclosure;

[0020] Figure 12A flowchart illustrating a method for determining the actual pose of the end effector of an actuator arm according to other embodiments of the present disclosure;

[0021] Figure 13 A flowchart illustrating a method for identifying an identifier according to some embodiments of the present disclosure is shown;

[0022] Figure 14 A schematic diagram showing pose identification patterns according to some embodiments of the present disclosure;

[0023] Figure 15 A flowchart illustrating a method for searching identifiers according to some embodiments of the present disclosure is shown;

[0024] Figure 16 A schematic diagram illustrating search identifiers according to some embodiments of the present disclosure;

[0025] Figure 17 A schematic block diagram of a computer device according to some embodiments of the present disclosure is shown;

[0026] Figure 18 A schematic diagram of a robot system according to some embodiments of the present disclosure is shown. Detailed Implementation

[0027] To make the technical problems solved by this disclosure, the technical solutions adopted, and the technical effects achieved clearer, the technical solutions of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that the described embodiments should be considered exemplary in all respects and not restrictive, and are merely exemplary embodiments of this disclosure, not all embodiments.

[0028] In this disclosure, the term "position" refers to the location of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom can be described using variations in Cartesian X, Y, and Z coordinates, such as three translational degrees of freedom along the Cartesian X, Y, and Z axes, respectively). In this disclosure, the term "attitude" refers to the rotational setting of an object or part of an object (e.g., three rotational degrees of freedom, which can be described using roll, pitch, and yaw). In this disclosure, the term "pose" refers to a combination of the position and attitude of an object or part of an object, which can be described, for example, using six parameters from the six degrees of freedom mentioned above.

[0029] In this disclosure, a reference coordinate system can be understood as a coordinate system capable of describing the pose of an object. Depending on the actual positioning requirements, the reference coordinate system can be selected with the origin of a virtual reference object or the origin of a physical reference object as its origin. In some embodiments, the reference coordinate system can be a world coordinate system, a camera coordinate system, or the operator's own perception coordinate system, etc. Here, the camera coordinate system refers to the coordinate system in which the image acquisition device is located.

[0030] In this disclosure, an object can be understood as an object or target that needs to be located, such as an actuator arm or the end effector of an actuator arm. The pose of the actuator arm or a portion thereof can refer to the pose of the coordinate system defined by the actuator arm or a portion thereof relative to a reference coordinate system.

[0031] Figure 1 A schematic diagram of an actuator detection system 100 (hereinafter also referred to as "System 100") according to some embodiments of the present disclosure is shown. Figure 1 As shown, system 100 may include a control device 120, an actuator arm 130, and an image acquisition device 150. The control device 120 may be communicatively connected to the image acquisition device 150 and the actuator arm 130 (e.g., a drive mechanism for the actuator arm 130). In some embodiments, the actuator arm 130 may include an actuator arm end effector 131 at its distal or end position. In some embodiments, an end effector 140 may be provided at the distal end of the actuator arm end effector 131.

[0032] In some embodiments, the control device 120 can be used to control the movement of the actuator arm 130 to adjust the position and orientation of the actuator arm 130, etc. The control device 120 can control the movement of the actuator arm 130 to move the actuator arm end cap 131 to a desired position and orientation. In some embodiments, such as... Figure 1 As shown, the system 100 may further include an input device 110a and / or a signal generation unit 110b, and the control device 120 may be communicatively connected to the input device 110a or the signal generation unit 110b. In some embodiments, the control device 120 may be communicatively connected to a drive device (e.g., a motor) not shown, and send a drive signal to the drive device based on a pre-input signal from the input device 110a or a signal randomly generated signal from the signal generation unit 110b, thereby causing the drive device to control the actuator arm 131 based on the drive signal, so that the end of the actuator arm 131 moves to the target pose corresponding to the drive signal.

[0033] In some embodiments, the actuator 130 may include a continuous deformable arm. A continuous deformable arm is, for example, as shown below. Figure 3 The actuator arm 300 is shown. In some embodiments, the actuator arm 130 may include a multi-degree-of-freedom actuator arm composed of multiple joints. For example, an actuator arm capable of 4 to 7 degrees of freedom of movement. For instance, an actuator arm capable of 6 degrees of freedom of movement.

[0034] In some embodiments, the image acquisition device 150 can be used to acquire positioning images. The positioning images may include part or all of the image of the actuator arm 130. In some embodiments, the image acquisition device 150 may include, but is not limited to, a dual-lens image acquisition device or a single-lens image acquisition device, such as a binocular or monocular camera. In some embodiments, the image acquisition device 150 can be used to acquire images of the actuator arm end cap 131. Multiple markings may be provided on the actuator arm end cap 131, including marking patterns and pattern corner dots. For example, a positioning label 132 may be provided on the actuator arm end cap 131 (the positioning label 132 may be, for example, a positioning tag 132). Figure 4 The label 400 shown. The positioning label 132 may include multiple identifiers, including multiple pose identifiers for identifying pose and at least one composite identifier for identifying pose and angle (detailed below).

[0035] like Figure 1 As shown, if the end effector arm 131 is within the field of view 151 of the image acquisition device 150, the acquired positioning image may include an image of the end effector arm 131. Depending on the application scenario, the image acquisition device 150 may be an industrial camera, underwater camera, miniature electronic camera, endoscope camera, etc. In some embodiments, the image acquisition device 150 may be fixed in position or have a variable position; for example, an industrial camera fixed at a monitoring position or an endoscope camera with adjustable position or orientation. In some embodiments, the image acquisition device 150 may perform at least one of visible light imaging, infrared imaging, CT (Computed Tomography) imaging, and acoustic imaging. Depending on the type of image acquired, those skilled in the art can select different image acquisition devices as the image acquisition device 150.

[0036] In some embodiments, the control device 120 may receive a positioning image from the image acquisition device 150 and process the positioning image. For example, the control device 120 may identify multiple markers located on the end effector 131 of the actuator arm in the positioning image and determine the relative coordinate system of the end effector 131 of the actuator arm to a reference coordinate system (e.g., world coordinate system, such as...). Figure 3 The pose shown is used as the actual pose.

[0037] In some embodiments, system 100 can determine the performance of the actuator arm 130 based on the actual and target poses of the actuator arm end effector 131. Those skilled in the art will understand that system 100 can be applied to specialized or general-purpose robotic systems in multiple fields (e.g., logistics, industrial manufacturing, medical, etc.). As an example, system 100 can be applied to robotic systems such as surgical robots, where the end effector 140 disposed at the distal end of the actuator arm end effector 131 can be, for example, a surgical actuator.

[0038] Figure 2 A schematic diagram of a segment 200 of an actuator arm according to some embodiments of the present disclosure is shown. The actuator arm (e.g., actuator arm 130) may include at least one deformable segment 200. Figure 2 As shown, the deformable segment 200 includes a fixed disk 210 and multiple structural bones 220. A first end of each structural bone 220 is fixedly connected to the fixed disk 210, and a second end is used to connect to a driving device (not shown). In some embodiments, the fixed disk 210 may be, but is not limited to, a ring-shaped structure, a disc-shaped structure, etc., and its cross-section may be circular, rectangular, polygonal, or various other shapes.

[0039] The driving device causes the segment 200 to deform by driving the structural bone 220. For example, the driving device causes the segment 200 to be in a position such as... Figure 2 The bending state is shown. In some embodiments, the second end of the multiple structural bones 220 passes through the base plate 230 and is connected to the driving device. In some embodiments, similar to the fixed plate 210, the base plate 230 can be, but is not limited to, a ring-shaped structure, a disc-shaped structure, etc., and the cross-section can be various shapes such as circular, rectangular, polygonal, etc. The driving device can include a linear motion mechanism, a driving segment, or a combination of both. The linear motion mechanism can be connected to the structural bone 220 to push or pull the structural bone 220, thereby driving the segment 200 to bend. The driving segment can include a fixed plate and multiple structural bones, wherein one end of the multiple structural bones is fixedly connected to the fixed plate. The other end of the multiple structural bones of the driving segment is connected to or integrally formed with the multiple structural bones 220 to drive the bending of the segment 200 by bending the driving segment.

[0040] In some embodiments, a spacer disk 240 is further included between the fixed disk 210 and the base disk 230, through which multiple structural bones 220 pass. Similarly, the drive segment may also include a spacer disk.

[0041] Figure 3 A schematic diagram of the structure of an actuator 300 according to some embodiments of the present disclosure is shown. For example... Figure 3 As shown, the actuator arm 300 is a deformable actuator arm, which may include an actuator arm end 310 and an actuator arm body 320. The actuator arm body 320 may include one or more components, such as a first component 3201 and a second component 3202. In some embodiments, the structures of the first component 3201 and the second component 3202 may be similar to those of other components. Figure 2 The shown component 200 is similar. In some implementations, such as... Figure 3As shown, the actuator arm body 320 also includes a first straight rod segment 3203 located between the first segment 3201 and the second segment 3202. The first end of the first straight rod segment 3203 is connected to the base plate of the second segment 3202, and the second end is connected to the fixed plate of the first segment 3201. In some embodiments, such as... Figure 3 As shown, the actuator arm body 320 also includes a second straight rod segment 3204, the first end of which is connected to the base plate of the first component 3201.

[0042] In some embodiments, the actuator 130 may be as follows: Figure 3 The actuator arm 300 is shown. (As shown in the image) Figure 3 As shown, each segment (first segment 3201 and second segment 3202) may include a base plate, a fixed plate, and multiple structural bones penetrating the base plate and the fixed plate. The multiple structural bones may be fixedly connected to the fixed plate and slidably connected to the base plate. The continuous deformable arm and its included segments can be described by a kinematic model (detailed below).

[0043] In some embodiments, the structure of each component may be specifically as follows: Figure 2 The shown component is 200. (As shown in the image) Figure 2 As shown, the base disk coordinate system The base disk is attached to the t-th (t=1,2,3…) section of the continuum, with its origin located at the center of the base disk, and the XY plane coinciding with the plane of the base disk. Pointing from the center of the base plate to the first structural bone (the first structural bone can be understood as any one of multiple structural bones chosen as a reference). Bending plane coordinate system 1 Its origin coincides with the origin of the base disk coordinate system, and the XY plane coincides with the bending plane. and Coincident. Fixed disk coordinate system. The origin of the fixed disk is located at the center of the fixed disk, and the XY plane coincides with the plane of the fixed disk. Pointing from the center of the fixed plate to the first structural bone. Curved plane coordinate system 2 Its origin is located at the center of the fixed disk, and the XY plane coincides with the bending plane. and coincide.

[0044] like Figure 2 The single segment 200 shown can be represented by a kinematic model. The position of the end of segment t (fixed disk coordinate system {te}) relative to the base disk coordinate system {tb}. tb P te ,attitude tb R te It can be determined based on the following formulas (1) and (2):

[0045]

[0046] tb R te = tb R t1 t1 R t2 t2 R te (2)

[0047] Among them, L t For the t-th segment, construct a virtual structural bone (e.g., Figure 2 The length of the virtual structural bone 221 shown in the figure, θ t In the t-th section, about or Rotate to Required rotation angle tb R t1 Let {t1} be the orientation of the bending plane coordinate system of segment t relative to the base disk coordinate system {tb}. t1 R t2 Let t be the orientation of the bending plane coordinate system 2{t2} of the t-th segment relative to the bending plane coordinate system 1{t1}. t2 R te Let {te} be the orientation of the fixed disk coordinate system {t2} of the t-th segment relative to the curved plane coordinate system 2{t2}.

[0048] tb R t1 , t1 R t2 and t2 R te It can be based on the following formulas (3), (4) and (5):

[0049]

[0050]

[0051]

[0052] Where, δ t For the t-th segment, the bending plane and The included angle.

[0053] like Figure 2 The joint parameter Ψ of the single segment 200 shown t It can be determined based on the following formula (6):

[0054] ψ t =[θ t ,δt ] T (6)

[0055] In some embodiments, the driving amount of multiple bone structures has a known mapping relationship with joint parameters. Based on the target joint parameters of the segment and the mapping relationship, the driving amount of the multiple bone structures can be determined. The driving amount of the multiple bone structures can be understood as moving a single segment from its initial state (e.g., θ) t =0) The length of the structural bone subjected to push or tension when bent to the target bending angle. In some embodiments, the mapping relationship between the driving amount of multiple structural bones and joint parameters can be determined based on the following formula (7):

[0056] q i ≡-r ti θ t cos(δ t +β ti (7)

[0057] Where, r ti Let β be the distance from the i-th structural bone in the t-th segment to the virtual structural bone. ti Let q be the angle between the i-th structural bone and the first structural bone in the t-th segment. i Let be the driving amount of the i-th structural bone. The driving signal of the driving device can be determined based on the driving amount of the i-th structural bone.

[0058] In some embodiments, the entire deformable arm can be described by a kinematic model. For example... Figure 3 As shown, transformations can be performed between multiple coordinate systems located at multiple positions of the deformable arm. For example, the actuator of the continuum deformable arm can be determined in a reference coordinate system (e.g., the world coordinate system {w}) based on the following formula (8):

[0059] w T tip = w T 1b 1b T 1e 1e T 2b 2b T 2e 2e T tip (8)

[0060] in, W T tip The homogeneous transformation matrix of the actuator of the continuum deformable arm relative to the reference coordinate system; W T 1b The homogeneous transformation matrix of the base disk of the first continuum segment relative to the reference coordinate system is represented. 1b T 1eRepresents the homogeneous transformation matrix of the fixed disk of the first continuous segment relative to the base disk of the first continuous segment; 1e T 2b Represents the homogeneous transformation matrix of the base disk of the second continuous segment relative to the fixed disk of the first continuous segment; 2b T 2e Represents the homogeneous transformation matrix of the fixed disk of the second continuum segment relative to the base disk of the second continuum segment; 2e T tip This represents the homogeneous transformation matrix of the actuator of the continuum deformable arm relative to the fixed disk of the second continuum segment. In some embodiments, the actuator is fixedly mounted on the fixed disk, therefore... 2e T tip It is known or predetermined.

[0061] It should be understood that deformable arms have different joint parameters in different operating states. For example, Figure 3 The actuator 300 shown includes at least four operating states. The four operating states of the actuator 300 are described below:

[0062] First working state: Only the second component 3202 participates in the actuator's pose control (e.g., only the second component 3202 enters the workspace), and the joint parameters of the actuator arm 300 can be determined based on the following formula (9):

[0063]

[0064] Where, ψ c1 To execute the joint parameters of arm 300 in the first working state, For the rotation angle of the actuator arm 300 around the axis, L2, θ2, δ2 and such Figure 2 In the structure shown in section 200, L t θ t and δ t They have the same physical meaning.

[0065] Second working state: The second component 3202 and the first linear segment 3203 participate in the pose control of the actuator (for example, the second component 3202 is fully in the workspace, and the first linear segment 3203 is partially in the workspace). At this time, the joint parameters of the actuator arm 300 can be determined based on the following formula (10):

[0066]

[0067] Where, ψ c2 To execute the joint parameters of arm 300 in the second working state, L r This is the feed rate for the first straight segment 3203.

[0068] Third working state: Second segment 3202, first straight segment 3203 and first segment 3201 participate in the position control of the actuator (for example, the second segment 3202 is fully in the workspace, the first straight segment 3203 is fully in the workspace, and the first segment 3201 is partially in the workspace). At this time, the joint parameters of the actuator arm 300 can be determined based on the following formula (11):

[0069]

[0070] Where, ψ c3 For the joint parameters of arm 300 in the third working state, L1, θ1, and δ1 are as follows: Figure 2 In the structure shown in section 200, L t θ t and δ t They have the same physical meaning.

[0071] Fourth working state: Second segment 3202, first linear segment 3203, first segment 3201 and second linear segment 3204 participate in the pose control of the actuator (for example, the second segment 3202 is fully in the workspace, the first linear segment 3203 is fully in the workspace, the first segment 3201 is fully in the workspace, and the second linear segment 3204 is partially in the workspace). At this time, the joint parameters of the actuator arm 300 can be determined based on the following formula (12):

[0072]

[0073] Where, ψ c4 To execute the joint parameters of arm 300 in the fourth working state, L s This is the feed rate for the second straight segment 3204.

[0074] In some embodiments, the execution arm (e.g., Figure 1 The actuator end 131 shown Figure 3 The actuator arm end 310 shown has multiple markings distributed thereon. In some embodiments, the multiple markings are disposed on the outer surface of the columnar portion of the actuator arm 130. For example, the multiple markings are distributed circumferentially on the actuator arm end 131, for example, disposed circumferentially on the outer surface of the columnar portion of the actuator arm end 131. In some embodiments, a positioning label 132 including multiple markings is disposed on the outer surface of the columnar portion of the actuator arm end 131. The multiple markings may include multiple pose markings for marking pose and at least one composite marking for marking pose and angle (e.g., about-axis angle or roll angle). In some embodiments, a positioning label (e.g., ...) is disposed on the outer surface of the columnar portion of the actuator arm end. Figure 4The label 400 shown may include multiple identifier patterns distributed circumferentially along the columnar portion on the positioning label, as well as multiple identifier pattern corner points within the identifier patterns. The multiple identifier patterns include multiple different composite identifier patterns and multiple pose identifier patterns, which may be identical. The composite identifier patterns and their corner points can be used to identify pose and angle, and the pose identifier patterns and their corner points can be used to identify pose. In some embodiments, the multiple different composite identifier patterns and the multiple pose identifier patterns are located in the same pattern distribution zone, such as... Figure 4 or Figure 5 As shown. In some embodiments, at least one composite marker pattern is included among N consecutive marker patterns, where 2 ≤ N ≤ 4, and the composite marker pattern among the N consecutive marker patterns is different from the pose marker pattern. For example, the multiple marker patterns can be evenly distributed on the outer surface of the columnar portion, and the multiple composite marker patterns can be evenly spaced among the multiple pose marker patterns, such as inserting one composite marker pattern every three pose marker patterns. Figure 4 As shown.

[0075] In some embodiments, the identification pattern may be disposed on a label on the end of the actuator arm, or printed on the end of the actuator arm, or may be a pattern formed by the physical structure of the end of the actuator arm itself, for example, it may include recesses or protrusions and combinations thereof. In some embodiments, the identification pattern may include a pattern formed in terms of brightness, grayscale, color, etc. In some embodiments, the identification pattern may include a pattern that actively (e.g., self-illuminating) or passively (e.g., reflecting light) provides information that can be detected by an image acquisition device. Those skilled in the art will understand that in some embodiments, the pose of the identification or the pose of the identification pattern may be represented by the pose of the corner coordinate system of the identification pattern. In some embodiments, the identification pattern is disposed on an area on the end of the actuator arm suitable for image acquisition by an image acquisition device, for example, an area that can be covered by the field of view of the image acquisition device during operation or an area that is not easily disturbed or obstructed during operation.

[0076] Figure 4 A schematic diagram of a label 400 including multiple identifiers according to some embodiments is shown. Figure 5 A schematic diagram is shown of a label 500 disposed on the periphery of the end of the actuator arm and forming a cylindrical shape. It can be understood that, for simplicity, label 400 may include the same marking pattern as label 500.

[0077] See Figure 4 Multiple identifiers include multiple pose identifier patterns 410 and multiple pose identifier pattern corner points P therein. 410 And composite logo pattern 420 and the corner point R of the composite logo pattern therein. 420 In some embodiments, such as Figure 4 As shown, multiple pose marker patterns 410 and composite marker patterns 420 are arranged in the same pattern distribution zone. In this disclosure, the corner points of the pose marker patterns are represented by the symbol "〇", and the corner points of the composite marker patterns are represented by the symbol "△". In some embodiments, the pose marker pattern 410 or the corner point P can be identified. 410 Determine the pose identifier by recognizing the composite identifier pattern 420 or the corner point R of the composite identifier pattern. 420 Determine the composite identifier.

[0078] See Figure 5 In the circumferential setting state, label 400 becomes label 500 with a spatially constructed cylindrical shape. In some embodiments, the axial angle or roll angle of each identifier can be represented by the axial angle of the identifier pattern or the corner point of the identifier pattern, wherein the identifier pattern includes a pose identifier pattern 510 and a composite identifier pattern 520. The axial angle of each identifier pattern or the corner point of the identifier pattern is known or predetermined. In some embodiments, the axial angle identified by each identifier can be determined based on the distribution of multiple identifiers (identifier patterns or corner points of identifier patterns). In some embodiments, multiple identifiers can be evenly distributed (e.g., the corner points of the identifier patterns in label 400 are evenly spaced, and the corner points of the identifier patterns in label 500 are evenly distributed angularly). In some embodiments, based on the distribution of multiple identifiers, each identifier can be used to identify a specific axial angle, and each identifier has a one-to-one correspondence with the identified axial angle. In this disclosure, the axial angle or roll angle refers to the angle around the Z-axis (e.g., the Z-axis of the end coordinate system of the actuator arm or the identifier coordinate system). In some embodiments, the Z-axis can be the tangential direction along the end of the actuator arm.

[0079] like Figure 5 As shown, multiple identification patterns in label 500 are evenly distributed circumferentially along the cylindrical structure, and the corner points of multiple identification patterns are evenly distributed on the cross-sectional circle 530. Therefore, the distribution angle (e.g., angle α0) of any adjacent corner points of the identification patterns is equal. Let the corner point P of the identification pattern pointed to by the X-axis be... 501 P 501 As a reference corner point for marking the 0° angle around the axis (corner point P of the marking pattern) 501 (Using the logo pattern as a reference pattern), then any corner point of the logo pattern and corner point P of the logo pattern can be used as a reference. 501 The positional relationship determines the angle around the axis of the corner mark of the logo pattern.

[0080] In some embodiments, the corner points of the identification pattern are located in a set coordinate system (e.g., Figure 5 The coordinate system shown is {wm0}≡[X] wm0 Y wm0 Z wm0 ] TThe angle about the axis indicated in the figure can be determined based on the following formula (13):

[0081] α m =α0(m-1) (13)

[0082] Where, α m To select the corner point of the logo pattern (e.g., corner point P of the logo pattern). 501 As the first corner point of the identification pattern, the angle around the axis of the m-th corner point of the identification pattern in the clockwise direction of the cross-sectional circle 530.

[0083] In some embodiments, the multiple pose marker patterns can be the same pattern or different patterns. In some embodiments, the multiple composite marker patterns are different patterns, each composite marker pattern can be used to identify a specific angle around an axis, and each composite marker pattern has a one-to-one correspondence with the identified angle around an axis.

[0084] Figure 6 A schematic diagram illustrating an implementation scenario 600 according to some embodiments of the present disclosure is shown. Figure 6 As shown, the actuator arm 640 includes an end effector 630 and a distal actuator 660. Multiple markers (e.g., pose marker pattern 610 and composite marker pattern 620) can be circumferentially disposed on the end effector 630. For example, as... Figure 4 The label 400 shown is circumferentially disposed on the end of the actuator arm 630. Multiple identification pattern corner points are distributed on the cross-sectional circle 631 of the end of the actuator arm 630. In some embodiments, an identification coordinate system {wm0}≡[X] is established based on the identified identifications. wm0 Y wm0 Z wm0 ] T The origin of the coordinate system {wm0} is the center of the cross-section circle 631, and the X-axis points from the origin to one of the corner points of the marker pattern (for example, the corner point P corresponding to one of the identified pose markers). 601 The Z-axis is parallel to the axial direction of the end of the actuator arm 630, and the Y-axis is perpendicular to the XZ plane.

[0085] In some embodiments, the end-effector coordinate system {wm}≡[X] is established based on multiple composite identifiers. wm Y wm Z wm ] T The origin of the coordinate system {wm} at the end of the actuator arm is the center of the cross-sectional circle 631, and the X-axis points to the corner point R of the composite marking pattern. 601 The Z-axis is parallel to or coincides with the axial direction of the end of the actuator arm 630, and the Y-axis is perpendicular to the XZ plane. In some embodiments, the distribution of multiple composite marking patterns can be based on, for example, the remaining composite marking patterns and the corner points R of the composite marking patterns.601 The positional relationship of the corresponding composite logo patterns determines the axial angle of the corner markers of the composite logo patterns contained within the composite logo pattern.

[0086] Some embodiments of this disclosure provide a method for detecting an actuator arm. Figure 7 A flowchart illustrating a detection method 700 of an actuator detection system (e.g., actuator detection system 100) according to some embodiments of the present disclosure is shown. Figure 7 As shown, some or all of the steps in method 700 can be executed by a control device (e.g., control device 120) of the execution arm detection system 100. Control device 120 can be configured on a computing device. Method 700 can be implemented by software, firmware, and / or hardware. In some embodiments, method 700 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 18 The processor 1820 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0087] See Figure 7 In step 701, a drive signal corresponding to the target pose of the end effector arm is determined for controlling the target pose. In some embodiments, the drive signal corresponding to the target pose of the end effector arm can be determined by the user via... Figure 1 The input device 110a shown is preset, and the control device 120 controls the end effector of the actuator arm to move to the corresponding target pose based on the preset drive signal. Those skilled in the art will understand that there is a mapping relationship between the drive signal and the pose of the actuator arm. Therefore, when the drive signal is preset, the mapping relationship between the drive signal and the pose of the actuator arm can be pre-stored in the memory of the system 100. The control device 120 can determine the target pose of the end effector of the actuator arm based on the preset drive signal using a lookup table method. In some embodiments, the drive signal corresponding to the target pose of the end effector of the actuator arm can be provided by… Figure 1 The signal generation unit 110b shown generates signals randomly, and the control device 120 controls the end effector of the actuator arm to move to the corresponding target pose based on the randomly generated drive signals.

[0088] In some embodiments, a target pose of the end effector of the actuator arm may be preset or randomly generated, and the control device 120 determines a drive signal for controlling the end effector of the actuator arm based on the target pose. In some embodiments, a target joint parameter set corresponding to the configuration of the actuator arm in the target pose may be preset or randomly generated, and the control device 120 determines a drive signal for controlling the end effector of the actuator arm based on the target joint parameter set. For example, an exemplary method for determining a drive signal for controlling the target pose of the end effector of the actuator arm includes, for instance, […]. Figure 8 or Figure 9 The method shown. In some embodiments, the target pose of the end effector is the target pose of the end effector in the reference coordinate system.

[0089] Continue reading Figure 7 In step 703, a positioning image is acquired. In some embodiments, the positioning image includes multiple markers on the end effector of the actuator arm. In some embodiments, the multiple markers include multiple pose markers for identifying pose and at least one composite marker for identifying pose and angle. In some embodiments, the markers can be obtained from, for example... Figure 1 The image acquisition device 150 shown receives a positioning image. For example, the control device 120 can receive a positioning image actively sent by the image acquisition device 150. Alternatively, the control device 120 can send an image request command to the image acquisition device 150, and the image acquisition device 150 responds to the image request command by sending a positioning image to the control device 120.

[0090] Continue reading Figure 7 In step 705, multiple markers located on the end of the actuator arm are identified in the positioning image. For example, an exemplary method for identifying multiple markers located on the end of the actuator arm may include, for instance... Figure 13 and Figure 15 The method is illustrated. In some embodiments, the control device 120 can identify some or all of the markers in the positioning image using an image processing algorithm. In some embodiments, the image processing algorithm may include a feature recognition algorithm that can extract or recognize features of the markers. For example, the image processing algorithm may include a corner detection algorithm for detecting corner points of the marker pattern. The corner detection algorithm may be one of, but is not limited to, corner detection based on grayscale images, corner detection based on binary images, and corner detection based on contour curves. For example, the image processing algorithm may be a color feature extraction algorithm for detecting color features in the marker pattern. As another example, the image processing algorithm may be a contour detection algorithm for detecting contour features of the marker pattern. In some embodiments, the control device can identify some or all of the markers in the positioning image using a recognition model.

[0091] Continue reading Figure 7 In step 707, the actual pose of the end effector of the actuator arm is determined based on at least one composite identifier and multiple pose identifiers. In some embodiments, the pose of the end effector coordinate system relative to the reference coordinate system can be determined based on the two-dimensional coordinates of the at least one composite identifier and multiple pose identifiers in the positioning image and the three-dimensional coordinates in the end effector coordinate system of the actuator arm, as the actual pose of the end effector of the actuator arm.

[0092] Continue reading Figure 7In step 709, the performance of the actuator is determined based on the target pose and the actual pose. In some embodiments, method 700 may further include generating a performance signal in response to the target pose and the actual pose reaching an error condition, the performance signal indicating that the actuator's performance is unqualified. For example, the control device may determine the target pose of the actuator's end effector in a reference coordinate system or the drive signal corresponding to the target pose, and determine the actual pose of the actuator's end effector based on the actuator's positioning image. When the target pose and the actual pose meet an error condition (e.g., greater than or equal to an error threshold), a performance signal indicating that the actuator's performance is unqualified is issued. In some embodiments, the performance signal may include multiple performance alarm signals. The multiple performance alarm signals may include factory performance alarm signals and operational performance alarm signals depending on the actuator's usage status. In some embodiments, the factory performance alarm signal indicates that the actuator has failed the factory performance test and its performance does not meet the factory standard. In some embodiments, the operational performance alarm signal indicates that an actuator that has been in service has failed the operational performance test and its performance does not meet the operational requirements, requiring maintenance or replacement.

[0093] In some embodiments, method 700 may further include determining a pose difference based on the target pose of the end effector of the actuator arm in the reference coordinate system and the actual pose of the end effector of the actuator arm in the reference coordinate system, and determining the performance of the actuator arm based on the pose difference and an error condition. The pose difference may include a position difference and an orientation difference. In some embodiments, the pose difference between the target pose and the actual pose of the end effector of the actuator arm can be determined by a single actuator arm detection, and the performance of the actuator arm can be determined based on whether the pose difference satisfies the error condition. In some embodiments, multiple pose differences between the target pose and the actual pose of the end effector of the actuator arm can be determined by multiple actuator arm detections, and the performance of the actuator arm can be determined based on whether these pose differences satisfy the error condition (e.g., the average or cumulative value of the multiple pose differences is greater than or equal to an error threshold). In some embodiments, method 700 may further include determining the actual pose of the end effector of the actuator arm at a predetermined detection cycle (e.g., performing steps 703-709 at a predetermined detection cycle) to determine or cumulatively determine the performance of the actuator arm in real time through multiple detection cycles.

[0094] In the k-th execution arm detection loop, the pose difference can be expressed as follows:

[0095]

[0096] in, Let $\frac{ ... Let P be the angle difference of the actuator arm during the k-th actuator arm detection cycle. t k R represents the target position of the actuator arm during the k-th actuator arm detection loop. tk The target pose of the execution arm during the k-th execution arm detection loop. R represents the actual position of the actuator arm during the k-th actuator arm detection loop. r k This represents the actual posture of the execution arm during the k-th execution arm detection loop. express and The corner between them.

[0097] Figure 8 This is a flowchart illustrating a method 800 for determining a drive signal for controlling the target pose of the end effector arm according to some embodiments of the present disclosure. Figure 8 As shown, some or all of the steps in method 800 can be executed by a control device (e.g., control device 120) of the execution arm detection system 100. Control device 120 can be configured on a computing device. Method 800 can be implemented by software, firmware, and / or hardware. In some embodiments, method 800 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 18 The processor 1820 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0098] See Figure 8 In step 801, the motion trajectory of the end effector of the actuator arm is preset, and the motion trajectory includes multiple target poses. In some embodiments, the motion trajectory of the end effector of the actuator arm can be determined by the user through... Figure 1 The input device 110a shown is preset, or can be accessed via... Figure 1 The signal generation unit 110b shown is pre-generated. The motion trajectory may include multiple target poses of the end effector of the actuator arm. In some embodiments, the motion trajectory may be a target path traversed by the end effector of the actuator arm, which may include multiple target path points arranged sequentially according to a predetermined motion cycle. These path points respectively indicate the target position and target pose of the end effector of the actuator arm in multiple motion cycles. The detection cycle of the actuator arm detection cycle may be the same as the motion cycle of the actuator arm motion cycle. Alternatively, the detection cycle of the actuator arm detection cycle may be an integer multiple of the motion cycle of the actuator arm motion cycle.

[0099] In step 803, multiple drive signals corresponding to multiple target poses are determined. In some embodiments, the target pose of the end effector of the actuator arm in the previous motion cycle in the motion trajectory is taken as the starting pose of the current motion cycle. Based on the difference between the target pose and the starting pose of the end effector of the actuator arm in the current motion cycle, the drive signal of at least one drive device controlling the motion of the actuator arm can be determined by using an inverse kinematic numerical iterative algorithm of the actuator arm kinematic model. It should be understood that the kinematic model can be a mathematical model representing the motion relationship between the joint space and the task space of the actuator arm. For example, the kinematic model can be established by methods such as the Denavit-Hartenberg (DH) parameter method and the exponential product representation method.

[0100] In some embodiments, method 800 may further include: sending a plurality of drive signals sequentially at a predetermined motion cycle; determining the actual pose of the end effector of the actuator arm at a predetermined detection cycle; and determining the performance of the actuator arm based on the plurality of target poses and the plurality of actual poses. For example, after determining the plurality of drive signals corresponding to the plurality of target poses in the motion trajectory, the control device may send the plurality of drive signals sequentially to the drive device at a predetermined motion cycle, determine the actual pose of the end effector of the actuator arm based on the positioning image of the actuator arm at a predetermined detection cycle, and determine the performance of the actuator arm based on the plurality of target poses and the plurality of actual poses.

[0101] In some embodiments, multiple actual poses arranged with a predetermined motion cycle constitute the actual trajectory of the end effector of the actuator arm. The error condition can be trajectory accuracy, which reflects the degree of deviation of the actual trajectory of the end effector of the actuator arm from the motion trajectory. For example, trajectory accuracy can be the maximum pose difference between multiple actual poses of the end effector of the actuator arm on the entire motion trajectory and multiple target poses, or it can be the average pose difference or cumulative pose difference between multiple actual poses of the end effector of the actuator arm on the entire motion trajectory and multiple target poses. In some embodiments, the control device can issue a performance signal indicating that the performance of the actuator arm is unqualified when the trajectory accuracy of the end effector of the actuator arm is greater than or equal to a predetermined threshold.

[0102] Figure 9 This is a flowchart illustrating a method 900 for determining a drive signal for controlling the target pose of the end effector arm according to other embodiments of this disclosure. Figure 9 As shown, some or all of the steps in method 900 can be executed by a control device (e.g., control device 120) of the execution arm detection system 100. Control device 120 can be configured on a computing device. Method 900 can be implemented by software, firmware, and / or hardware. In some embodiments, method 900 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 18The processor 1820 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0103] See Figure 9 In step 901, a target joint parameter set for the joints of the actuator arm is preset or randomly generated. In some embodiments, the pose of the actuator arm can be represented by a set of joint information (e.g., a matrix composed of this joint information) of the joints included in the actuator arm, and the joint parameter set of the actuator arm's joints has a mapping relationship with the pose of the actuator arm. In some embodiments, the target joint parameter set of the actuator arm's joints can be determined by the user through... Figure 1 The input device 110a shown is preset or via Figure 1 The signal generation unit 110b shown generates signals randomly, and the target joint parameter set determines the configuration of the actuator arm and the target pose of the actuator arm's end effector. In some embodiments, if the target joint parameter set of the actuator arm's joints is preset, the mapping relationship between the joint parameter set of the actuator arm's joints and the pose of the actuator arm can be stored in advance, and the target pose of the actuator arm's end effector can be determined by a lookup table method based on the preset target joint parameter set.

[0104] In step 903, a drive signal and the corresponding target pose are determined based on the target joint parameter set. In some embodiments, the control device may determine the drive signal of at least one drive device driving the actuator arm and the target pose of the actuator arm's end effector determined by the target joint parameter set based on the target joint parameter set of the actuator arm's joints, using the actuator arm's kinematic model. Step 903 can be implemented similarly to step 803 in method 800.

[0105] In some embodiments, method 900 may further include: determining the driving amount of multiple structural bones based on a target joint parameter set; and determining a driving signal based on the driving amount of the multiple structural bones. For example, based on the mapping relationship between the target joint parameters and the driving amount of the multiple structural bones, the driving amount of multiple joints included in the actuator arm in the current detection cycle can be determined, and then a driving signal of at least one driving device (e.g., a motor) can be determined based on the driving amount. In some embodiments, the actuator arm is taken as a continuous deformable arm as an example. A continuous deformable arm can be as follows: Figure 3 The actuator 300 shown can determine the driving amount of each structural bone of each segment based on formula (7), and then determine the driving signal of the driving device based on the driving amount.

[0106] Those skilled in the art will understand that each joint of the actuator arm includes its own joint parameter range, and the joint parameter range of each joint constitutes the joint space of the actuator arm. When randomly generating the target joint parameter set for the actuator arm's joints, it is possible that one or more randomly generated target joint parameters exceed the joint parameter range, resulting in the actuator arm's pose determined by the randomly generated target joint parameter set being outside the actuator arm's workspace. In some embodiments, method 1000 can converge the target joint parameter set of the actuator arm's joints within the joint parameter range.

[0107] Figure 10 This is a flowchart illustrating a method 1000 for updating a target joint parameter set of a joint of an actuator arm according to some embodiments of the present disclosure. Figure 10 As shown, some or all of the steps in method 1000 can be executed by a control device (e.g., control device 120) of the execution arm detection system 100. Control device 120 can be configured on a computing device. Method 1000 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1000 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 18 The processor 1820 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0108] See Figure 10 In step 1001, in response to at least one target joint parameter in the randomly generated set of target joint parameters exceeding the joint parameter range, at least one target joint parameter is updated to the limit value of the corresponding joint parameter range. In some embodiments, the target joint parameters in the randomly generated set of target joint parameters are compared with the joint parameter range. In response to a target joint parameter exceeding the joint parameter range, the out-of-range target joint parameter is updated to the limit value of the corresponding joint parameter range closest to the target joint parameter.

[0109] In step 1003, a drive signal and the corresponding target pose are determined based on the updated target joint parameter set. In some embodiments, after converging out-of-range target joint parameters to the limit values ​​of their joint parameter range, the drive signal of at least one drive device driving the actuator arm and the target pose of the actuator arm's end effector corresponding to the target joint parameter set are determined based on the updated target joint parameter set using the kinematic model of the actuator arm. Step 1003 can be implemented similarly to step 803 in method 800 and step 903 in method 900.

[0110] Those skilled in the art will understand that when the joints of the actuator are near their range limits, the actual pose of its end effector is more likely to deviate from the target pose. By updating the target joint parameters that exceed the joint parameter range to the limit values ​​of the corresponding joint parameter range, the target pose that exceeds the workspace of the actuator can be updated to a target pose where at least one target joint parameter is at its limit value. This allows for actuator performance detection at the actuator's limit position, improving detection efficiency. In some embodiments, method 1000 can be used to update the randomly generated drive signal or the target pose of the actuator's end effector. In some embodiments, other methods can also be used to update or filter the randomly generated signal (e.g., drive signal, target pose of the actuator's end effector, or target joint information set of the actuator's joints). For example, the randomly generated signal can be filtered to remove signals that exceed the range of key parameters, preventing the target pose of the actuator's end effector from being outside the actuator's limit position. In some embodiments, a signal corresponding to the target pose of the actuator's end effector can also be randomly generated within a preset range, thereby avoiding interference with actuator detection due to the actuator's inability to move to the target pose.

[0111] In some embodiments, method 700 may further include determining the two-dimensional coordinates of a plurality of markers in a positioning image. In some embodiments, the coordinates of the markers can be represented by the coordinates of the corner points of the marker pattern. For example, the two-dimensional coordinates of the markers in the positioning image and the three-dimensional coordinates in the end-effector coordinate system of the actuator arm can be represented by the coordinates of the corner points of the marker pattern. In some embodiments, determining the two-dimensional coordinates of a plurality of markers in the positioning image may include determining the two-dimensional coordinates of at least one composite marker and a plurality of pose markers in the positioning image. In some embodiments, method 700 may further include determining the three-dimensional coordinates of at least one composite marker and a plurality of pose markers in the end-effector coordinate system of the actuator arm based on at least one composite marker.

[0112] In some embodiments, method 700 may further include determining the pose of the end-effector coordinate system relative to the reference coordinate system based on the two-dimensional coordinates of at least one composite marker corner point and multiple pose marker corner points in the positioning image, their three-dimensional coordinates in the end-effector coordinate system of the actuator arm, and the transformation relationship between the camera coordinate system and the reference coordinate system. In some embodiments, the transformation relationship between the camera coordinate system and the reference coordinate system may be known. For example, the reference coordinate system may be the world coordinate system, and the transformation relationship between the camera coordinate system and the world coordinate system may be determined based on the camera's orientation. In other embodiments, the reference coordinate system may be the camera coordinate system itself, depending on actual needs. In some embodiments, based on camera imaging principles and projection models, the pose of the end-effector coordinate system relative to the camera coordinate system is determined based on the two-dimensional coordinates of at least one composite marker corner point and multiple pose marker corner points in the positioning image, and their three-dimensional coordinates in the end-effector coordinate system of the actuator arm. Based on the pose of the end-effector coordinate system relative to the camera coordinate system and the transformation relationship between the camera coordinate system and the reference coordinate system, the pose of the end-effector coordinate system relative to the reference coordinate system can be obtained.

[0113] In some embodiments, camera intrinsic parameters may also be considered. For example, camera intrinsic parameters may be as follows: Figure 2 The image acquisition device 150 shown has camera intrinsic parameters. These parameters can be known or obtained through calibration. In some embodiments, the camera coordinate system can be understood as a coordinate system established with the camera origin. For example, a coordinate system established with the camera's optical center as the origin or a coordinate system established with the camera's lens center as the origin. When the camera is a stereo camera, the origin of the camera coordinate system can be the center of the left lens, the center of the right lens, or any point on the line connecting the centers of the left and right lenses (e.g., the midpoint of that line).

[0114] In some embodiments, the pose of the end-effector coordinate system {wm} relative to the reference coordinate system (e.g., the world coordinate system) {w} can be determined based on the following formula (15):

[0115]

[0116] in, w R wm The orientation of the end effector coordinate system {wm} relative to the reference coordinate system. w P wm The position of the end effector's coordinate system relative to the reference coordinate system. w R lens The pose of the camera coordinate system relative to the reference coordinate system. w P lens The position of the camera coordinate system relative to the reference coordinate system. lens R wmThe orientation of the end effector's coordinate system relative to the camera coordinate system. lens P wm The position of the end effector coordinate system relative to the camera coordinate system.

[0117] Figure 11 A flowchart illustrating a method 1100 for determining the actual pose of the end effector of an actuator arm according to some embodiments of the present disclosure is shown. Figure 11 As shown, some or all of the steps in method 1100 can be executed by a control device (e.g., control device 120) of the execution arm detection system 100. Control device 120 can be configured on a computing device. Method 1100 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1100 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 18 The processor 1820 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0118] See Figure 11 In step 1101, the three-dimensional coordinates of at least one composite marker and multiple pose markers in the marker coordinate system are determined. In some embodiments, the three-dimensional coordinates of each marker pattern corner point in the marker coordinate system {wm0} can be determined based on the following formula (16):

[0119] C m =[r·cosα] m r·sinα m 0] T (16)

[0120] Among them, C m To use the selected corner point of the marker pattern as the first corner point of the marker pattern (e.g., pose marker pattern corner point P) 601 ), in the clockwise direction of cross-section circle 631, the three-dimensional coordinates of the m-th corner point of the marking pattern in the marking coordinate system, where r is the radius.

[0121] In some embodiments, the axial angle α of the m-th corner marker of the marker pattern is determined based on formula (13). m Then, the angle α around the axis determined by formula (13) m Formula (16) determines the three-dimensional coordinates C of the m-th corner point of the marker pattern in the marker coordinate system {wm0}. m .

[0122] See Figure 11In step 1103, based on at least one composite identifier, the roll angle of the identifier coordinate system relative to the end-effector coordinate system of the actuator arm is determined. In some embodiments, a first axial angle of the at least one composite identifier in the end-effector coordinate system of the actuator arm, and a second axial angle of the composite identifier in the identifier coordinate system, can be determined. Based on the first and second axial angles, the roll angle of the identifier coordinate system relative to the end-effector coordinate system of the actuator arm can be determined. In some embodiments, see... Figure 6 The roll angle Δα can refer to the rotation angle of the identifier coordinate system {wm0} relative to the end coordinate system {wm} of the actuator arm about the Z-axis. In some embodiments, the roll angle Δα can be determined based on the following formula (17):

[0123] Δα=α1-α2 (17)

[0124] Where α1 is the first angle around the axis, and α2 is the second angle around the axis. The first angle around the axis is the corner point of the composite logo pattern (e.g., the corner point R of the composite logo pattern). 602 The first angle around the axis is marked in the coordinate system of the end effector arm, and the second angle around the axis is the corner point of the composite mark pattern (e.g., corner point R of the composite mark pattern). 602 The angle around the axis is indicated in the coordinate system.

[0125] In some embodiments, the X-axis of the identifier coordinate system {wm0} points to the corner point of the composite identifier pattern (e.g., the corner point R of the composite identifier pattern). 602 Method 1100 may further include determining a first about-axis angle in which the composite identifier is identified in the end-effector coordinate system of the actuator arm as a roll angle of the identifier coordinate system relative to the end-effector coordinate system of the actuator arm. In some embodiments, the first about-axis angle may be determined based on the pattern included in the composite identifier.

[0126] See Figure 11 In step 1105, based on the roll angle of the marker coordinate system relative to the end effector coordinate system of the actuator arm and the three-dimensional coordinates of at least one composite marker and multiple pose markers in the marker coordinate system, the three-dimensional coordinates of at least one composite marker and multiple pose markers in the end effector coordinate system of the actuator arm are determined. It can be understood that, given the roll angle of the marker coordinate system relative to the end effector coordinate system of the actuator arm, the three-dimensional coordinates of multiple marker pattern corner points (e.g., corner points of composite marker patterns and corner points of pose marker patterns) in the marker coordinate system can be transformed into three-dimensional coordinates in the end effector coordinate system of the actuator arm through coordinate transformation.

[0127] See Figure 11In step 1107, based on the two-dimensional coordinates of at least one composite identifier and multiple pose identifiers in the positioning image and the three-dimensional coordinates in the end-effector coordinate system of the actuator arm, the pose of the end-effector coordinate system relative to the reference coordinate system is determined as the actual pose of the end-effector of the actuator arm. In some embodiments, step 1107 in method 1100 can be implemented similarly to determining the actual pose of the end-effector of the actuator arm in method 700.

[0128] Figure 12 A flowchart illustrating a method 1200 for determining the actual pose of the end effector of an actuator arm according to other embodiments of the present disclosure is shown. Method 1200 may be... Figure 11 Alternative embodiments of method 1100, such as... Figure 12 As shown, some or all of the steps in method 1200 can be executed by a control device (e.g., control device 120) of the execution arm detection system 100. Control device 120 can be configured on a computing device. Method 1200 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1200 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 18 The processor 1820 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0129] See Figure 12 In step 1201, the pose of the marker coordinate system relative to the reference coordinate system is determined based on the two-dimensional coordinates of at least one composite marker and multiple pose markers in the positioning image and their three-dimensional coordinates in the marker coordinate system. In some embodiments, the three-dimensional coordinates of at least one composite marker and multiple pose markers in the marker coordinate system can be implemented similarly to step 1101 in method 1100.

[0130] See Figure 12 In step 1203, based on at least one composite identifier, the roll angle of the identifier coordinate system relative to the end-effector coordinate system of the actuator arm is determined. In some embodiments, determining the roll angle of the identifier coordinate system relative to the end-effector coordinate system of the actuator arm can be implemented similarly to step 1103 in method 1100.

[0131] See Figure 12 In step 1205, based on the roll angle of the identifier coordinate system relative to the end coordinate system of the actuator arm and the pose of the identifier coordinate system relative to the reference coordinate system, the pose of the end coordinate system of the actuator arm relative to the reference coordinate system is determined as the actual pose of the end of the actuator arm.

[0132] For example, the pose of the end-effector coordinate system {wm} relative to the reference coordinate system (e.g., the world coordinate system) {w} can be determined based on the following formula (18):

[0133]

[0134] in, w R wm The attitude of the end effector's coordinate system relative to the reference coordinate system. w P wm The position of the end effector's coordinate system relative to the reference coordinate system. w R wm0 To indicate the attitude of the coordinate system relative to the reference coordinate system, w P wm0 To indicate the position of the coordinate system relative to the reference coordinate system, rot z (Δα) represents the roll angle Δα around the Z-axis of the end coordinate system of the actuator arm.

[0135] Figure 13 A flowchart illustrating a method 1300 for identifying an identifier according to some embodiments of the present disclosure is shown. Figure 13 As shown, some or all of the steps in method 1300 can be executed by a control device (e.g., control device 120) of the execution arm detection system 100. Control device 120 can be configured on a computing device. Method 1300 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1300 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 18 The processor 1820 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0136] See Figure 13 In step 1301, a plurality of candidate markers are determined from the positioning image. In some embodiments, the markers may include marker pattern corner points in a marker pattern. The coordinates or origin of the coordinate system of a candidate marker can be represented by a candidate marker pattern corner point. In some embodiments, a candidate marker pattern corner point may refer to a possible marker pattern corner point obtained after preliminary processing or preliminary identification of the positioning image.

[0137] In some embodiments, method 1300 may include determining a region of interest (ROI) in the localization image. For example, the ROI may be cropped from the localization image, and multiple candidate identifiers may be determined from the ROI. The ROI may be the entire localization image or a partial region. For example, the ROI of the current frame may be cropped based on a region within a certain range of multiple identifier pattern corner points determined in the previous frame image (e.g., the localization image of the previous image processing cycle). For localization images that are not the first frame, the ROI may be a region within a certain distance centered on a virtual point formed by the coordinates of multiple identifier pattern corner points from the previous image processing cycle. The certain distance range may be a fixed multiple of the average spacing distance of the identifier pattern corner points, such as twice. It should be understood that the predetermined multiple may also be a variable multiple of the average spacing distance of the multiple candidate identifier pattern corner points in the previous image processing cycle.

[0138] In some embodiments, method 1300 may include determining the corner likelihood (CL) value of each pixel in the localization image. In some embodiments, the corner likelihood value of a pixel may be a numerical value characterizing the probability that the pixel is a feature point (e.g., a corner). In some embodiments, the localization image may be preprocessed before calculating the corner likelihood value of each pixel, and then the corner likelihood value of each pixel in the preprocessed image may be determined. Image preprocessing may include, for example, at least one of image grayscale conversion, image denoising, and image enhancement. For example, image preprocessing may include: cropping a Region of Interest (ROI) from the localization image and converting the ROI to a corresponding grayscale image.

[0139] In some embodiments, determining the corner likelihood value of each pixel in the ROI may include, for example, performing a convolution operation on each pixel within the ROI to obtain the first and / or second derivatives of each pixel. The corner likelihood value of each pixel is then calculated using the first and / or second derivatives of each pixel within the ROI. For example, the corner likelihood value of each pixel can be determined based on the following formula (19):

[0140]

[0141] Where τ is a set constant, for example, set to 2; I x I 45 I y I n45 These are the first derivatives of the pixel in the four directions: 0, π / 4, π / 2, and -π / 4; I xy and I 45_45 These are the second derivatives of the pixel in the directions of 0, π / 2 and π / 4, -π / 4, respectively.

[0142] In some embodiments, method 1300 may include dividing the ROI into multiple sub-regions. For example, a non-maximum suppression method may be used to evenly segment a ROI into multiple sub-images. In some embodiments, the ROI may be evenly segmented into multiple sub-images of 5×5 pixels. The above embodiments are exemplary and not limiting. It should be understood that the location image or ROI may also be segmented into multiple sub-images of other sizes, such as multiple sub-images of 9×9 pixels.

[0143] In some embodiments, method 1300 may include determining the pixel with the largest corner likelihood value in each sub-region to form a pixel set. For example, the pixel with the largest CL value in each sub-image may be determined, and the pixel with the largest CL value in each sub-image may be compared with a first threshold to determine a set of pixels with a CL value greater than the first threshold. In some embodiments, the first threshold may be set to 0.06. It should be understood that the first threshold may also be set to other values.

[0144] See Figure 13 In step 1303, a first identifier is identified from a plurality of candidate identifiers. In some embodiments, the first identifier is identified based on an identifier pattern matching template. In some embodiments, the identifier pattern matching template includes at least one pose identifier pattern matching template and multiple composite identifier pattern matching templates with different patterns. In some embodiments, the composite identifier is identified based on multiple composite identifier pattern matching templates with different patterns. For example, if the identifier patterns of pose identifiers are the same, the pose identifier pattern matching template can be matched with the candidate identifiers first. If the matching fails, multiple different composite identifier pattern matching templates can be matched with the candidate identifiers one by one until a match is successful.

[0145] In some embodiments, a marker pattern matching template is used to match the pattern at the corner of a candidate marker pattern to identify the first marker. For example, a candidate marker pattern corner that meets a preset pose pattern matching degree standard is determined as the first marker pattern corner. In some embodiments, the marker pattern matching template and the pattern in the vicinity of the marker pattern corner have the same or similar features. If the matching degree between the marker pattern matching template and the pattern in the vicinity of the candidate marker pattern corner reaches a preset pattern matching degree standard (e.g., the matching degree is higher than a threshold), it can be considered that the pattern in the vicinity of the candidate marker pattern corner has the same or similar features as the marker pattern matching template, and thus the current candidate marker pattern corner can be considered as the marker pattern corner.

[0146] In some embodiments, the pixel with the largest CL value in the pixel set is identified as a candidate identifier pattern corner point. For example, all pixels in the pixel set can be sorted in descending order of CL value, and the pixel with the largest CL value can be selected as the candidate identifier pattern corner point. In some embodiments, after the candidate identifier pattern corner point is determined, an identifier pattern matching template is used to match the pattern at the candidate identifier pattern corner point. If a preset pattern matching degree standard is met, the candidate identifier pattern corner point is determined as the first identified identifier pattern corner point.

[0147] In some embodiments, method 1300 may further include, in response to a matching failure, determining the pixel with the largest corner likelihood value among the remaining pixels in the pixel set as a candidate identifier pattern corner point. For example, if the candidate identifier pattern corner point does not meet a preset matching degree standard, then the pixel with the second largest CL value (the pixel with the second largest CL value) is selected as the candidate identifier pattern corner point, and the identifier pattern matching template is used to match the pattern at the candidate identifier pattern corner point, and so on, until the first identifier pattern corner point is identified.

[0148] In some embodiments, the logo pattern can be a black and white checkerboard pattern, therefore the logo pattern matching template can be the same checkerboard pattern, utilizing the grayscale distribution G of the logo pattern matching template. M The pixel neighborhood grayscale distribution G of the pixel corresponding to the corner point of the candidate identifier pattern image The correlation coefficient (CC) between pixels is used for matching. The grayscale distribution G of the pixel neighborhood is also considered. image This refers to the grayscale distribution of pixels within a certain range (e.g., 10×10 pixels) centered on the given pixel. The correlation coefficient can be determined based on the following formula (20):

[0149]

[0150] Where Var() is the variance function and Cov() is the covariance function. In some embodiments, when the correlation coefficient is less than 0.8, the gray-level distribution in the pixel neighborhood has a low correlation with the identifier pattern matching template. In this case, the candidate identifier pattern corner with the highest likelihood value is determined not to be an identifier pattern corner; otherwise, the candidate identifier pattern corner with the highest likelihood value is considered to be an identifier pattern corner.

[0151] In some embodiments, method 1300 may further include determining the edge direction of the corner points of the candidate identifier pattern. For example, as Figure 14 As shown, the corner point of the candidate pose identifier pattern is corner point P in pose identifier pattern 1400. 1401 Then the corner point P 1401 The edge direction can refer to the direction of the corner point P. 1401The direction of the edge, such as Figure 14 The direction indicated by the dashed arrow.

[0152] In some embodiments, the edge direction can be determined by the first-order derivative (I0) of each pixel in the X and Y directions of the planar coordinate system with respect to a certain neighborhood (e.g., 10×10 pixels) centered on the corner point of the candidate identifier pattern. x and I y The edge direction can be determined based on the following formula (21):

[0153] I angle =arctan(I y / I x ),

[0154] Among them, the first derivative (I) x and I y This can be obtained by performing a convolution operation on each pixel within a certain neighborhood range. In some embodiments, this is achieved by performing a convolution operation on the edge direction I of each pixel within the neighborhood range. angle and the corresponding weight I weight Clustering calculations are performed to obtain the edge direction of the pixel, and weight I is selected. weight The class with the largest proportion corresponds to I angle As the edge direction. It should be noted that if multiple edge directions exist, then weight I is selected. weight The I corresponding to the largest proportion of multiple classes angle As the edge direction.

[0155] In some embodiments, the clustering calculation method can be any one of the following: K-means, BIRCH (Balanced Iterative Reducing and Clustering using Hierarchies), DBSCAN (Density-Based Spatial Clustering of Applications with Noise), or GMM (Gaussian Mixed Model).

[0156] In some embodiments, method 1300 may include rotating a marker pattern matching template based on its edge direction. Rotating the marker pattern matching template based on its edge direction can align the template with the image at the corner point of a candidate marker pattern. The edge direction of the corner point of the candidate marker pattern can be used to determine the orientation of the image at that corner point in the positioning image. In some embodiments, rotating the marker pattern matching template based on its edge direction can adjust it to have the same or nearly the same orientation as the image at the corner point of the candidate marker pattern to facilitate image matching.

[0157] See Figure 13 In step 1305, starting with the first identifier, other identifiers are searched. In some embodiments, in response to the identification of a composite identifier, other identifiers are identified based on a pose identifier pattern matching template. In some embodiments, other identifiers include pose identifiers or composite identifiers.

[0158] Figure 15 A flowchart illustrating a method 1500 for searching for identifiers according to some embodiments of the present disclosure is shown. Figure 15 As shown, some or all of the steps in method 1500 can be executed by a control device (e.g., control device 120) of the execution arm detection system 100. Control device 120 can be configured on a computing device. Method 1500 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1500 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 18 The processor 1820 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0159] See Figure 15 In step 1501, a second identifier is determined starting from the first identifier. In some embodiments, the corner point of the first identifier pattern is used as the starting point, and the corner point of the second identifier pattern is searched in a set search direction. In some embodiments, the set search direction may include at least one of the following directions: directly in front of the corner point of the first identifier pattern (corresponding to a 0° angle direction), directly behind it (corresponding to a 120° angle direction), directly above it (90° angle direction), directly below it (-90° angle direction), and diagonally (e.g., ±45° angle direction).

[0160] In some embodiments, the number of search directions is set to n, for example, searching in 8 directions, with each search direction v sn It can be determined based on the following formula (22):

[0161] v sn =[cos(n·π / 4)sin(n·π / 4)], (n=1,2,…,8) (22)

[0162] In some embodiments, the search direction set in the current step can be determined based on the deviation angle between adjacent corner points of multiple marker patterns determined in the previous frame. For example, the predetermined search direction can be determined based on the following formula (23):

[0163]

[0164] Among them, (x j ,y j ) represents the two-dimensional coordinates of the corner points of multiple marker patterns determined in the previous frame (or the previous image processing cycle); n last The number of corner points of the multiple marker patterns determined in the previous frame; v s1 The first set search direction; v s2 This is the second search direction set.

[0165] In some embodiments, such as Figure 16 As shown, the first identification pattern corner point P 1601 Using the coordinates of the location as the starting point, search for the corner point P of the second marker pattern in the set search direction. 1602 The coordinates of the location. For example, the corner point P of the first identifier pattern. 1601 Using the coordinates as the starting point for the search, the search box (e.g., ...) Figure 16 (The dashed box in the image) moves in the set search direction V with a certain search step size. 1601 Search for the corner points of the icon pattern.

[0166] In some embodiments, if there is at least one candidate identifier within the search box, the candidate identifier pattern corner point with the highest corner point likelihood value within the search box is preferentially selected as the second identifier pattern corner point P. 1602 With the search box limited to a suitable size, the first identifier pattern corner point P is used. 1601 The coordinates of the second marker P are used as the starting point for the search. 1602 During the search, the candidate icon with the highest corner likelihood value among the candidate icons appearing in the search box is more likely to be the corner of the icon pattern. Therefore, it can be considered that the candidate icon with the highest corner likelihood value in the search box is the corner of the second icon pattern, P. 1602To improve data processing speed, in other embodiments, to improve the accuracy of corner point recognition of the marker pattern, the candidate marker pattern corner point with the highest corner point likelihood value among the candidate markers appearing in the search box is selected for corner point recognition to determine whether the candidate marker pattern corner point with the highest corner point likelihood value is a marker pattern corner point. For example, a pose marker pattern matching template or a composite marker pattern matching template can be used to match the image within a certain range of the candidate marker pattern corner point with the highest corner point likelihood value. The candidate marker pattern corner point that meets the preset pattern matching degree standard can be considered as the searched second marker pattern corner point P. 1602 .

[0167] In some embodiments, continue reading Figure 16 The size of the search box can be gradually increased, thereby gradually increasing the search range. The search step size can change synchronously with the side length of the search box. In other embodiments, the size of the search box can also be a fixed size.

[0168] In some embodiments, the identification pattern can be a black and white graphic, and pattern matching can be performed based on the correlation coefficient in formula (20). If the correlation coefficient is greater than the threshold, the candidate identification pattern corner with the largest likelihood value is considered to be the identification pattern corner, and is denoted as the second identification pattern corner.

[0169] See Figure 15 In step 1503, a search direction is determined based on the first identifier and the second identifier. In some embodiments, the search direction includes a first search direction and a second search direction. The first search direction may be a direction starting from the coordinate position of the corner point of the first identifier pattern and moving away from the corner point of the second identifier pattern. The second search direction may be a direction starting from the coordinate position of the corner point of the second identifier pattern and moving away from the corner point of the first identifier pattern. For example, Figure 16 The search direction V shown 1602 .

[0170] In step 1505, starting with the first or second identifier, an identifier is searched in the search direction. In some embodiments, if the corner point of the first identifier pattern is used as the new starting point, the first search direction described above can be used as the search direction for the identifier pattern corner point. If the corner point of the second identifier pattern is used as the new starting point, the second search direction described above can be used as the search direction for the identifier pattern corner point. In some embodiments, a new identifier pattern corner point is searched (e.g., Figure 16 The third identifier pattern corner point P 1603 This can be performed similarly to step 1501. In some embodiments, the search step size can be the first identifier pattern corner point P. 1601 Second identification pattern corner point P 1602 The distance between them is L1.

[0171] In some embodiments, in response to a search distance greater than a search distance threshold, the pixel with the largest corner likelihood value among the remaining pixels in the pixel set is determined as a candidate identifier pattern corner point; and an identifier pattern matching template is matched with the identifier pattern at the candidate identifier pattern corner point location to identify a first identifier. In some embodiments, after determining the pixel with the largest corner likelihood value among the remaining pixels in the pixel set as a new candidate identifier pattern corner point, a new first identifier can be identified based on a method similar to step 1303. In some embodiments, a search distance greater than a search distance threshold can be understood as a search distance greater than a search distance threshold in some or all search directions. In some embodiments, the search distance threshold may include a set multiple of the distance between the (N-1)th pose identifier pattern corner point and the (N-2)th pose identifier pattern corner point, where N≥3. For example, the search distance threshold is twice the distance between the first two identifier pattern corner points. Thus, the maximum search distance for the third marker corner is twice the distance between the first and second marker corners. If no marker corner is found within this search distance in the search direction, the pixel with the highest corner likelihood value among the remaining pixels in the pixel set is identified as a new candidate pose marker corner, and a new first marker is identified. The current search process then stops. In some embodiments, similar to method 1300, a new first marker corner can be determined, and similar to method 1500, the remaining marker corners can be searched starting from the new marker corner.

[0172] In some embodiments, in response to the number of identified markers being greater than or equal to a marker number threshold, the pose of the end effector relative to a reference coordinate system can be determined based on the identified markers, and the search for markers can be stopped accordingly. For example, in response to the number of identified marker pattern corner points being greater than or equal to a marker number threshold, the search for marker pattern corner points can be stopped. For example, when four marker pattern corner points are identified, the search for marker pattern corner points can be stopped.

[0173] In some embodiments, in response to the number of identified identifiers being less than an identifier number threshold, the pixel with the highest corner likelihood value among the remaining pixels in the pixel set is determined as a candidate identifier pattern corner point; and an identifier pattern matching template is matched with the identifier pattern at the position of the candidate identifier pattern corner point to identify the first identifier. In some embodiments, if the total number of identified identifier pattern corner points is less than the identifier number threshold, the search based on the first identifier pattern in the above steps is considered to have failed. In some embodiments, if composite identifiers are not included among all identified identifiers, for example, if the identified identifier pattern corner points do not include composite identifier pattern corner points, the search based on the first identifier pattern in the above steps is considered to have failed. In some embodiments, in the case of search failure, the pixel with the highest corner likelihood value among the remaining pixels in the pixel set is determined as a new candidate identifier pattern corner point, and then a new first identifier can be identified based on a method similar to step 1303. In some embodiments, similar to method 1300, a new first identifier pattern corner point can be re-determined, and similar to method 1500, the remaining identifier pattern corner points can be searched starting from the new identifier pattern corner point.

[0174] In some embodiments, if the identified identifiers include composite identifiers, the type of the remaining identifiers found may be uncertain (it should be understood that identifier types include pose identifiers and composite identifiers). For example, if the first identifier is a composite identifier, it may be uncertain whether the second identifier is a pose identifier or a composite identifier.

[0175] In some embodiments, if the identified identifiers do not include composite identifiers, the type of the newly found identifier is determined. For example, if the first identifier is not a composite identifier, it is necessary to determine whether the second identifier is a pose identifier or a composite identifier. If neither the first nor the second identifier is a composite identifier, it is necessary to determine whether the third identifier is a pose identifier or a composite identifier, and so on.

[0176] In some embodiments, after the corner points of the marker pattern are searched or identified, sub-pixel positioning can be performed on the identified corner points of the marker pattern to improve the positional accuracy of the corner points of the marker pattern.

[0177] In some embodiments, the CL values ​​of pixels can be fitted based on a model to determine the coordinates of the corner points of the identifier pattern after subpixel localization. For example, the fitting function for the CL value of each pixel in the ROI can be a quadratic surface function, the extreme points of which are subpixel points. The fitting function can be determined based on the following formulas (24) and (25):

[0178] S(x,y)=ax 2 +by 2 +cx+dy+exy+f (24)

[0179]

[0180] Where S(x, y) is the fitting function for the CL values ​​of all pixels in each ROI, and a, b, c, d, e, and f are coefficients; x c The x-coordinate and y-coordinate of the pose identifier c The y-coordinate is the pose identifier.

[0181] In some embodiments of this disclosure, a computer device is also provided, including a memory and a processor. The memory may be used to store at least one instruction, and the processor is coupled to the memory for executing the at least one instruction to perform some or all of the steps in the method of this disclosure, such as... Figures 7-13 and Figure 15 Some or all of the steps in the method disclosed herein.

[0182] Figure 17 A schematic block diagram of a computer device 1700 according to some embodiments of the present disclosure is shown. See also Figure 17 The computer device 1700 may include a central processing unit (CPU) 1701, a system memory 1704 including random access memory (RAM) 1702 and read-only memory (ROM) 1703, and a system bus 1705 connecting the various components. The computer device 1700 may also include an input / output system and a mass storage device 1707 for storing an operating system 1713, application programs 1714, and other program modules 1715. The input / output devices include an input / output controller 1710, primarily composed of a display 1708 and input devices 1709.

[0183] Mass storage device 1707 is connected to central processing unit 1701 via a mass storage controller (not shown) connected to system bus 1705. Mass storage device 1707 or computer-readable media provides non-volatile storage for computer devices. Mass storage device 1707 may include computer-readable media (not shown) such as hard disk or compact disc read-only memory (CD-ROM) drives.

[0184] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, flash memory or other solid-state storage technologies, CD-ROM, or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The aforementioned system memories and mass storage devices can be collectively referred to as memory.

[0185] Computer device 1700 can be connected to network 1712 via network interface unit 1711 connected to system bus 1705.

[0186] The system memory 1704 or mass storage device 1707 is also used to store one or more instructions. The central processing unit 1701 implements all or part of the steps of the methods in some embodiments of this disclosure by executing the one or more instructions.

[0187] In some embodiments of this disclosure, a computer-readable storage medium is also provided, storing at least one instruction that is executed by a processor to cause a computer to perform some or all of the steps in the methods of some embodiments of this disclosure, such as... Figures 7-13 and Figure 15 Some or all of the steps in the disclosed method. Examples of computer-readable storage media include memory for computer programs (instructions), such as read-only memory (ROM), random access memory (RAM), compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage devices.

[0188] Figure 18 A schematic diagram of a robot system 1800 according to some embodiments of the present disclosure is shown. In some embodiments of the present disclosure, see [link to schematic diagram]. Figure 18The robot system 1800 may include a tool 1850, a drive unit 1860, an image acquisition device 1810, and a control device (e.g., a processor 1820). The tool 1850 may include an actuator arm 1840 and an end effector 1830 disposed at the distal end of the actuator arm 1840. Multiple markers may be formed or disposed on the end effector 1830, including multiple pose markers and at least one composite marker. An actuator may be disposed at the distal end of the end effector 1830. The drive unit 1860 may be used to control the pose of the actuator arm 1840 and its end effector 1830. The image acquisition device 1810 may be used to acquire positioning images of the actuator arm 1840. In some embodiments, see... Figure 18 The robot system 1800 may further include an input device 1870a and a signal generation unit 1870b. The input device 1870a can be used to receive the target pose, motion trajectory, or drive signal controlling the target pose of the end effector 1830 input by a user operation. The signal generation unit 1870b can be used to randomly generate the target pose, motion trajectory, or drive signal controlling the target pose of the end effector 1830. The processor 1820 is connected to the input device 1870a, the signal generation unit 1870b, the drive device 1860, and the image acquisition device 1810, and is used to execute some or all of the steps in the methods of some embodiments of this disclosure, such as... Figures 7-13 and Figure 15 Some or all of the steps in the method disclosed herein.

[0189] Note that the above are merely exemplary embodiments and technical principles of this disclosure. Those skilled in the art will understand that this disclosure is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this disclosure. Therefore, although this disclosure has been described in detail through the above embodiments, this disclosure is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this disclosure, the scope of which is determined by the scope of the appended claims.

Claims

1. An arm detection method of performing, characterized by, comprises: determining a driving signal for controlling a target pose of an end of the execution arm, the driving signal corresponding to the target pose; acquiring a positioning image of the execution arm; in the positioning image, identifying a plurality of markers located on the end of the execution arm, the plurality of markers comprising a plurality of pose markers for identifying poses and at least one composite marker for identifying a pose and an angle; based on the at least one composite marker and the plurality of pose markers, determining an actual pose of the end of the execution arm; and based on the target pose and the actual pose, determining a performance of the execution arm; wherein, based on the at least one composite marker and the plurality of pose markers, determining the actual pose of the end of the execution arm comprises: determining three-dimensional coordinates of the at least one composite marker and the plurality of pose markers in a marker coordinate system; based on the at least one composite marker, determining a roll angle of the marker coordinate system relative to an end coordinate system of the execution arm; based on the roll angle of the marker coordinate system relative to the end coordinate system of the execution arm and the three-dimensional coordinates of the at least one composite marker and the plurality of pose markers in the marker coordinate system, determining three-dimensional coordinates of the at least one composite marker and the plurality of pose markers in the end coordinate system of the execution arm; and based on the two-dimensional coordinates of the at least one composite marker and the plurality of pose markers in the positioning image and the three-dimensional coordinates of the at least one composite marker and the plurality of pose markers in the end coordinate system of the execution arm, determining a pose of the end coordinate system of the execution arm relative to a reference coordinate system as the actual pose. determining a driving signal for controlling a target pose of an end of the execution arm comprises:

2. The method of claim 1, wherein, pre-setting or randomly generating the driving signal. determining a driving signal for controlling a target pose of an end of the execution arm comprises:

3. The method of claim 1, wherein, pre-setting or randomly generating the driving signal. determining a driving signal for controlling a target pose of an end of the execution arm comprises: pre-setting or randomly generating the driving signal.

4. The method of claim 1, wherein, determining a driving signal for controlling a target pose of an end of the execution arm comprises: pre-setting a motion trajectory of the end of the execution arm, the motion trajectory comprising a plurality of the target poses; and determining a plurality of the driving signals corresponding to a plurality of the target poses.

5. The method of claim 4, wherein: determining an actual pose of the end of the execution arm comprises: determining a plurality of actual poses of the end of the execution arm corresponding to a plurality of the driving signals at a predetermined detection period, based on the target pose and the actual pose, determining a performance of the execution arm comprises: based on a plurality of the target poses and a plurality of the actual poses, determining the performance of the execution arm.

6. The method of claim 1, wherein, determining a driving signal for controlling a target pose of an end of the execution arm comprises: pre-setting or randomly generating a target joint parameter set of a joint of the execution arm; and based on the target joint parameter set, determining the driving signal and the corresponding target pose.

7. The method of claim 6, wherein: in response to at least one target joint parameter in the randomly generated target joint parameter set exceeding a joint parameter range, updating the at least one target joint parameter to a limit value of the corresponding joint parameter range; and based on the updated target joint parameter set, determining the driving signal and the corresponding target pose.

8. The method of claim 6, wherein, The execution arm comprises: at least one joint, the joint comprising a fixed disc and a plurality of structural bones, a first end of the plurality of structural bones being fixedly connected to the fixed disc, and a second end of the plurality of structural bones being configured to be connected to a driving device; The method further comprises: based on the target joint parameter set, determining a driving amount of the plurality of structural bones; and based on the driving amount of the plurality of structural bones, determining the driving signal.

9. The method of claim 2, wherein, Further comprising: based on the pre-set driving signal, determining the target pose of the end of the execution arm through a mapping relationship between the driving signal and the target pose.

10. The method of claim 6, wherein, Further comprising: based on the pre-set target joint parameter set, determining the target pose of the end of the execution arm through a mapping relationship between the target joint parameter set and the target pose.

11. The method of claim 1, wherein, Further comprising: determining a plurality of candidate markers from the positioning image; identifying a first marker in the plurality of markers from the plurality of candidate markers; and searching for other markers with the first marker as a starting point.

12. The method of claim 11, wherein, Further comprising: in response to identifying the composite marker, identifying other markers based on a pose marker pattern matching template.

13. The method of claim 11, wherein, The marker comprises a marker pattern and a marker pattern corner point in the marker pattern, and the method further comprises: determining a region of interest in the positioning image; dividing the region of interest into a plurality of sub-regions; determining a pixel with the maximum corner point likelihood value in each sub-region to form a pixel set; determining a pixel with the maximum corner point likelihood value in the plurality of candidate markers as a candidate marker pattern corner point; and matching the marker pattern matching template with the marker pattern at the position of the candidate marker pattern corner point to identify the first marker, wherein the marker pattern matching template comprises at least one pose marker pattern matching template and a plurality of pattern different composite marker pattern matching template.

14. The method of claim 13, wherein, Further comprising: in response to a failed match, updating a pixel with the maximum corner point likelihood value in the remaining pixels in the pixel set as a candidate marker pattern corner point; and matching the marker pattern matching template with the marker pattern at the position of the updated candidate marker pattern corner point to identify the first marker.

15. The method of claim 13, wherein, Further comprising: searching for a second marker with the first marker as a starting point; determining a search direction based on the first marker and the second marker; and taking the first marker or the second marker as a starting point to search for markers in the search direction. Further comprising:

16. The method of claim 15, wherein, in response to a search distance greater than a search distance threshold, determining a pixel with the maximum corner point likelihood value of the remaining pixels in the pixel set as a candidate marker pattern corner point; and matching the marker pattern matching template with the marker pattern at the position of the candidate marker pattern corner point to re-identify the first marker. Further comprising:

17. The method of claim 15, wherein, ​ In response to the identified number of markers being greater than or equal to the marker number threshold, determining the actual pose of the end of the execution arm based on the identified markers.

18. The method of claim 15, wherein, Further comprising: In response to the identified number of markers being less than the marker number threshold, determining a pixel in the set of pixels having a maximum corner likelihood value as a candidate marker corner; And Matching the marker pattern matching template with a marker pattern at the position of the candidate marker corner to re-identify the first marker.

19. The method of any one of claims 1, 11-18, wherein an outer surface of a columnar portion of the end of the execution arm is provided with a positioning tag comprising the plurality of markers, the plurality of markers comprising a plurality of different composite markers and a plurality of pose markers, the plurality of different composite markers and the plurality of pose markers being located in a same pattern distribution band.

20. The method of claim 19, wherein at least one composite marker is included in N consecutive markers of the plurality of markers, wherein the composite marker is different from a marker pattern of a pose marker, and 2 21. The method of claim 1, wherein, Further comprising: Determining an actual pose of the end of the execution arm at a predetermined detection period to determine a performance of the execution arm in real time or cumulatively through a plurality of detection cycles.

22. The method of claim 1, wherein, Determining the performance of the execution arm based on the target pose and the actual pose comprises: In response to the target pose and the actual pose reaching an error condition, generating a performance signal indicating that the performance of the execution arm is unqualified.

23. A computer device comprising: a memory for storing at least one instruction; and a processor coupled to the memory and configured to execute the at least one instruction to perform the execution arm detection method of any one of claims 1-22.

24. A computer-readable storage medium for storing at least one instruction, which when executed by a computer causes the computer to perform the execution arm detection method of any one of claims 1-22.

25. A robotic system comprising: an execution arm having an end provided with a plurality of markers, the plurality of markers comprising a plurality of pose markers and at least one composite marker; at least one driving device for driving the execution arm; an image acquisition device for acquiring a positioning image of the execution arm; and a control device configured to be connected with the at least one driving device and the image acquisition device, and perform the execution arm detection method of any one of claims 1-3, 6-22.

26. A robotic system comprising: an execution arm having an end provided with a plurality of markers, the plurality of markers comprising a plurality of pose markers and at least one composite marker; at least one driving device for driving the execution arm; an image acquisition device for acquiring a positioning image of the execution arm; and a control device configured to be connected with the at least one driving device and the image acquisition device, and perform the execution arm detection method of claim 4 or 5. Further comprising: an input device communicatively connected with the control device for receiving an input driving signal or a target pose; 27. The robotic system of claim 25, wherein, and / or ​ ​ The signal generation unit is in communication connection with the control device, and is configured to randomly generate a driving signal or a target pose.

28. The robotic system of claim 26, wherein, Further comprising: The input device is in communication connection with the control device, and is configured to receive an input driving signal, a target pose, or a motion trajectory of the end of the manipulator arm. And / or The signal generation unit is in communication connection with the control device, and is configured to randomly generate a driving signal, a target pose, or a motion trajectory of the end of the manipulator arm.

Images