Method for detecting an execution arm based on a plurality of pose identifiers and robotic system

Abstract

The present disclosure relates to the technical field of detection, and discloses an execution arm detection method. The execution arm detection method comprises: determining a driving signal for controlling a target pose of a terminal end of an execution arm, the driving signal corresponding to the target pose; acquiring a positioning image; in the positioning image, identifying a plurality of pose marks located on the terminal end of the execution arm, the plurality of pose marks comprising different pose mark patterns; determining an actual pose of the terminal end of the execution arm based on the plurality of pose marks; and determining a performance of the execution arm based on the target pose and the actual pose.

Description

Technical Field

[0001] This disclosure relates to the field of detection technology, and in particular to an actuator arm detection method and robot system based on multiple pose identifiers. Background Technology

[0002] Before the robot starts working, the performance of the actuator arm needs to be tested. Among them, the accuracy of the actuator arm's pose is an important standard for measuring the performance of the actuator arm.

[0003] The pose of an actuator arm includes the position and orientation of its end effector. Typically, the pose of the actuator arm can be measured using devices such as laser trackers. Based on the obtained pose, it can be determined whether the end effector has moved to the desired position and orientation, thereby determining the actuator arm's performance. However, devices such as laser trackers have technical drawbacks, including high cost, inability to measure pose in a single measurement, and the need for periodic calibration.

[0004] Therefore, there is a need to provide a simple and effective method for detecting the performance of the actuator arm in order to assess its quality. Summary of the Invention

[0005] In some embodiments, this disclosure provides an actuator arm detection method. The method may include: determining a drive signal for controlling a target pose of the actuator arm's end effector, the drive signal corresponding to the target pose; acquiring a positioning image; identifying multiple pose markers located on the end effector of the actuator arm in the positioning image, the multiple pose markers including different pose marker patterns; determining the actual pose of the actuator arm's end effector based on the multiple pose markers; and determining the performance of the actuator arm based on the target pose and the actual pose.

[0006] In some embodiments, this disclosure provides a computer device including: a memory for storing at least one instruction; and a processor coupled to the memory and configured to execute at least one instruction to perform a method according to any of some embodiments of this disclosure.

[0007] In some embodiments, this disclosure provides a computer-readable storage medium for storing at least one instruction, which, when executed by a computer, causes the computer to perform any of the methods described in some embodiments of this disclosure.

[0008] In some embodiments, this disclosure provides a robot system, including: an actuator arm, the end of which is provided with a plurality of pose markers, the plurality of pose markers including different pose marker patterns; at least one drive device for driving the actuator arm; an image acquisition device for acquiring positioning images of the actuator arm; and a control device configured to be connected to the at least one drive device and the image acquisition device to perform the method of any one of the embodiments of this disclosure. Attached Figure Description

[0009] Figure 1 A schematic diagram of an actuator detection system according to some embodiments of the present disclosure is shown;

[0010] Figure 2 A schematic diagram of the structure of an actuator arm according to some embodiments of the present disclosure is shown;

[0011] Figure 3 This diagram illustrates the structure of an actuator arm according to some embodiments of the present disclosure;

[0012] Figure 4 A schematic diagram showing a label including multiple pose identifiers according to some embodiments of the present disclosure;

[0013] Figure 5 A schematic diagram showing a label disposed on the periphery of the end of an actuator arm and formed into a cylindrical shape according to some embodiments of the present disclosure;

[0014] Figure 6 A flowchart illustrating a detection method for an actuator detection system according to some embodiments of the present disclosure is shown.

[0015] Figure 7 A flowchart illustrating a method for determining a drive signal for controlling the target pose of the end effector arm according to some embodiments of the present disclosure;

[0016] Figure 8 A flowchart illustrating a method for determining a drive signal for controlling the target pose of the end effector arm according to other embodiments of the present disclosure;

[0017] Figure 9 A flowchart illustrating a method for updating a target joint parameter set of a joint of an actuator arm according to some embodiments of the present disclosure;

[0018] Figure 10 A flowchart illustrating a method for determining the three-dimensional coordinates of a plurality of pose identifiers relative to the end effector coordinate system of an actuator arm, according to some embodiments of the present disclosure;

[0019] Figure 11 A flowchart illustrating a method for determining the three-dimensional coordinates of a plurality of pose identifiers relative to the end effector coordinate system of an actuator arm, according to some other embodiments of the present disclosure;

[0020] Figure 12 A flowchart illustrating a method for identifying pose identifiers according to some embodiments of the present disclosure is shown.

[0021] Figure 13 A schematic diagram showing pose identification patterns according to some embodiments of the present disclosure;

[0022] Figure 14A flowchart illustrating a method for searching pose identifiers according to some embodiments of the present disclosure;

[0023] Figure 15 A schematic diagram illustrating a search pose identifier according to some embodiments of the present disclosure;

[0024] Figure 16 A flowchart illustrating a method for searching a second pose identifier according to some embodiments of the present disclosure is shown;

[0025] Figure 17 A flowchart illustrating a method for searching pose identifiers according to some embodiments of the present disclosure;

[0026] Figure 18 A schematic block diagram of a computer device according to some embodiments of the present disclosure is shown;

[0027] Figure 19 A schematic diagram of a robot system according to some embodiments of the present disclosure is shown. Detailed Implementation

[0028] To make the technical problems solved by this disclosure, the technical solutions adopted, and the technical effects achieved clearer, the technical solutions of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that the described embodiments should be considered exemplary in all respects and not restrictive, and are merely exemplary embodiments of this disclosure, not all embodiments.

[0029] In this disclosure, the term "position" refers to the location of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom can be described using variations in Cartesian X, Y, and Z coordinates, such as three translational degrees of freedom along the Cartesian X, Y, and Z axes, respectively). In this disclosure, the term "attitude" refers to the rotational setting of an object or part of an object (e.g., three rotational degrees of freedom, which can be described using roll, pitch, and yaw). In this disclosure, the term "pose" refers to a combination of the position and attitude of an object or part of an object, which can be described, for example, using six parameters from the six degrees of freedom mentioned above.

[0030] In this disclosure, a reference coordinate system can be understood as a coordinate system capable of describing the pose of an object. Depending on the actual positioning requirements, the reference coordinate system can be selected with the origin of a virtual reference object or the origin of a physical reference object as its origin. In some embodiments, the reference coordinate system can be a world coordinate system, a camera coordinate system, or the operator's own perception coordinate system, etc. Here, the camera coordinate system refers to the coordinate system in which the image acquisition device is located.

[0031] In this disclosure, an object can be understood as an object or target that needs to be located, such as an actuator arm or the end effector of an actuator arm. The pose of the actuator arm or a portion thereof can refer to the pose of the coordinate system defined by the actuator arm or a portion thereof relative to a reference coordinate system.

[0032] Figure 1 A schematic diagram of an actuator detection system 100 (hereinafter also referred to as "System 100") according to some embodiments of the present disclosure is shown. Figure 1 As shown, system 100 may include a control device 120, an actuator arm 130, and an image acquisition device 150. The control device 120 may be communicatively connected to the image acquisition device 150 and the actuator arm 130 (e.g., a drive mechanism for the actuator arm 130). In some embodiments, the actuator arm 130 may include an actuator arm end effector 131 at its distal or end position. In some embodiments, an end effector 140 may be provided at the distal end of the actuator arm end effector 131.

[0033] In some embodiments, the control device 120 can be used to control the movement of the actuator arm 130 to adjust the position and orientation of the actuator arm 130, etc. The control device 120 can control the movement of the actuator arm 130 to move the actuator arm end cap 131 to a desired position and orientation. In some embodiments, such as... Figure 1 As shown, the system 100 may further include an input device 110a and / or a signal generation unit 110b, and the control device 120 may be communicatively connected to the input device 110a or the signal generation unit 110b. In some embodiments, the control device 120 may be communicatively connected to a drive device (e.g., a motor) not shown, and send a drive signal to the drive device based on a pre-input signal from the input device 110a or a signal randomly generated signal from the signal generation unit 110b, thereby causing the drive device to control the actuator arm 131 based on the drive signal, so that the end of the actuator arm 131 moves to the target pose corresponding to the drive signal.

[0034] In some embodiments, the actuator 130 may include a continuous deformable arm. A continuous deformable arm is, for example, as shown below. Figure 3 The actuator arm 300 is shown. In some embodiments, the actuator arm 130 may include a multi-degree-of-freedom actuator arm composed of multiple joints. For example, an actuator arm capable of 4 to 7 degrees of freedom of movement. For instance, an actuator arm capable of 6 degrees of freedom of movement.

[0035] In some embodiments, the image acquisition device 150 can be used to acquire positioning images. The positioning images may include part or all of the image of the actuator arm 130. In some embodiments, the image acquisition device 150 may include, but is not limited to, a dual-lens image acquisition device or a single-lens image acquisition device, such as a binocular or monocular camera. In some embodiments, the image acquisition device 150 can be used to acquire images of the actuator arm end cap 131, which may be provided with multiple different pose markers, including different pose marker patterns. For example, a positioning label 132 may be provided on the actuator arm end cap 131 (the positioning label 132 may be, for example, a...). Figure 4 The label 400 shown. The positioning label 132 may include multiple pose identifiers, and the multiple pose identifiers include different pose identifier patterns (detailed below).

[0036] like Figure 1 As shown, if the end effector arm 131 is within the field of view 151 of the image acquisition device 150, the acquired positioning image may include an image of the end effector arm 131. Depending on the application scenario, the image acquisition device 150 may be an industrial camera, underwater camera, miniature electronic camera, endoscope camera, etc. In some embodiments, the image acquisition device 150 may be fixed in position or have a variable position; for example, an industrial camera fixed at a monitoring position or an endoscope camera with adjustable position or orientation. In some embodiments, the image acquisition device 150 may perform at least one of visible light imaging, infrared imaging, CT (Computed Tomography) imaging, and acoustic imaging. Depending on the type of image acquired, those skilled in the art can select different image acquisition devices as the image acquisition device 150.

[0037] In some embodiments, the control device 120 may receive a positioning image from the image acquisition device 150 and process the positioning image. For example, the control device 120 may identify multiple pose markers located on the end effector 131 of the actuator arm in the positioning image and determine the relative position of the end effector 131 of the actuator arm to a reference coordinate system (e.g., world coordinate system, such as...). Figure 3 The pose shown is used as the actual pose.

[0038] In some embodiments, system 100 can determine the performance of the actuator arm 130 based on the actual and target poses of the actuator arm end effector 131. Those skilled in the art will understand that system 100 can be applied to specialized or general-purpose robotic systems in multiple fields (e.g., logistics, industrial manufacturing, medical, etc.). As an example, system 100 can be applied to robotic systems such as surgical robots, where the end effector 140 disposed at the distal end of the actuator arm end effector 131 can be, for example, a surgical actuator.

[0039] Figure 2 A schematic diagram of a segment 200 of an actuator arm according to some embodiments of the present disclosure is shown. The actuator arm (e.g., actuator arm 130) may include at least one deformable segment 200. Figure 2 As shown, the deformable segment 200 includes a fixed disk 210 and multiple structural bones 220. A first end of each structural bone 220 is fixedly connected to the fixed disk 210, and a second end is used to connect to a driving device (not shown). In some embodiments, the fixed disk 210 may be, but is not limited to, a ring-shaped structure, a disc-shaped structure, etc., and its cross-section may be circular, rectangular, polygonal, or various other shapes.

[0040] The driving device causes the segment 200 to deform by driving the structural bone 220. For example, the driving device causes the segment 200 to be in a position such as... Figure 2 The bending state is shown. In some embodiments, the second end of the multiple structural bones 220 passes through the base plate 230 and is connected to the driving device. In some embodiments, similar to the fixed plate 210, the base plate 230 can be, but is not limited to, a ring-shaped structure, a disc-shaped structure, etc., and the cross-section can be various shapes such as circular, rectangular, polygonal, etc. The driving device can include a linear motion mechanism, a driving segment, or a combination of both. The linear motion mechanism can be connected to the structural bone 220 to push or pull the structural bone 220, thereby driving the segment 200 to bend. The driving segment can include a fixed plate and multiple structural bones, wherein one end of the multiple structural bones is fixedly connected to the fixed plate. The other end of the multiple structural bones of the driving segment is connected to or integrally formed with the multiple structural bones 220 to drive the bending of the segment 200 by bending the driving segment.

[0041] In some embodiments, a spacer disk 240 is further included between the fixed disk 210 and the base disk 230, through which multiple structural bones 220 pass. Similarly, the drive segment may also include a spacer disk.

[0042] Figure 3 A schematic diagram of the structure of an actuator 300 according to some embodiments of the present disclosure is shown. For example... Figure 3 As shown, the actuator arm 300 is a deformable actuator arm, which may include an actuator arm end 310 and an actuator arm body 320. The actuator arm body 320 may include one or more components, such as a first component 3201 and a second component 3202. In some embodiments, the structures of the first component 3201 and the second component 3202 may be similar to those of other components. Figure 2 The shown component 200 is similar. In some implementations, such as... Figure 3As shown, the actuator arm body 320 also includes a first straight rod segment 3203 located between the first segment 3201 and the second segment 3202. The first end of the first straight rod segment 3203 is connected to the base plate of the second segment 3202, and the second end is connected to the fixed plate of the first segment 3201. In some embodiments, such as... Figure 3 As shown, the actuator arm body 320 also includes a second straight rod segment 3204, the first end of which is connected to the base plate of the first component 3201.

[0043] In some embodiments, the actuator 130 may be as follows: Figure 3 The actuator arm 300 is shown. (As shown in the image) Figure 3 As shown, each segment (first segment 3201 and second segment 3202) may include a base plate, a fixed plate, and multiple structural bones penetrating the base plate and the fixed plate. The multiple structural bones may be fixedly connected to the fixed plate and slidably connected to the base plate. The continuous deformable arm and its included segments can be described by a kinematic model (detailed below).

[0044] In some embodiments, the structure of each component may be specifically as follows: Figure 2 The shown component is 200. (As shown in the image) Figure 2 As shown, the base disk coordinate system The base disk is attached to the t-th (t=1,2,3…) section of the continuum, with its origin located at the center of the base disk, and the XY plane coinciding with the plane of the base disk. Pointing from the center of the base plate to the first structural bone (the first structural bone can be understood as any one of multiple structural bones chosen as a reference). Curved plane coordinate system. Its origin coincides with the origin of the base disk coordinate system, and the XY plane coincides with the bending plane. and Coincident. Fixed disk coordinate system. The origin of the fixed disk is located at the center of the fixed disk, and the XY plane coincides with the plane of the fixed disk. Pointing from the center of the fixed plate to the first structural bone. Curved plane coordinate system. Its origin is located at the center of the fixed disk, and the XY plane coincides with the bending plane. and coincide.

[0045] like Figure 2 The single segment 200 shown can be represented by a kinematic model. The position of the end of segment t (fixed disk coordinate system {te}) relative to the base disk coordinate system {tb}. tb P te ,attitude tb R te It can be determined based on the following formulas (1) and (2):

[0046]

[0047] tb R te = tb R t1 t1 R t2 t2 R te (2)

[0048] Among them, L t For the t-th segment, construct a virtual structural bone (e.g., Figure 2 The length of the virtual structural bone 221 shown in the figure, θ t In the t-th section, about or Rotate to Required rotation angle tb R t1 Let {t1} be the orientation of the bending plane coordinate system of segment t relative to the base disk coordinate system {tb}. t1 R t2 Let t be the orientation of the bending plane coordinate system 2{t2} of the t-th segment relative to the bending plane coordinate system 1{t1}. t2 R te Let {te} be the orientation of the fixed disk coordinate system {t2} of the t-th segment relative to the curved plane coordinate system 2{t2}.

[0049] tb R t1 , t1 R t2 and t2 R te It can be based on the following formulas (3), (4) and (5):

[0050]

[0051]

[0052]

[0053] Where, δ t For the t-th segment, the bending plane and The included angle.

[0054] like Figure 2 The joint parameter Ψ of the single segment 200 shown t It can be determined based on the following formula (6):

[0055] ψ t =[θ t ,δt ] T (6)

[0056] In some embodiments, the driving amount of multiple bone structures has a known mapping relationship with joint parameters. Based on the target joint parameters of the segment and the mapping relationship, the driving amount of the multiple bone structures can be determined. The driving amount of the multiple bone structures can be understood as moving a single segment from its initial state (e.g., θ) t =0) The length of the structural bone subjected to push or tension when bent to the target bending angle. In some embodiments, the mapping relationship between the driving amount of multiple structural bones and joint parameters can be determined based on the following formula (7):

[0057] q i ≡-r ti θ t cos(δ t +β ti (7)

[0058] Where, r ti Let β be the distance from the i-th structural bone in the t-th segment to the virtual structural bone. ti Let q be the angle between the i-th structural bone and the first structural bone in the t-th segment. i Let be the driving amount of the i-th structural bone. The driving signal of the driving device can be determined based on the driving amount of the i-th structural bone.

[0059] In some embodiments, the entire deformable arm can be described by a kinematic model. For example... Figure 3 As shown, transformations can be performed between multiple coordinate systems located at multiple positions of the deformable arm. For example, the actuator of the continuum deformable arm can be determined in a reference coordinate system (e.g., the world coordinate system {w}) based on the following formula (8):

[0060] w T tip = w T 1b 1b T 1e 1e T 2b 2b T 2e 2e T tip (8)

[0061] in, W T tip The homogeneous transformation matrix of the actuator of the continuum deformable arm relative to the reference coordinate system; W T 1b The homogeneous transformation matrix of the base disk of the first continuum segment relative to the reference coordinate system is represented. 1b T leRepresents the homogeneous transformation matrix of the fixed disk of the first continuous segment relative to the base disk of the first continuous segment; 1e T 2b Represents the homogeneous transformation matrix of the base disk of the second continuous segment relative to the fixed disk of the first continuous segment; 2b T 2e Represents the homogeneous transformation matrix of the fixed disk of the second continuum segment relative to the base disk of the second continuum segment; 2e T tip This represents the homogeneous transformation matrix of the actuator of the continuum deformable arm relative to the fixed disk of the second continuum segment. In some embodiments, the actuator is fixedly mounted on the fixed disk, therefore... 2e T tip It is known or predetermined.

[0062] It should be understood that deformable arms have different joint parameters in different operating states. For example, Figure 3 The actuator 300 shown includes at least four operating states. The four operating states of the actuator 300 are described below:

[0063] First working state: Only the second component 3202 participates in the actuator's pose control (e.g., only the second component 3202 enters the workspace), and the joint parameters of the actuator arm 300 can be determined based on the following formula (9):

[0064]

[0065] Where, ψ c1 To execute the joint parameters of arm 300 in the first working state, For the rotation angle of the actuator arm 300 around the axis, L2, θ2, δ2 and as shown Figure 2 In the structure shown in section 200, L t θ t and δ t They have the same physical meaning.

[0066] Second working state: The second component 3202 and the first linear segment 3203 participate in the pose control of the actuator (for example, the second component 3202 is fully in the workspace, and the first linear segment 3203 is partially in the workspace). At this time, the joint parameters of the actuator arm 300 can be determined based on the following formula (10):

[0067]

[0068] Where, ψ c2 To execute the joint parameters of arm 300 in the second working state, L r This is the feed rate for the first straight segment 3203.

[0069] Third working state: Second segment 3202, first straight segment 3203 and first segment 3201 participate in the position control of the actuator (for example, the second segment 3202 is fully in the workspace, the first straight segment 3203 is fully in the workspace, and the first segment 3201 is partially in the workspace). At this time, the joint parameters of the actuator arm 300 can be determined based on the following formula (11):

[0070]

[0071] Where, ψ c3 For the joint parameters of arm 300 in the third working state, L1, θ1, and δ1 are as follows: Figure 2 In the structure shown in section 200, L t θ t and δ t They have the same physical meaning.

[0072] Fourth working state: Second segment 3202, first linear segment 3203, first segment 3201 and second linear segment 3204 participate in the pose control of the actuator (for example, the second segment 3202 is fully in the workspace, the first linear segment 3203 is fully in the workspace, the first segment 3201 is fully in the workspace, and the second linear segment 3204 is partially in the workspace). At this time, the joint parameters of the actuator arm 300 can be determined based on the following formula (12):

[0073]

[0074] Where, ψ c4 To execute the joint parameters of arm 300 in the fourth working state, L s This is the feed rate for the second straight segment 3204.

[0075] In some embodiments, a plurality of pose markers are distributed on the actuator arm (e.g., on the actuator arm end 131). In some embodiments, the plurality of pose markers are disposed on the outer surface of the columnar portion of the actuator arm 130. For example, the plurality of pose markers are distributed circumferentially on the actuator arm end 131, for example, disposed circumferentially on the outer surface of the columnar portion of the actuator arm end 131. In some embodiments, a positioning label 132 including a plurality of pose markers is disposed on the outer surface of the columnar portion of the actuator arm end 131. The plurality of pose markers include a plurality of different pose marker patterns distributed circumferentially on the positioning label along the columnar portion and pose marker pattern corner points in the pose marker patterns.

[0076] In some embodiments, the pose identifier may include a pose identifier pattern and pose identifier pattern corner points within the pose identifier pattern. In some embodiments, the pose identifier pattern may be disposed on a label on the end of the actuator arm, or may be printed on the end of the actuator arm, or may be a pattern formed by the physical structure of the end of the actuator arm itself, for example, it may include recesses or protrusions and combinations thereof. In some embodiments, the pose identifier pattern may include a pattern formed with brightness, grayscale, color, etc. In some embodiments, the pose identifier pattern may include a pattern that actively (e.g., self-illuminating) or passively (e.g., reflecting light) provides information that can be detected by an image acquisition device. Those skilled in the art will understand that in some embodiments, the pose of the pose identifier or the pose of the pose identifier pattern may be represented by the pose of the pose identifier pattern corner point coordinate system. In some embodiments, the pose identifier pattern is disposed on an area on the end of the actuator arm suitable for image acquisition by an image acquisition device, for example, an area that can be covered by the field of view of the image acquisition device during operation or an area that is not easily disturbed or obstructed during operation.

[0077] Figure 4 A schematic diagram of a tag 400 including multiple pose identifiers according to some embodiments is shown. Figure 5 A schematic diagram shows a label 500 disposed on the periphery of the end of the actuator arm and forming a cylindrical shape. It can be understood that, for simplicity, label 400 may include the same pose identification pattern as label 500.

[0078] See Figure 4 Multiple pose identifiers may include multiple different pose identifier patterns 410. Multiple pose identifiers may also include multiple pose identifier pattern corner points P4 within the multiple different pose identifier patterns 410, which are represented by the symbol "0" in this disclosure. In some embodiments, pose identifiers can be determined by identifying the pose identifier pattern 410 or the pose identifier pattern corner points P4 therein.

[0079] See Figure 5In the circumferential setting state, label 400 becomes label 500 with a cylindrical spatial structure. In some embodiments, the axial angle or roll angle of the pose identifier can be represented by the axial angle of the pose identifier pattern or the corner point of the pose identifier pattern. The axial angle of each pose identifier pattern or corner point is known or predetermined. In some embodiments, the axial angle identified by each pose identifier can be determined based on the distribution of multiple pose identifiers (e.g., pose identifier patterns or corner points of pose identifier patterns). In some embodiments, the multiple pose identifiers can be uniformly distributed (e.g., the corner points of the pose identifier patterns in label 400 are evenly spaced, and the corner points of the pose identifier patterns in label 500 are evenly distributed). In other embodiments, the multiple pose identifiers can be non-uniformly distributed. In some embodiments, based on the distribution of multiple pose identifiers, each pose identifier pattern can be used to identify a specific axial angle, and each pose identifier pattern has a one-to-one correspondence with the identified axial angle. In this disclosure, the angle about the axis or roll angle refers to the angle about the Z-axis (e.g., the Z-axis of the end-effector coordinate system {wm}). In some embodiments, the Z-axis may be along the tangent direction of the end-effector.

[0080] like Figure 5 As shown, multiple different pose marker patterns 510 in the label 500 are evenly distributed circumferentially along the cylindrical structure. The corner points of multiple pose marker patterns are evenly distributed on the cross-sectional circle 520 of the XY plane of the end coordinate system {wm} of the actuator arm. Then, the distribution angle (e.g., angle α0) of any adjacent pose marker pattern corner points is equal. Set the pose marker pattern corner point P5 pointing to the X-axis. P5 is used as the reference corner point for marking the 0° angle around the axis (the pose marker pattern where the pose marker pattern corner point P5 is located is used as the reference pattern). Then, the angle around the axis of the pose marker pattern corner point can be determined according to the positional relationship between any pose marker pattern corner point and the pose marker pattern corner point P5. In some embodiments, the angle around the axis of the pose marker pattern corner point can be determined based on the following formula (13):

[0081] α m =α0(m-1) (13)

[0082] Where, α m Let P5 be the first pose marker corner point, and the angle around the axis of the m-th pose marker corner point be in the clockwise direction of the cross-sectional circle 520.

[0083] Some embodiments of this disclosure provide a method for detecting an actuator arm. Figure 6 A flowchart illustrating a detection method 600 of an actuator detection system (e.g., actuator detection system 100) according to some embodiments of the present disclosure is shown. Figure 6As shown, some or all of the steps in method 600 can be executed by a control device (e.g., control device 120) of the execution arm detection system 100. Control device 120 can be configured on a computing device. Method 600 can be implemented by software, firmware, and / or hardware. In some embodiments, method 600 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 19 The processor 1920 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0084] See Figure 6 In step 601, a drive signal corresponding to the target pose of the end effector arm is determined for controlling the target pose. In some embodiments, the drive signal corresponding to the target pose of the end effector arm can be determined by the user via... Figure 1 The input device 110a shown is preset, and the control device 120 controls the end effector of the actuator arm to move to the corresponding target pose based on the preset drive signal. Those skilled in the art will understand that there is a mapping relationship between the drive signal and the pose of the actuator arm. Therefore, when the drive signal is preset, the mapping relationship between the drive signal and the pose of the actuator arm can be pre-stored in the memory of the system 100. The control device 120 can determine the target pose of the end effector of the actuator arm based on the preset drive signal using a lookup table method. In some embodiments, the drive signal corresponding to the target pose of the end effector of the actuator arm can be provided by… Figure 1 The signal generation unit 110b shown generates signals randomly, and the control device 120 controls the end effector of the actuator arm to move to the corresponding target pose based on the randomly generated drive signals.

[0085] In some embodiments, a target pose of the end effector of the actuator arm may be preset or randomly generated, and the control device 120 determines a drive signal for controlling the end effector of the actuator arm based on the target pose. In some embodiments, a target joint parameter set corresponding to the configuration of the actuator arm in the target pose may be preset or randomly generated, and the control device 120 determines a drive signal for controlling the end effector of the actuator arm based on the target joint parameter set. For example, an exemplary method for determining a drive signal for controlling the target pose of the end effector of the actuator arm includes, for instance, […]. Figure 7 or Figure 8 The method shown. In some embodiments, the target pose of the end effector is the target pose of the end effector in the reference coordinate system.

[0086] See Figure 6 In step 603, a positioning image is acquired. In some embodiments, the positioning image includes multiple pose markers on the end effector of the actuator arm. In some embodiments, these markers can be obtained from, for example... Figure 1The image acquisition device 150 shown receives a positioning image. For example, the control device 120 can receive a positioning image actively sent by the image acquisition device 150. Alternatively, the control device 120 can send an image request command to the image acquisition device 150, and the image acquisition device 150 responds to the image request command by sending a positioning image to the control device 120.

[0087] Continue reading Figure 6 In step 605, multiple pose markers located on the end of the actuator arm are identified in the positioning image. These multiple pose markers include different pose marker patterns. For example, an exemplary method for identifying multiple pose markers located on the end of the actuator arm may include, for instance... Figure 12 , Figure 14 , Figure 16 and Figure 17 The method is illustrated. In some embodiments, the control device 120 can identify some or all of the pose markers in the positioning image using an image processing algorithm. In some embodiments, the image processing algorithm may include a feature recognition algorithm, which can extract or recognize features of the pose markers. For example, the image processing algorithm may include a corner detection algorithm for detecting corner points of the pose marker pattern. The corner detection algorithm may be one of, but is not limited to, corner detection based on grayscale images, corner detection based on binary images, and corner detection based on contour curves. For example, the image processing algorithm may be a color feature extraction algorithm for detecting color features in the pose marker pattern. As another example, the image processing algorithm may be a contour detection algorithm for detecting contour features of the pose marker pattern. In some embodiments, the control device can identify some or all of the pose markers in the positioning image using a recognition model.

[0088] Continue reading Figure 6 In step 607, the actual pose of the end effector of the actuator arm is determined based on multiple pose identifiers. In some embodiments, method 600 further includes: determining the two-dimensional coordinates of the multiple pose identifiers in a positioning image; and determining the pose of the end effector coordinate system relative to a reference coordinate system based on the two-dimensional coordinates of the multiple pose identifiers in the positioning image and the three-dimensional coordinates of the multiple pose identifiers relative to the end effector coordinate system of the actuator arm, as the actual pose of the end effector of the actuator arm. In some embodiments, the coordinates of the pose identifiers can be represented by the coordinates of the corner points of the pose identifier pattern. For example, the two-dimensional coordinates of the pose identifiers in the positioning image and the three-dimensional coordinates of the pose identifiers in the end effector coordinate system of the actuator arm can be represented by the coordinates of the corner points of the pose identifier pattern. In some embodiments, the pose of the end effector coordinate system relative to a reference coordinate system can be determined based on the two-dimensional coordinates of the corner points of the multiple pose identifier patterns in the positioning image and the three-dimensional coordinates of the end effector coordinate system of the actuator arm, as the actual pose of the end effector of the actuator arm.

[0089] Continue reading Figure 6In step 609, the performance of the actuator is determined based on the target pose and the actual pose. In some embodiments, method 600 may further include generating a performance signal in response to the target pose and the actual pose reaching an error condition, the performance signal indicating that the actuator's performance is unqualified. For example, the control device may determine the target pose of the actuator's end effector in a reference coordinate system or the drive signal corresponding to the target pose, and determine the actual pose of the actuator's end effector based on the actuator's positioning image. When the target pose and the actual pose meet an error condition (e.g., greater than or equal to an error threshold), a performance signal indicating that the actuator's performance is unqualified is issued. In some embodiments, the performance signal may include multiple performance alarm signals. The multiple performance alarm signals may include factory performance alarm signals and operational performance alarm signals depending on the actuator's usage status. In some embodiments, the factory performance alarm signal indicates that the actuator has failed the factory performance test and its performance does not meet the factory standard. In some embodiments, the operational performance alarm signal indicates that an actuator that has been in service has failed the operational performance test and its performance does not meet the operational requirements, requiring maintenance or replacement.

[0090] In some embodiments, method 600 may further include determining a pose difference based on the target pose of the end effector of the actuator arm in a reference coordinate system and the actual pose of the end effector of the actuator arm in the reference coordinate system, and determining the performance of the actuator arm based on the pose difference and an error condition. The pose difference may include a position difference and an orientation difference. In some embodiments, the pose difference between the target pose and the actual pose of the end effector of the actuator arm can be determined by a single actuator arm detection, and the performance of the actuator arm can be determined based on whether the pose difference satisfies an error condition. In some embodiments, multiple pose differences between the target pose and the actual pose of the end effector of the actuator arm can be determined by multiple actuator arm detections, and the performance of the actuator arm can be determined based on whether these pose differences satisfy an error condition (e.g., the average or cumulative value of the multiple pose differences is greater than or equal to an error threshold). In some embodiments, method 600 may further include determining the actual pose of the end effector of the actuator arm at a predetermined detection cycle (e.g., performing steps 603-607 at a predetermined detection cycle) to determine or cumulatively determine the performance of the actuator arm in real time through multiple detection cycles.

[0091] In the k-th execution arm detection loop, the pose difference can be expressed as follows:

[0092]

[0093] in, Let $\frac{ ... Let P be the angle difference of the actuator arm during the k-th actuator arm detection cycle. t k R represents the target position of the actuator arm during the k-th actuator arm detection loop. tk The target pose of the execution arm during the k-th execution arm detection loop. R represents the actual position of the actuator arm during the k-th actuator arm detection loop. r k This represents the actual posture of the execution arm during the k-th execution arm detection loop. express and The corner between them.

[0094] Figure 7 This is a flowchart illustrating a method 700 for determining a drive signal for controlling the target pose of the end effector arm according to some embodiments of the present disclosure. Figure 7 As shown, some or all of the steps in method 700 can be executed by a control device (e.g., control device 120) of the execution arm detection system 100. Control device 120 can be configured on a computing device. Method 700 can be implemented by software, firmware, and / or hardware. In some embodiments, method 700 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 19 The processor 1920 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0095] See Figure 7 In step 701, the motion trajectory of the end effector of the actuator arm is preset, and the motion trajectory includes multiple target poses. In some embodiments, the motion trajectory of the end effector of the actuator arm can be determined by the user through... Figure 1 The input device 110a shown is preset, or can be accessed via... Figure 1 The signal generation unit 110b shown is pre-generated. The motion trajectory may include multiple target poses of the end effector of the actuator arm. In some embodiments, the motion trajectory may be a target path traversed by the end effector of the actuator arm, which may include multiple target path points arranged sequentially according to a predetermined motion cycle. These path points respectively indicate the target position and target pose of the end effector of the actuator arm in multiple motion cycles. The detection cycle of the actuator arm detection cycle may be the same as the motion cycle of the actuator arm motion cycle. Alternatively, the detection cycle of the actuator arm detection cycle may be an integer multiple of the motion cycle of the actuator arm motion cycle.

[0096] In step 703, multiple drive signals corresponding to multiple target poses are determined. In some embodiments, the target pose of the end effector of the actuator arm in the previous motion cycle in the motion trajectory is taken as the starting pose of the current motion cycle. Based on the difference between the target pose and the starting pose of the end effector of the actuator arm in the current motion cycle, the drive signal of at least one drive device controlling the motion of the actuator arm can be determined by using an inverse kinematic numerical iterative algorithm of the actuator arm kinematic model. It should be understood that the kinematic model can be a mathematical model representing the motion relationship between the joint space and the task space of the actuator arm. For example, the kinematic model can be established by methods such as the Denavit-Hartenberg (DH) parameter method and the exponential product representation method.

[0097] In some embodiments, method 700 may further include: sending a plurality of drive signals sequentially at a predetermined motion cycle; determining the actual pose of the end effector of the actuator arm at a predetermined detection cycle; and determining the performance of the actuator arm based on the plurality of target poses and the plurality of actual poses. For example, after determining the plurality of drive signals corresponding to the plurality of target poses in the motion trajectory, the control device may send the plurality of drive signals sequentially to the drive device at a predetermined motion cycle, determine the actual pose of the end effector of the actuator arm based on the positioning image of the actuator arm at a predetermined detection cycle, and determine the performance of the actuator arm based on the plurality of target poses and the plurality of actual poses.

[0098] In some embodiments, multiple actual poses arranged with a predetermined motion cycle constitute the actual trajectory of the end effector of the actuator arm. The error condition can be trajectory accuracy, which reflects the degree of deviation of the actual trajectory of the end effector of the actuator arm from the motion trajectory. For example, trajectory accuracy can be the maximum pose difference between multiple actual poses of the end effector of the actuator arm on the entire motion trajectory and multiple target poses, or it can be the average pose difference or cumulative pose difference between multiple actual poses of the end effector of the actuator arm on the entire motion trajectory and multiple target poses. In some embodiments, the control device can issue a performance signal indicating that the performance of the actuator arm is unqualified when the trajectory accuracy of the end effector of the actuator arm is greater than or equal to a predetermined threshold.

[0099] Figure 8 This is a flowchart illustrating a method 800 for determining a drive signal for controlling the target pose of the end effector arm according to other embodiments of this disclosure. Figure 8 As shown, some or all of the steps in method 800 can be executed by a control device (e.g., control device 120) of the execution arm detection system 100. Control device 120 can be configured on a computing device. Method 800 can be implemented by software, firmware, and / or hardware. In some embodiments, method 800 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 19The processor 1920 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0100] See Figure 8 In step 801, a target joint parameter set for the joints of the actuator arm is preset or randomly generated. In some embodiments, the pose of the actuator arm can be represented by a set of joint information (e.g., a matrix composed of this joint information) of the joints included in the actuator arm, and the joint parameter set of the actuator arm's joints has a mapping relationship with the pose of the actuator arm. In some embodiments, the target joint parameter set of the actuator arm's joints can be determined by the user through... Figure 1 The input device 110a shown is preset or via Figure 1 The signal generation unit 110b shown generates signals randomly, and the target joint parameter set determines the configuration of the actuator arm and the target pose of the actuator arm's end effector. In some embodiments, if the target joint parameter set of the actuator arm's joints is preset, the mapping relationship between the joint parameter set of the actuator arm's joints and the pose of the actuator arm can be stored in advance, and the target pose of the actuator arm's end effector can be determined by a lookup table method based on the preset target joint parameter set.

[0101] In step 803, a drive signal and the corresponding target pose are determined based on the target joint parameter set. In some embodiments, the control device may determine the drive signal of at least one drive device driving the actuator arm and the target pose of the actuator arm's end effector determined by the target joint parameter set based on the target joint parameter set of the actuator arm's joints, using the actuator arm's kinematic model. Step 803 can be implemented similarly to step 703 in method 700.

[0102] In some embodiments, method 800 may further include: determining the driving amount of multiple structural bones based on a target joint parameter set; and determining a driving signal based on the driving amount of the multiple structural bones. For example, based on the mapping relationship between the target joint parameters and the driving amount of the multiple structural bones, the driving amount of multiple joints included in the actuator arm in the current detection cycle can be determined, and then a driving signal of at least one driving device (e.g., a motor) can be determined based on the driving amount. In some embodiments, the actuator arm is taken as a continuous deformable arm as an example. A continuous deformable arm can be as follows: Figure 3 The actuator 300 shown can determine the driving amount of each structural bone of each segment based on formula (7), and then determine the driving signal of the driving device based on the driving amount.

[0103] Those skilled in the art will understand that each joint of the actuator arm includes its own joint parameter range, and the joint parameter range of each joint constitutes the joint space of the actuator arm. When randomly generating the target joint parameter set for the actuator arm's joints, it is possible that one or more randomly generated target joint parameters exceed the joint parameter range, resulting in the actuator arm's pose determined by the randomly generated target joint parameter set being outside the actuator arm's workspace. In some embodiments, method 900 can converge the target joint parameter set of the actuator arm's joints within the joint parameter range.

[0104] Figure 9 This is a flowchart illustrating a method 900 for updating a target joint parameter set of a joint of an actuator arm according to some embodiments of the present disclosure. Figure 9 As shown, some or all of the steps in method 900 can be executed by a control device (e.g., control device 120) of the execution arm detection system 100. Control device 120 can be configured on a computing device. Method 900 can be implemented by software, firmware, and / or hardware. In some embodiments, method 900 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 19 The processor 1920 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0105] See Figure 9 In step 901, in response to at least one target joint parameter in the randomly generated set of target joint parameters exceeding the joint parameter range, at least one target joint parameter is updated to the limit value of the corresponding joint parameter range. In some embodiments, the target joint parameters in the randomly generated set of target joint parameters are compared with the joint parameter range. In response to a target joint parameter exceeding the joint parameter range, the out-of-range target joint parameter is updated to the limit value of the corresponding joint parameter range closest to the target joint parameter.

[0106] In step 903, a drive signal and the corresponding target pose are determined based on the updated target joint parameter set. In some embodiments, after converging out-of-range target joint parameters to the limit values ​​of their joint parameter range, the drive signal of at least one drive device driving the actuator arm and the target pose of the actuator arm's end effector corresponding to the target joint parameter set are determined based on the updated target joint parameter set using the kinematic model of the actuator arm. Step 903 can be implemented similarly to step 703 in method 700 and step 803 in method 800.

[0107] Those skilled in the art will understand that when the actuator arm's joints are near their range limits, the actual pose of its end effector is more likely to deviate from the target pose. By updating the target joint parameters that exceed the joint parameter range to the corresponding limit values ​​of the joint parameter range, the target pose that exceeds the actuator arm's workspace can be updated to a target pose where at least one target joint parameter is at its limit value. This allows for actuator arm performance detection at the actuator arm's limit position, improving detection efficiency. In some embodiments, method 900 can be used to update the randomly generated drive signal or the target pose of the actuator arm's end effector. In some embodiments, other methods can also be used to update or filter the randomly generated signal (e.g., drive signal, the target pose of the actuator arm's end effector, or the target joint information set of the actuator arm's joints). For example, the randomly generated signal can be filtered to remove signals that exceed the range of key parameters, preventing the target pose of the actuator arm's end effector from being outside the actuator arm's limit position. In some embodiments, a signal corresponding to the target pose of the actuator arm's end effector can also be randomly generated within a preset range, thereby avoiding interference with actuator arm detection due to the actuator arm's inability to move to the target pose.

[0108] In some embodiments, method 600 may further include: determining the pose of the end-effector coordinate system relative to the reference coordinate system based on the two-dimensional coordinates of multiple pose marker corner points in the positioning image, the three-dimensional coordinates of the multiple pose marker corner points in the end-effector coordinate system of the actuator arm, and the transformation relationship between the camera coordinate system and the reference coordinate system. In some embodiments, the transformation relationship between the camera coordinate system and the reference coordinate system may be known. For example, the reference coordinate system may be the world coordinate system, and the transformation relationship between the camera coordinate system and the world coordinate system may be determined based on the camera's orientation. In other embodiments, the reference coordinate system may be the camera coordinate system itself, depending on actual needs. In some embodiments, based on the camera imaging principle and projection model, the pose of the end-effector coordinate system relative to the camera coordinate system is determined based on the two-dimensional coordinates of the multiple pose marker corner points in the positioning image and the three-dimensional coordinates of the multiple pose marker corner points in the end-effector coordinate system of the actuator arm. Based on the pose of the end-effector coordinate system relative to the camera coordinate system and the transformation relationship between the camera coordinate system and the reference coordinate system, the pose of the end-effector coordinate system relative to the reference coordinate system can be obtained. In some embodiments, the camera's intrinsic parameters may also be considered. For example, the camera's intrinsic parameters could be as follows: Figure 1The image acquisition device 150 shown has camera intrinsic parameters. These parameters can be known or obtained through calibration. In some embodiments, the camera coordinate system can be understood as a coordinate system established with the camera origin. For example, a coordinate system established with the camera's optical center as the origin or a coordinate system established with the camera's lens center as the origin. When the camera is a stereo camera, the origin of the camera coordinate system can be the center of the left lens, the center of the right lens, or any point on the line connecting the centers of the left and right lenses (e.g., the midpoint of that line).

[0109] In some embodiments, the pose of the end-effector coordinate system {wm} relative to the reference coordinate system (e.g., the world coordinate system) {w} can be determined based on the following formula (15):

[0110] w R wm = w R lens lens R wm

[0111] w P wm = w R lens ( lens R wm + lens P wm )+ w P lens (15)

[0112] in, w R wm The attitude of the end effector's coordinate system relative to the reference coordinate system. w P wm The position of the end effector's coordinate system relative to the reference coordinate system. w R lens The pose of the camera coordinate system relative to the reference coordinate system. w P lens The position of the camera coordinate system relative to the reference coordinate system. lens R wm The orientation of the end effector's coordinate system relative to the camera coordinate system. lens P wm The position of the end effector coordinate system relative to the camera coordinate system.

[0113] This disclosure provides several embodiments of a method for determining the three-dimensional coordinates of a plurality of pose markers relative to the end effector coordinate system of an actuator arm. In some embodiments, the three-dimensional coordinates of the plurality of pose markers relative to the end effector coordinate system of the actuator arm are determined based on the distribution of the plurality of pose markers. For example, the three-dimensional coordinates of the corner points of the plurality of pose marker patterns in the end effector coordinate system of the actuator arm are determined based on the distribution of the corner points of the plurality of pose marker patterns.

[0114] Figure 10 A flowchart illustrating a method 1000 for determining the three-dimensional coordinates of a plurality of pose identifiers relative to the end effector coordinate system of an actuator arm, according to some embodiments of the present disclosure. Figure 10 As shown, some or all of the steps in method 1000 can be executed by a control device (e.g., control device 120) of the execution arm detection system 100. Control device 120 can be configured on a computing device. Method 1000 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1000 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 19 The processor 1920 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0115] See Figure 10 In step 1001, based on the distribution of multiple pose markers, the axial angles of the multiple pose markers relative to the Z-axis of the end effector coordinate system of the actuator arm are determined. In some embodiments, the axial angles of the multiple pose markers relative to the Z-axis of the end effector coordinate system of the actuator arm can be determined based on multiple pose marker patterns. For example, each pose marker pattern can identify a specific axial angle, and different pose marker patterns correspond one-to-one with the identified axial angles. Based on the identification of the pose marker patterns and the correspondence between the pose marker patterns and the axial angles, the axial angles identified by the identified pose marker patterns can be determined. It should be understood that the distribution of each pose marker pattern is known or predetermined. In some embodiments, the distribution of multiple pose marker patterns or the corner points of multiple pose marker patterns can be as follows: Figure 4 The distribution is shown. In some embodiments, the angle around the axis of each pose marker corner marker can also be determined based on formula (13).

[0116] See Figure 10 In step 1003, based on the about-axis angles of the multiple pose markers, the three-dimensional coordinates of the multiple pose markers relative to the end effector coordinate system of the actuator arm are determined. In some embodiments, such as Figure 5As shown, each pose marker corner point is located on the circumference of the cross-sectional circle 520, and the center and radius r of the cross-sectional circle 520 are known. Taking the pose marker corner point P5 as the reference corner point, the three-dimensional coordinates of the pose marker corner point P5 in the end coordinate system {wm} of the actuator arm are (r, 0, 0). In some embodiments, the three-dimensional coordinates of each pose marker corner point in the end coordinate system {wm} of the actuator arm can be determined based on the following formula (16):

[0117] C m =[r·cosα] m r·sinα m 0] T (16)

[0118] Among them, C m With pose marker corner point P5 as the first pose marker corner point, the specific angle around the axis of the mth pose marker corner point can be based on the three-dimensional coordinates of multiple pose markers in the end coordinate system {wm} of the actuator arm, following the clockwise direction of the cross-sectional circle 520.

[0119] In some embodiments, the axial angle α of the corner point marker of the m-th pose marker pattern is determined based on formula (13). m Then, the angle α around the axis determined by formula (13) m The three-dimensional coordinates C are determined by formula (16). m .

[0120] Figure 11 A flowchart illustrating a method 1100 for determining the three-dimensional coordinates of a plurality of pose identifiers relative to the end effector coordinate system of an actuator arm, according to other embodiments of the present disclosure. Method 1100 may be... Figure 10 Alternative embodiments of the method 1000 shown. For example... Figure 11 As shown, some or all of the steps in method 1100 can be executed by a control device (e.g., control device 120) of the execution arm detection system 100. Control device 120 can be configured on a computing device. Method 1100 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1100 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 19 The processor 1920 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0121] See Figure 11In step 1101, the arrangement order of the multiple pose identifiers is determined based on at least two of them. In some embodiments, the arrangement order of the multiple pose identifiers can be represented by the arrangement order of multiple pose identifier patterns. In some embodiments, the arrangement order of the multiple pose identifiers is determined by identifying any two pose identifier patterns. It should be understood that the multiple pose identifiers include different pose identifier patterns, and when any two pose identifier patterns are known, the arrangement order can be determined based on the known distribution of the multiple pose identifier patterns (e.g., Figure 4 The distribution of different pose marker patterns in label 400 shown, or Figure 5 The distribution of different pose marker patterns in the label 500 shown determines the arrangement order of multiple pose markers in the positioning image, such as clockwise or counterclockwise arrangement.

[0122] See Figure 11 In step 1103, based on the arrangement order of multiple pose markers, the three-dimensional coordinates of the multiple pose markers relative to the end effector coordinate system of the actuator arm are determined. In some embodiments, based on the known distribution of the multiple pose markers, the three-dimensional coordinates of each pose marker in the end effector coordinate system of the actuator arm can be determined. The three-dimensional coordinates of each pose marker can be represented by the three-dimensional coordinates of the corner points of the pose marker pattern in the end effector coordinate system of the actuator arm, and each pose marker pattern corresponds to a coordinate point in the end effector coordinate system of the actuator arm. After determining the arrangement order of the multiple pose marker patterns, the remaining pose marker patterns can be determined based on the identified pose marker patterns, and then the three-dimensional coordinates of each pose marker pattern in the end effector coordinate system of the actuator arm can be determined. In some embodiments, multiple pose marker corner points in the positioning image are identified, and any two corresponding pose marker patterns among the multiple pose marker corner points are determined. Based on the two identified pose marker patterns, the arrangement order of the corner points of the multiple pose marker patterns is determined, and then the three-dimensional coordinates of each pose marker pattern corner point in the end effector coordinate system of the actuator arm can be determined. Furthermore, based on the arrangement order, the distribution of all pose marker patterns can be determined, thereby matching a specific pose pattern matching template with the pose marker pattern at the corresponding position on the positioning image, improving data processing speed. In some embodiments, the pattern matching between the pose pattern matching template and the pattern at the corner of the pose marker pattern can be implemented similarly to step 1203 in method 1200.

[0123] This disclosure provides some embodiments of a method for identifying pose markers. Figure 12 A flowchart illustrating a method 1200 for identifying pose identifiers according to some embodiments of the present disclosure is shown. Figure 12As shown, some or all of the steps in method 1200 can be executed by a control device (e.g., control device 120) of the execution arm detection system 100. Control device 120 can be configured on a computing device. Method 1200 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1200 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 19 The processor 1920 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0124] refer to Figure 12 In step 1201, a plurality of candidate pose identifiers are determined from the positioning image. In some embodiments, the pose identifiers may include corner points of the pose identifier pattern in the pose identifier pattern. The coordinates or origin of the coordinate system of the candidate pose identifiers can be represented by the corner points of the candidate pose identifier pattern. In some embodiments, the corner points of the candidate pose identifier pattern may refer to possible corner points of the pose identifier pattern obtained after preliminary processing or preliminary identification of the positioning image.

[0125] In some embodiments, method 1200 may further include determining a region of interest (ROI) in the localization image. For example, the ROI may be cropped from the localization image, and multiple candidate pose markers may be determined from the ROI. The ROI may be the entire localization image or a partial region. For example, the ROI of the current frame may be cropped based on a region within a certain range of the corner points of multiple pose marker patterns determined in the previous frame (e.g., the localization image of the previous image processing cycle). For localization images that are not the first frame, the ROI may be a region within a certain distance centered on a virtual point formed by the coordinates of the corner points of multiple pose marker patterns from the previous image processing cycle. The certain distance range may be a fixed multiple of the average spacing distance of the corner points of the pose marker patterns, such as twice. It should be understood that the predetermined multiple may also be a variable multiple of the average spacing distance of the corner points of multiple candidate pose marker patterns in the previous image processing cycle.

[0126] In some embodiments, method 1200 may further include determining the corner likelihood (CL) value of each pixel in the localization image. In some embodiments, the corner likelihood value of a pixel may be a numerical value characterizing the probability that the pixel is a feature point (e.g., a corner). In some embodiments, the localization image may be preprocessed before calculating the corner likelihood value of each pixel, and then the corner likelihood value of each pixel in the preprocessed image may be determined. Image preprocessing may include, for example, at least one of image grayscale conversion, image denoising, and image enhancement. For example, image preprocessing may include: cropping a Region of Interest (ROI) from the localization image and converting the ROI to a corresponding grayscale image.

[0127] In some embodiments, determining the corner likelihood value of each pixel in the ROI may include, for example, performing a convolution operation on each pixel within the ROI to obtain the first and / or second derivatives of each pixel. The corner likelihood value of each pixel is then calculated using the first and / or second derivatives of each pixel within the ROI. For example, the corner likelihood value of each pixel can be determined based on the following formula (17):

[0128]

[0129] Where τ is a set constant, for example, set to 2; I x I 45 I y I n45 These are the first derivatives of the pixel in the four directions: 0, π / 4, π / 2, and -π / 4; I xy and I 45_45 These are the second derivatives of the pixel in the directions of 0, π / 2 and π / 4, -π / 4, respectively.

[0130] In some embodiments, method 1200 may further include dividing the ROI into multiple sub-regions. For example, a non-maximum suppression method can be used to evenly segment a ROI into multiple sub-images. In some embodiments, the ROI can be evenly segmented into multiple sub-images of 5×5 pixels. The above embodiments are exemplary and not limiting. It should be understood that the location image or ROI can also be segmented into multiple sub-images of other sizes, such as multiple sub-images of 9×9 pixels.

[0131] In some embodiments, method 1200 may further include determining the pixel with the largest corner likelihood value in each sub-region to form a pixel set. In some embodiments, the pixel set serves as a plurality of candidate identifiers determined from the localization image. For example, the pixel with the largest CL value in each sub-image may be determined, and the pixel with the largest CL value in each sub-image may be compared with a first threshold to determine a set of pixels with a CL value greater than the first threshold. In some embodiments, the first threshold may be set to 0.06. It should be understood that the first threshold may also be set to other values.

[0132] See Figure 12Step 1203: Based on multiple different pose pattern matching templates, identify the first pose identifier from multiple candidate pose identifiers. In some embodiments, the multiple different pose pattern matching templates are matched with the patterns at the corner points of the candidate pose identifier patterns to identify the first pose identifier. For example, the corner points of the candidate pose identifier patterns that meet a preset pose pattern matching degree standard are determined as the corner points of the first pose identifier pattern. In some embodiments, the pose pattern matching template and the pattern in the vicinity of the corner point of the pose identifier pattern have the same or similar features. If the matching degree between the pose pattern matching template and the pattern in the vicinity of the corner point of the candidate pose identifier pattern reaches a preset pose pattern matching degree standard (e.g., the matching degree is higher than a threshold), it can be considered that the pattern in the vicinity of the corner point of the candidate pose identifier pattern has the same or similar features as the pose pattern matching template, and thus the current corner point of the candidate pose identifier pattern can be considered as the corner point of the pose identifier pattern.

[0133] In some embodiments, the pixel with the largest CL value in the pixel set is identified as a candidate pose identifier pattern corner point. For example, all pixels in the pixel set can be sorted in descending order of CL value, and the pixel with the largest CL value can be selected as the candidate pose identifier pattern corner point. In some embodiments, after determining the candidate pose identifier pattern corner point, a pose pattern matching template is matched with the pattern at the candidate pose identifier pattern corner point. If a preset pose pattern matching degree standard is met, the candidate pose identifier pattern corner point is determined as the first identified pose identifier pattern corner point.

[0134] In some embodiments, method 1200 may further include, in response to a matching failure, determining the pixel with the largest corner likelihood value among the remaining pixels in the pixel set as a candidate pose identifier pattern corner point. For example, if the candidate pose identifier pattern corner point does not meet a preset matching degree standard, then the pixel with the second largest CL value (the pixel with the second largest CL value) is selected as the candidate pose identifier pattern corner point, and the pose pattern matching template is matched with the pattern at the candidate pose identifier pattern corner point, and so on, until the first pose identifier pattern corner point is identified.

[0135] In some embodiments, the pose identification pattern can be a black and white alternating pattern (e.g., a checkerboard pattern), therefore the pose pattern matching template can be the same pattern, utilizing the grayscale distribution G of the pose pattern matching template. M The pixel neighborhood grayscale distribution G of the pixel corresponding to the corner point of the candidate pose identifier pattern image The correlation coefficient (CC) between pixels is used for matching. The grayscale distribution G of the pixel neighborhood is also considered. image This refers to the grayscale distribution of pixels within a certain range (e.g., 10×10 pixels) centered on the given pixel. The correlation coefficient can be determined based on the following formula (18):

[0136]

[0137] Where Var() is the variance function and Cov() is the covariance function. In some embodiments, when the correlation coefficient is less than 0.8, the gray-level distribution in the pixel neighborhood has a low correlation with the pose pattern matching template. In this case, the candidate pose pattern corner with the highest corner likelihood value is determined not to be a pose pattern corner. Otherwise, the candidate pose pattern corner with the highest corner likelihood value is considered to be a pose pattern corner.

[0138] In some embodiments, method 1200 may further include determining the edge orientation of the corner points of the candidate pose identifier pattern. For example, as Figure 13 As shown, the corner point of the candidate pose identifier pattern is corner point P in pose identifier pattern 1300. 13 Corner point P 13 The edge direction can refer to the direction of the corner point P. 13 The direction of the edge, such as Figure 13 The direction indicated by the dashed arrow.

[0139] In some embodiments, the edge direction can be determined by the first-order derivative (I0) of each pixel in the X and Y directions of the planar coordinate system with respect to a certain neighborhood (e.g., 10×10 pixels) centered on the corner point of the candidate pose identifier pattern. x and I y The edge direction can be determined based on the following formula (19):

[0140]

[0141] Among them, the first derivative (I) x and I y This can be obtained by performing a convolution operation on each pixel within a certain neighborhood range. In some embodiments, this is achieved by performing a convolution operation on the edge direction I of each pixel within the neighborhood range. angle and the corresponding weight I weight Clustering calculations are performed to obtain the edge direction of the pixel, and weight I is selected. weight The class with the largest proportion corresponds to I angle As the edge direction. It should be noted that if multiple edge directions exist, then weight I is selected. weight The I corresponding to the largest proportion of multiple classes angle As the edge direction.

[0142] In some embodiments, the clustering calculation method can be any one of the following: K-means, BIRCH (Balanced Iterative Reducing and Clustering using Hierarchies), DBSCAN (Density-Based Spatial Clustering of Applications with Noise), or GMM (Gaussian Mixed Model).

[0143] In some embodiments, method 1200 may further include rotating the pose pattern matching template based on the edge direction. Rotating the pose pattern matching template according to the edge direction allows the template to be aligned with the image at the corner point of a candidate pose identifier pattern. The edge direction of the corner point of the candidate pose identifier pattern can be used to determine the orientation of the image at that corner point in the positioning image. In some embodiments, rotating the pose pattern matching template according to the edge direction adjusts it to be the same as or nearly the same as the image orientation at the corner point of the candidate pose identifier pattern to facilitate image matching.

[0144] See Figure 12 Step 1205: Starting from the first pose identifier, search for pose identifiers. For example, Figure 14 A flowchart illustrating a method 1400 for searching pose identifiers according to some embodiments of the present disclosure is shown. Figure 14 As shown, some or all of the steps in method 1400 can be executed by a control device (e.g., control device 120) of the execution arm detection system 100. Control device 120 can be configured on a computing device. Method 1400 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1400 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 19 The processor 1920 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium.

[0145] See Figure 14In step 1401, the second pose identifier is searched starting from the first pose identifier. In some embodiments, the corner point of the first pose identifier pattern is used as the starting point, and the corner point of the second pose identifier pattern is searched in a set search direction. In some embodiments, the set search direction may include at least one of the following: directly in front of the corner point of the first pose identifier pattern (corresponding to the 0° angle direction), directly behind (corresponding to the 120° angle direction), directly above (90° angle direction), directly below (-90° angle direction), and diagonally (e.g., ±45° angle direction).

[0146] In some embodiments, the number of search directions is set to n, for example, searching in 8 directions, with each search direction v sn It can be determined based on the following formula (20):

[0147] v sn =[cos(n·π / 4)sin(n·π / 4)], (n=1,2,…,8) (20)

[0148] In some embodiments, the search direction set in the current step can be determined based on the deviation angle between adjacent pose identifier corner points among the multiple pose identifier corner points determined in the previous frame. For example, the predetermined search direction is determined based on the following formula (21):

[0149]

[0150] Among them, (x j ,y j ) represents the two-dimensional coordinates of the corner points of multiple pose marker patterns determined in the previous frame (or the previous image processing cycle); n last The number of corner points of the multiple pose marker patterns determined in the previous frame; v s1 The first set search direction; v s2 This is the second search direction set.

[0151] In some embodiments, such as Figure 15 As shown, the first pose is used to identify the corner point P of the pattern. 1501 Using the coordinates of the given location as the starting point, search for the corner point P of the second pose marker pattern in the set search direction. 1502 The coordinate position can specifically include: identifying the corner point P of the pattern using the first pose. 1501 Using the coordinates as the starting point for the search, the search box (e.g., ...) Figure 15 (The dashed box in the image) moves in the set search direction V with a certain search step size. 1501 Search for the corner points of the pose marker pattern. If there is at least one candidate corner point of the pose marker pattern within the search box, then the candidate corner point with the highest likelihood value within the search box is selected as the second pose marker pattern corner point P.1502 With the search box limited to a suitable size, the first pose is used to identify the corner point P of the pattern. 1501 The coordinates of the point are used as the starting point for the second pose identification pattern corner point P. 1502 During the search, the candidate pose marker corner with the highest corner likelihood value among the candidate pose marker corners appearing in the search box is more likely to be the actual pose marker corner. Therefore, it can be considered that the candidate pose marker corner with the highest corner likelihood value in the search box is the second pose marker corner P. 1502 To improve data processing speed, in other embodiments, to improve the accuracy of pose marker pattern corner point recognition, the candidate pose marker pattern corner point with the highest corner point likelihood value among the candidate pose marker pattern corner points appearing in the search box is selected for corner point recognition to determine whether the candidate pose marker pattern corner point with the highest corner point likelihood value is a pose marker pattern corner point. For example, the pose pattern matching template is matched with the image within a certain range of the candidate pose marker pattern corner point with the highest corner point likelihood value. The candidate pose marker pattern corner point that meets the preset pose pattern matching degree standard can be considered as the searched second pose marker pattern corner point P. 1502 .

[0152] In some embodiments, continue reading Figure 15 The size of the search box can be gradually increased, thereby gradually increasing the search range. The search step size can change synchronously with the side length of the search box. In other embodiments, the size of the search box can also be a fixed size.

[0153] In some embodiments, the pose identification pattern can be a black and white checkerboard pattern, and pattern matching can be performed based on the correlation coefficient in formula (18). If the correlation coefficient is greater than the threshold, the candidate pose identification pattern corner with the largest corner likelihood value is considered to be the pose identification pattern corner, and is denoted as the second pose identification pattern corner.

[0154] Figure 16 A flowchart illustrating a method 1600 for searching a second pose identifier according to some embodiments of the present disclosure is shown. Figure 16 As shown, some or all of the steps in method 1600 can be executed by a control device (e.g., control device 120) of the execution arm detection system 100. Control device 120 can be configured on a computing device. Method 1600 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1600 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 19 The processor 1920 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium. In some embodiments, step 1401 in method 1400 may be implemented similarly to method 1600.

[0155] See Figure 16 In step 1601, starting from the first pose identifier, candidate pose identifier pattern corner points of the second pose identifier are searched. In some embodiments, the search for candidate pose identifier pattern corner points of the second pose identifier can be combined with... Figure 15 The search is shown for the corner point P of the second pose identifier pattern. 1502 Similarly, implement it.

[0156] In step 1603, based on the distribution of multiple pose identifiers, a first pose pattern matching template and a second pose pattern matching template are determined. The first pose pattern matching template and the second pose pattern matching template correspond to pose identifiers adjacent to the first pose identifier. In some embodiments, step 1603 can be performed before or after step 1601, or step 1603 can be performed synchronously with step 1601. In some embodiments, the pose identifier patterns included in the pose identifiers included in the first pose identifier and the distribution of multiple pose identifier patterns can be used to determine the pose identifier patterns included in the pose identifiers adjacent to the first pose identifier, thereby determining the first pose pattern matching template and the second pose pattern matching template.

[0157] In step 1605, the first pose pattern matching template and / or the second pose pattern matching template are matched with the patterns at the corner positions of the candidate pose identifier patterns of the second pose identifier to identify the second pose identifier. In some embodiments, the first pose pattern matching template and / or the second pose pattern matching template can be matched with the patterns at the corner positions of the candidate pose identifier patterns of the second pose identifier based on the correlation coefficient in formula (18). If the correlation coefficient is greater than a threshold, the corner points of the candidate pose identifier patterns of the second pose identifier are determined as the corner points of the pose identifier patterns of the second pose identifier, and the patterns corresponding to the pose pattern matching templates (first pose pattern matching template or second pose pattern matching template) with a correlation coefficient greater than the threshold are determined as the pose identifier patterns of the second pose identifier.

[0158] See Figure 14 In step 1403, a search direction is determined based on the first pose identifier and the second pose identifier. In some embodiments, the search direction includes a first search direction and a second search direction. The first search direction may be a direction starting from the coordinate position of a corner point of the first pose identifier pattern and moving away from the corner point of the second pose identifier pattern. The second search direction may be a direction starting from the coordinate position of a corner point of the second pose identifier pattern and moving away from the corner point of the first pose identifier pattern. For example, Figure 15 The search direction V shown 1502 .

[0159] See Figure 14In step 1405, starting with either the first pose identifier or the second pose identifier, a search for pose identifiers is performed in the search direction. In some embodiments, if the first pose identifier pattern corner point is used as the new starting point, the first search direction described above can be used as the search direction for the pose identifier pattern corner point. If the second pose identifier pattern corner point is used as the new starting point, the second search direction described above can be used as the search direction for the pose identifier pattern corner point. In some embodiments, a new pose identifier pattern corner point is searched (e.g., Figure 15 The third pose marker pattern corner point P in 1503 This can be performed similarly to step 1401 in method 1400 or method 1700. In some embodiments, the search step size can be the first pose identifier pattern corner point P. 1501 Second pose identifier pattern corner point P 1502 The distance between them is L1.

[0160] Figure 17 A flowchart illustrating a method 1700 for searching pose identifiers according to some embodiments of the present disclosure is shown. Figure 17 As shown, some or all of the steps in method 1700 can be executed by a control device (e.g., control device 120) of the execution arm detection system 100. Control device 120 can be configured on a computing device. Method 1700 can be implemented by software, firmware, and / or hardware. In some embodiments, method 1700 can be implemented as computer-readable instructions. These instructions can be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 19 The processor 1920 shown reads and executes these instructions. In some embodiments, these instructions may be stored on a computer-readable medium. In some embodiments, step 1405 in method 1400 may be implemented similarly to method 1700.

[0161] See Figure 17 In step 1701, starting from the first pose identifier or the second pose identifier, candidate pose identifier pattern corner points of the third pose identifier are searched. In some embodiments, the search for candidate pose identifier pattern corner points of the third pose identifier can be combined with... Figure 15 The search is shown for the corner point P of the third pose identifier pattern. 1503 Similarly, implement it.

[0162] In step 1703, a third pose pattern matching template is determined based on the distribution of multiple pose identifiers. The third pose pattern matching template corresponds to a pose identifier adjacent to the first pose identifier or adjacent to the second pose identifier. In some embodiments, the pose identifier pattern included in the pose identifier pattern included in the first pose identifier or the second pose identifier and the distribution of multiple pose identifier patterns can be used to determine the pose identifier pattern included in the pose identifier adjacent to the first pose identifier or the second pose identifier, thereby determining the third pose pattern matching template.

[0163] In step 1705, the third pose pattern matching template is matched with the pattern at the corner position of the candidate pose identifier pattern of the third pose identifier to identify the third pose identifier. In some embodiments, step 1705 can be implemented similarly to step 1605.

[0164] In some embodiments, in response to a search distance greater than a search distance threshold, the pixel with the largest corner likelihood value among the remaining pixels in the pixel set is determined as a candidate pose identifier pattern corner point; and multiple different pose pattern matching templates are matched with the patterns at the corner point positions of the candidate pose identifier pattern to identify the first pose identifier. In some embodiments, after determining the pixel with the largest corner likelihood value among the remaining pixels in the pixel set as a new candidate pose identifier pattern corner point, a new first pose identifier can be identified based on a method similar to step 1203. In some embodiments, a search distance greater than a search distance threshold can be understood as a search distance greater than a search distance threshold in some or all search directions. In some embodiments, the search distance threshold may include a set multiple of the distance between the (N-1)th pose identifier pattern corner point and the (N-2)th pose identifier pattern corner point, where N≥3.

[0165] For example, the search distance threshold is twice the distance between the first two pose marker corner points. Thus, the maximum search distance for the third pose marker corner point is twice the distance between the first and second pose marker corner points. If no pose marker corner point is found within this search distance in the search direction, the pixel with the highest corner likelihood value among the remaining pixels in the pixel set is determined as a new candidate pose marker corner point, and a new first pose marker is identified. The current search process then stops. In some embodiments, similar to method 1200, a new first pose marker corner point can be determined, and similar to method 1400, the remaining pose marker corner points can be searched starting from this new corner point.

[0166] In some embodiments, in response to the number of identified pose marker pattern corner points being greater than or equal to a pose marker number threshold, the pose of the end effector relative to the reference coordinate system can be determined based on the search for the pose markers, and the search for pose marker pattern corner points will stop accordingly. For example, when four pose marker pattern corner points are identified, the search for pose marker pattern corner points will stop.

[0167] In some embodiments, in response to the number of identified pose identifiers being less than a pose identifier number threshold, the pixel with the highest corner likelihood value among the remaining pixels in the pixel set is determined as a candidate pose identifier pattern corner point; and multiple different pose pattern matching templates are matched with the patterns at the corner point positions of the candidate pose identifier patterns to identify the first pose identifier. In some embodiments, if the total number of identified pose identifiers (e.g., pose identifier pattern corner points) is less than a set pose identifier number threshold, the search based on the first pose identifier in the above steps is considered to have failed. In some embodiments, in the case of search failure, the pixel with the highest corner likelihood value among the remaining pixels in the pixel set is determined as a new candidate pose identifier pattern corner point, and then a new first pose identifier can be identified based on a method similar to step 1203. In some embodiments, similar to method 1200, a new first pose identifier pattern corner point can be re-determined, and similar to method 1400, the remaining pose identifier pattern corner points can be searched starting from the new pose identifier pattern corner point.

[0168] In some embodiments, after the corner points of the pose marker pattern are searched or identified, sub-pixel positioning can be performed on the determined corner points of the pose marker pattern to improve the positional accuracy of the corner points of the pose marker pattern.

[0169] In some embodiments, the CL values ​​of pixels can be fitted based on a model to determine the coordinates of the corner points of the pose identifier pattern after subpixel localization. For example, the fitting function for the CL value of each pixel in the ROI can be a quadratic surface function, the extreme points of which are subpixel points. The fitting function can be determined based on the following formulas (22) and (23):

[0170] S(x,y)=ax 2 +by 2 +cx+dy+exy+f (22)

[0171]

[0172] Where S(x, y) is the fitting function for the CL values ​​of all pixels in each ROI, and a, b, c, d, e, and f are coefficients; x c The x-coordinate and y-coordinate of the pose identifier c The y-coordinate is the pose identifier.

[0173] In some embodiments of this disclosure, a computer device is also provided, including a memory and a processor. The memory may be used to store at least one instruction, and the processor is coupled to the memory for executing the at least one instruction to perform some or all of the steps in the method of this disclosure, such as... Figures 6-12 , Figure 14 , Figure 16 and Figure 17 Some or all of the steps in the method disclosed herein.

[0174] Figure 18 A schematic block diagram of a computer device 1800 according to some embodiments of the present disclosure is shown. See also Figure 18 The computer device 1800 may include a central processing unit (CPU) 1801, a system memory 1804 including random access memory (RAM) 1802 and read-only memory (ROM) 1803, and a system bus 1805 connecting the various components. The computer device 1800 may also include an input / output system and a mass storage device 1807 for storing an operating system 1813, application programs 1814, and other program modules 1815. The input / output devices include an input / output controller 1810, primarily composed of a display 1808 and input devices 1809.

[0175] Mass storage device 1807 is connected to central processing unit 1801 via a mass storage controller (not shown) connected to system bus 1805. Mass storage device 1807 or computer-readable media provides non-volatile storage for computer devices. Mass storage device 1807 may include computer-readable media (not shown) such as hard disk or compact disc read-only memory (CD-ROM) drives.

[0176] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, flash memory or other solid-state storage technologies, CD-ROM, or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The aforementioned system memories and mass storage devices can be collectively referred to as memory.

[0177] Computer device 1800 can be connected to network 1812 via network interface unit 1811 connected to system bus 1805.

[0178] The system memory 1804 or mass storage device 1807 is also used to store one or more instructions. The central processing unit 1801 implements all or part of the steps of the methods in some embodiments of this disclosure by executing the one or more instructions.

[0179] In some embodiments of this disclosure, a computer-readable storage medium is also provided, storing at least one instruction that is executed by a processor to cause a computer to perform some or all of the steps in the methods of some embodiments of this disclosure, such as... Figures 6-12 , Figure 14 , Figure 16 and Figure 17 Some or all of the steps in the disclosed method. Examples of computer-readable storage media include memory for computer programs (instructions), such as read-only memory (ROM), random access memory (RAM), compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage devices.

[0180] Figure 19 A schematic diagram of a robot system 1900 according to some embodiments of the present disclosure is shown. In some embodiments of the present disclosure, see [reference needed]. Figure 19 The robot system 1900 may include a tool 1950, a drive unit 1960, an image acquisition device 1910, and a control device (e.g., a processor 1920). The tool 1950 may include an actuator arm 1940 and an end effector 1930 disposed at the distal end of the actuator arm 1940. Multiple pose markers may be formed or disposed on the end effector 1930, the multiple pose markers including different pose marker patterns. An actuator may be disposed at the distal end of the end effector 1930. The drive unit 1960 may be used to control the pose of the actuator arm 1940 and its end effector 1930. The image acquisition device 1910 may be used to acquire positioning images of the actuator arm 1940. In some embodiments, see... Figure 19 The robot system 1900 may further include an input device 1970a and a signal generation unit 1970b. The input device 1970a can be used to receive the target pose, motion trajectory, or drive signal controlling the target pose of the end effector 1930 input by a user operation. The signal generation unit 1970b can be used to randomly generate the target pose, motion trajectory, or drive signal controlling the target pose of the end effector 1930. The processor 1920 is connected to the input device 1970a, the signal generation unit 1970b, the drive device 1960, and the image acquisition device 1910, and is used to execute some or all of the steps in the methods of some embodiments of this disclosure, such as... Figures 6-12 , Figure 14 , Figure 16 and Figure 17 Some or all of the steps in the method disclosed herein.

[0181] Note that the above are merely exemplary embodiments and technical principles of this disclosure. Those skilled in the art will understand that this disclosure is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this disclosure. Therefore, although this disclosure has been described in detail through the above embodiments, this disclosure is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this disclosure, the scope of which is determined by the scope of the appended claims.

Claims

1. A method for detecting an actuator arm, characterized in that, include: Determine a drive signal for controlling the target pose of the end effector arm, the drive signal corresponding to the target pose; Obtain the positioning image of the actuator arm; In the positioning image, multiple pose markers located on the end of the actuator arm are identified, and the multiple pose markers include different pose marker patterns; Based on the multiple pose identifiers, the actual pose of the end effector of the actuator arm is determined; as well as The performance of the actuator arm is determined based on the target pose and the actual pose. The drive signal used to control the target pose of the end of the actuator arm includes either a pre-set or randomly generated drive signal. Based on the multiple pose identifiers, determining the actual pose of the end effector of the actuator arm includes: Determine the three-dimensional coordinates of the plurality of pose markers relative to the end effector coordinate system of the actuator arm; Determine the two-dimensional coordinates of the plurality of pose markers in the positioning image; and Based on the two-dimensional coordinates of the multiple pose markers in the positioning image and the three-dimensional coordinates of the multiple pose markers relative to the end coordinate system of the actuator arm, the pose of the end coordinate system of the actuator arm relative to the reference coordinate system is determined as the actual pose. Determining the three-dimensional coordinates of the plurality of pose markers relative to the end effector coordinate system of the actuator arm includes: Based on the distribution of the plurality of pose markers, the axial angles of the plurality of pose markers relative to the Z-axis of the end-effector coordinate system of the actuator arm are determined, wherein the Z-axis of the end-effector coordinate system of the actuator arm is along the tangent direction of the end of the actuator arm; and Based on the axial angles of the plurality of pose markers, the three-dimensional coordinates of the plurality of pose markers relative to the end effector coordinate system of the actuator are determined.

2. The method according to claim 1, characterized in that, The method for determining the drive signal used to control the target pose of the end effector arm is replaced by: The target pose of the end effector of the actuator is preset or randomly generated; and The driving signal is determined based on the target pose.

3. The method according to claim 1, characterized in that, The method for determining the drive signal used to control the target pose of the end effector arm is replaced by: The motion trajectory of the end effector of the actuator arm is preset, and the motion trajectory includes multiple target poses; as well as Determine the plurality of driving signals corresponding to the plurality of target poses.

4. The method according to claim 3, characterized in that, Determining the actual pose of the end effector of the actuator arm includes: determining multiple actual poses of the end effector of the actuator arm corresponding to multiple drive signals at a predetermined detection cycle. Determining the performance of the actuator based on the target pose and the actual pose includes: determining the performance of the actuator based on multiple target poses and multiple actual poses.

5. The method according to claim 1, characterized in that, The method for determining the drive signal used to control the target pose of the end effector arm is replaced by: The target joint parameter set of the actuator arm joints is preset or randomly generated; and Based on the target joint parameter set, the driving signal and the corresponding target pose are determined.

6. The method according to claim 5, characterized in that, In response to at least one target joint parameter in the randomly generated set of target joint parameters exceeding the joint parameter range, the at least one target joint parameter is updated to the limit value of the corresponding joint parameter range; as well as Based on the updated target joint parameter set, the driving signal and the corresponding target pose are determined.

7. The method according to claim 5, characterized in that, The actuator includes: At least one component, the component including a fixing plate and multiple structural bones, a first end of the multiple structural bones being fixedly connected to the fixing plate, and a second end of the multiple structural bones being used to connect to a driving device; The method further includes: Based on the target joint parameter set, the driving force of the multiple structural bones is determined; and The driving signal is determined based on the driving amount of the multiple structural bones.

8. The method according to claim 1, characterized in that, Also includes: Based on the pre-set drive signal, the target pose of the end effector of the actuator is determined through the mapping relationship between the drive signal and the target pose.

9. The method according to claim 5, characterized in that, Also includes: Based on the pre-set target joint parameter set, the target pose of the end effector of the actuator arm is determined through the mapping relationship between the target joint parameter set and the target pose.

10. The method according to claim 1, characterized in that, Also includes: Multiple candidate pose identifiers are determined from the positioning image; Based on multiple different pose pattern matching templates, the first pose identifier is identified from the multiple candidate pose identifiers; as well as Starting from the first pose identifier, search for pose identifiers.

11. The method according to claim 10, characterized in that, The pose identifier includes corner points of the pose identifier pattern within the pose identifier pattern; the method includes: Determine the region of interest in the localization image; The region of interest is divided into multiple sub-regions; The pixel with the largest corner likelihood value in each sub-region is determined to form a pixel set; The pixel with the highest corner likelihood value in the pixel set is selected as the candidate corner point of the pose identifier pattern; and The multiple different pose pattern matching templates are matched with the patterns at the corner positions of the candidate pose identifier pattern to identify the first pose identifier.

12. The method according to claim 11, characterized in that, include: In response to a matching failure, the pixel with the highest corner likelihood value among the remaining pixels in the pixel set is updated as the corner point of the candidate pose identifier pattern; as well as The multiple different pose pattern matching templates are matched with the patterns at the corner positions of the updated candidate pose identifier pattern to identify the first pose identifier.

13. The method according to claim 11 or 12, characterized in that, Also includes: Starting from the first pose identifier, search for the second pose identifier; Based on the first pose identifier and the second pose identifier, the search direction is determined; as well as Starting from the first pose identifier or the second pose identifier, search for pose identifiers in the search direction.

14. The method according to claim 13, characterized in that, Searching for the second pose identifier starting from the first pose identifier includes: Starting from the first pose identifier, search for the corner points of the candidate pose identifier pattern of the second pose identifier; Based on the distribution of the multiple pose identifiers, a first pose pattern matching template and a second pose pattern matching template are determined, wherein the first pose pattern matching template and the second pose pattern matching template correspond to the pose identifiers adjacent to the first pose identifier; and The first pose pattern matching template and / or the second pose pattern matching template are matched with the pattern at the corner position of the candidate pose identifier pattern of the second pose identifier to identify the second pose identifier.

15. The method according to claim 13, characterized in that, Searching for a pose identifier in the search direction, using the first pose identifier or the second pose identifier as the starting point, includes: Starting from the first pose identifier or the second pose identifier, search for candidate pose identifier pattern corner points of the third pose identifier; Based on the distribution of the plurality of pose identifiers, a third pose pattern matching template is determined, wherein the third pose pattern matching template corresponds to a pose identifier adjacent to the first pose identifier or adjacent to the second pose identifier; and The third pose pattern matching template is matched with the pattern at the corner position of the candidate pose identifier pattern of the third pose identifier to identify the third pose identifier.

16. The method according to claim 13, characterized in that, Also includes: In response to a search distance greater than a search distance threshold, the pixel with the largest corner likelihood value among the remaining pixels in the pixel set is determined as a candidate pose identifier pattern corner point; as well as The multiple different pose pattern matching templates are matched with the patterns at the corner positions of the candidate pose identifier patterns to re-identify the first pose identifier.

17. The method according to claim 13, characterized in that, Also includes: In response to the fact that the number of identified pose markers is less than the pose marker number threshold, the pixel with the largest corner likelihood value among the remaining pixels in the pixel set is determined as the candidate pose marker pattern corner point. as well as The multiple different pose pattern matching templates are matched with the patterns at the corner positions of the candidate pose identifier patterns to re-identify the first pose identifier.

18. The method according to any one of claims 10-12, characterized in that, The method for determining the three-dimensional coordinates of the plurality of pose markers relative to the end effector coordinate system of the actuator arm is replaced by: The arrangement order of the plurality of pose identifiers is determined based on at least two of them; as well as Based on the arrangement order of the multiple pose markers, the three-dimensional coordinates of the multiple pose markers relative to the end effector coordinate system of the actuator arm are determined.

19. The method according to any one of claims 1, 9-12, characterized in that, The plurality of pose markers are disposed on the outer surface of the columnar portion at the end of the actuator arm.

20. The method according to claim 1, characterized in that, Also includes: The actual pose of the end of the actuator arm is determined by a predetermined detection cycle, so that the performance of the actuator arm can be determined in real time or cumulatively through multiple detection cycles.

21. The method according to claim 1, characterized in that, Determining the performance of the actuator arm based on the target pose and the actual pose includes: In response to the target pose and the actual pose reaching an error condition, a performance signal is generated, the performance signal indicating that the performance of the actuator arm is unqualified.

22. A computer device, comprising: Memory, used to store at least one instruction; as well as A processor, coupled to the memory and configured to execute the at least one instruction to perform the arm detection method according to any one of claims 1-21.

23. A computer-readable storage medium for storing at least one instruction, which, when executed by a computer, causes the computer to perform the arm detection method according to any one of claims 1-21.

24. A robot system, comprising: An actuator arm, the end of which is provided with multiple pose markers, the multiple pose markers including different pose marker patterns; At least one drive device for driving the actuator arm; An image acquisition device is used to acquire positioning images of the actuator arm; as well as A control device is configured to connect to the at least one drive device and the image acquisition device to perform the arm detection method according to any one of claims 1-2 and 5-21.

25. A robot system, comprising: An actuator arm, the end of which is provided with multiple pose markers, the multiple pose markers including different pose marker patterns; At least one drive device for driving the actuator arm; An image acquisition device is used to acquire positioning images of the actuator arm; as well as A control device is configured to be connected to the at least one drive device and the image acquisition device to perform the arm detection method according to claim 3 or 4.

26. The robot system according to claim 24, characterized in that, Also includes: An input device, communicatively connected to the control device, is used to receive input drive signals or target poses; and / or The signal generation unit is communicatively connected to the control device and is used to randomly generate drive signals or target poses.

27. The robot system according to claim 25, characterized in that, Also includes: An input device, which is communicatively connected to the control device, is used to receive input drive signals, target poses, or the motion trajectory of the end effector of the actuator arm. and / or The signal generation unit is communicatively connected to the control device and is used to randomly generate drive signals, target poses, or motion trajectories of the end effector of the actuator arm.

Images