Error Detection Method Based on Composite Identification and Robot System

Abstract

The present disclosure relates to the technical field of error detection, and discloses an error detection method. The error detection method includes: obtaining a target posture of an end of an actuator arm; acquiring a positioning image; identifying multiple identifiers located on the end of the actuator arm in the positioning image, the multiple identifiers including multiple posture identifiers for identifying postures and at least one composite identifier for identifying postures and angles; determining the actual posture of the end of the actuator arm based on the at least one composite identifier and the multiple posture identifiers; and generating a control signal related to a fault in response to the target posture and the actual posture satisfying an error detection condition.

Description

Technical Field

[0001] The present disclosure relates to the technical field of error detection, and in particular to an error detection method and a robot system based on composite identification. Background Art

[0002] Typically, a robotic system used for teleoperation includes an actuator arm for performing operations and a master manipulator for controlling the arm's motion. In practical scenarios, the actuator arm is configured to enter an operating area. The operator controls the arm's movement within the operating area by teleoperating the master manipulator, and then performs operations through the actuator at the end of the arm. This type of robot achieves motion control of the arm through the master manipulator through motion conversion between the master manipulator and the actuator arm.

[0003] Robots require high operational precision and user-friendly human-machine interaction. During teleoperation, real-time detection of the actuator arm's posture error is necessary to determine whether the actuator arm has correctly moved to the position and posture corresponding to the operator's desired operation of the master manipulator, thereby providing real-time insights into the robot system's operating status. Summary of the Invention

[0004] In some embodiments, the present disclosure provides an error detection method. The method may include: obtaining a target pose of an end of an actuator arm; acquiring a positioning image; identifying, in the positioning image, multiple identifiers located on the end of the actuator arm, the multiple identifiers including multiple pose identifiers for identifying poses and at least one composite identifier for identifying poses and angles; determining an actual pose of the end of the actuator arm based on the at least one composite identifier and the multiple pose identifiers; and generating a control signal associated with a fault in response to the target pose and the actual pose satisfying an error detection condition.

[0005] In some embodiments, the present disclosure provides a computer device, comprising: a memory for storing at least one instruction; and a processor, coupled to the memory and for executing the at least one instruction to perform any method in some embodiments of the present disclosure.

[0006] In some embodiments, the present disclosure provides a computer-readable storage medium for storing at least one instruction, which, when executed by a computer, causes the computer to perform any one of the methods in some embodiments of the present disclosure.

[0007] In some embodiments, the present disclosure provides a robot system, comprising: a main manipulator, including a robotic arm, a handle provided on the robotic arm, and at least one main manipulator sensor provided at at least one joint of the robotic arm, wherein the at least one main manipulator sensor is used to obtain joint information of at least one joint; an actuator arm, wherein a plurality of identifiers are provided at the end of the actuator arm, and the plurality of identifiers include a plurality of posture identifiers and at least one composite identifier; at least one driving device for driving the actuator arm; at least one driving device sensor, coupled to the at least one driving device and used to obtain status information of the at least one driving device; an image acquisition device, for acquiring a positioning image of the actuator arm; and a control device, configured to be connected to the main manipulator, at least one driving device, and at least one driving device sensor image acquisition device to execute any one of the methods in some embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A schematic structural diagram of a robot system according to some embodiments of the present disclosure is shown;

[0009] Figure 2 A schematic diagram illustrating an error detection system according to some embodiments of the present disclosure is shown;

[0010] Figure 3 A flowchart showing an error detection method according to some embodiments of the present disclosure is shown;

[0011] Figure 4 A flowchart illustrating a method for determining a target pose of an end portion of an actuator arm according to some embodiments of the present disclosure is provided;

[0012] Figure 5 A schematic diagram of a coordinate system in master-slave motion mapping according to some embodiments of the present disclosure is shown;

[0013] Figure 6 A schematic diagram showing a tag including multiple identifiers according to some embodiments of the present disclosure;

[0014] Figure 7 A schematic diagram showing a cylindrical label disposed on the peripheral side of the end of an actuator arm according to some embodiments of the present disclosure;

[0015] Figure 8 A schematic diagram illustrating implementation scenarios according to some embodiments of the present disclosure;

[0016] Figure 9 A flowchart illustrating a method for determining an actual pose of an end portion of an actuator arm according to some embodiments of the present disclosure;

[0017] Figure 10 A flowchart illustrating a method for determining the actual posture of the end of an actuator arm according to other embodiments of the present disclosure;

[0018] Figure 11 A flowchart illustrating a method for identifying an identifier according to some embodiments of the present disclosure is shown;

[0019] Figure 12 A schematic diagram illustrating a posture identification pattern according to some embodiments of the present disclosure;

[0020] Figure 13 A flowchart illustrating a method for searching for identification according to some embodiments of the present disclosure is shown;

[0021] Figure 14 A schematic diagram illustrating a search identifier according to some embodiments of the present disclosure;

[0022] Figure 15 A schematic block diagram showing a computer device according to some embodiments of the present disclosure;

[0023] Figure 16 A schematic diagram of a robotic system according to some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0024] To make the technical problems solved, the technical solutions adopted, and the technical effects achieved by the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. It should be understood by those skilled in the art that the described embodiments should be considered as illustrative and non-restrictive in all aspects, and are merely exemplary embodiments of the present disclosure, not all embodiments.

[0025] In the description of this disclosure, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of this disclosure. They do not indicate or imply that the devices or components referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limitations on this disclosure. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0026] In the description disclosed in the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", and "coupled" should be understood in a broad sense. For example, it can be a fixed connection or a detachable connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection between the two components. For ordinary technicians in this field, the specific meanings of the above terms in the disclosure of the present invention can be understood according to the specific circumstances. In the disclosure of the present invention, the end close to the operator is defined as the proximal end, the near part or the rear end, the rear part, and the end close to the work object is defined as the distal end, the end, the far end or the front end, the front part. Those skilled in the art will understand that the embodiments of the present disclosure can be applicable to actuator arms provided on mechanical devices operating in a variety of environments, including but not limited to the surface, underground, underwater, in space, and in living organisms.

[0027] In the present disclosure, the term "position" refers to the positioning of an object or a part of an object in three-dimensional space (for example, three translational degrees of freedom can be described using changes in Cartesian X, Y, and Z coordinates, such as three translational degrees of freedom along the Cartesian X axis, Y axis, and Z axis, respectively). In the present disclosure, the term "posture" refers to the rotational setting of an object or a part of an object (for example, three rotational degrees of freedom, which can be described using roll, pitch, and yaw). In the present disclosure, the term "pose" refers to the combination of the position and pose of an object or a part of an object, for example, it can be described using six parameters of the six degrees of freedom mentioned above. In the present disclosure, the pose of the handle of the main manipulator can be represented by a set of joint information of the main manipulator joints (for example, a one-dimensional matrix composed of these joint information). The pose of the actuator arm can be determined by the drive information of the actuator arm. In the present disclosure, the joint information of the joint can include the angle of rotation of the corresponding joint relative to the corresponding joint axis or the distance moved relative to the initial position.

[0028] In this disclosure, a reference coordinate system can be understood as a coordinate system that can describe the position and posture of an object. Depending on the actual positioning requirements, the reference coordinate system can be selected to have the origin of a virtual reference object or the origin of a physical reference object as the coordinate system origin. In some embodiments, the reference coordinate system can be a world coordinate system, a coordinate system of the space where the master manipulator, actuator arm, or camera is located, or a coordinate system perceived by the operator, etc.

[0029] In this disclosure, an object can be understood as an object or target to be positioned, such as an actuator arm or the end of an actuator arm. The pose of an actuator arm or a portion thereof (such as the end thereof) can refer to the pose of a coordinate system defined by the actuator arm or a portion thereof relative to a reference coordinate system.

[0030] Figure 1 FIG. 1 shows a schematic diagram of a robot system 100 according to some embodiments of the present disclosure. Figure 1 As shown, the robotic system 100 may include a master trolley 110, a slave trolley 130, and a control device 120. The control device 120 may be communicatively connected to the master trolley 110 and the slave trolley 130, for example, by a cable connection or a wireless connection, to achieve communication between the master trolley 110 and the slave trolley 130. The master trolley 110 includes a master manipulator for remote operation by an operator, and the slave trolley 130 includes at least one actuator arm for performing an operation. The control device 120 implements a master-slave mapping between the master manipulator in the master trolley 110 and the actuator arm in the slave trolley 130, thereby enabling the master manipulator to control the motion of the actuator arm. In some embodiments, the actuator arm is configured to enter an operating area through a tube sleeve, sheath, etc., wherein the tube sleeve, sheath, etc. can be fixed to an opening (e.g., an artificial opening or a natural opening) formed on a wall, an animal body, etc., and the operating area can be the area where the operation is performed. The actuator arm may be a continuous deformable arm, and an end device (e.g., an actuator) may be provided at the distal end of the actuator arm. The end device may include, but is not limited to, an excavation device, an underwater operation device, a sorting device, a surgical instrument, etc. It will be appreciated by those skilled in the art that the master trolley 110 and the slave trolley 130 may adopt other structures or forms, such as a base, a bracket, or a building.

[0031] In some embodiments, in addition to working tools, the actuator arm can also be used as a visual tool. The end-use instrument of the visual tool may include, but is not limited to, an image acquisition device or a lighting device. In some embodiments, the master control vehicle may include a master manipulator and a display for displaying an image of the operating area. The image acquisition device can be used to capture images of the operating area and transmit the captured images to the slave vehicle. The images are processed by a video processing module in the slave vehicle and then displayed on the slave vehicle's display. The operator obtains the posture of the end of the actuator arm relative to the reference coordinate system in real time through the images on the display. The posture of the master manipulator relative to the reference coordinate system is the posture actually perceived by the operator. The posture changes perceived by the operator through teleoperation of the master manipulator and the posture changes of the end of the actuator arm perceived by the operator on the display conform to a preset posture relationship. Thus, by remotely controlling the master manipulator, the posture transformation of the master manipulator is converted into posture changes of the end of the actuator arm based on the preset posture relationship, thereby achieving posture control of the end of the actuator arm. In this way, when the operator holds the handle of the main manipulator and moves it to operate the actuator arm, based on the principle of intuitive operation, the posture change of the end instrument of the actuator arm felt by the operator is consistent with the posture change of the main manipulator felt by the operator, which helps to improve the operator's remote operation experience and remote operation accuracy.

[0032] During teleoperation, the actuator arm may sometimes fail to accurately move to the position and posture corresponding to the operator's desired operation of the master manipulator. In the present disclosure, the posture error of the actuator arm is detected during teleoperation to determine whether the actuator arm has correctly moved according to the operator's intention, thereby providing real-time information on the operating status of the robotic system. Those skilled in the art will appreciate that the posture error detection methods according to some embodiments of the present disclosure can also be performed during non-teleoperation.

[0033] Figure 2 FIG. 2 is a schematic diagram of an error detection system 200 according to some embodiments of the present disclosure. Figure 2 As shown, the system 200 may include a main manipulator 210, a control device 220, an actuator arm 230, and an image acquisition device 250. The actuator arm 230 may be implemented as a deformable arm or a rigid arm. In some embodiments, the actuator arm 230 may include an actuator arm end 231 at the distal end or distal end, and an end instrument 240 may be provided on the actuator arm end 231.

[0034] The control device 220 can be in communication with at least one drive device, sending drive information to the drive device to control the movement of the actuator arm 230 so that the actuator arm end 231 moves to a desired position and posture. For example, the at least one drive device that controls the movement of the actuator arm 230 can be a servo motor that can receive instructions from the control device to control the movement of the actuator arm 230. In some embodiments, the control device 220 can determine the target posture of the actuator arm end 231 based on the posture of the main manipulator 210 and the mapping relationship between the main manipulator 210 and the actuator arm 230.

[0035] The image acquisition device 250 is in communication with the control device 220. In some embodiments, the image acquisition device 250 can be used to acquire a positioning image, and the image acquisition device 250 can include but is not limited to a dual-lens image acquisition device or a single-lens image acquisition device, such as a binocular or monocular camera. The positioning image may include an image of part or all of the execution arm 230 located in the operating area. In some embodiments, the image acquisition device 250 can be used to acquire an image of the end 231 of the execution arm. A plurality of identifiers can be provided on the end 231 of the execution arm, and these identifiers include an identifier pattern and a pattern corner point. For example, a positioning tag 232 (the positioning tag 232 can be, for example, Figure 6 The positioning tag 232 may include multiple identifiers, including multiple posture identifiers for identifying postures and at least one composite identifier for identifying posture and angle (described in detail below).

[0036] like Figure 2As shown, if the actuator arm end 231 is within the observation field of view 251 of the image acquisition device 250, the captured positioning image may include an image of the actuator arm end 231. Depending on different application scenarios, the image acquisition device 250 may be an industrial camera, an underwater camera, a microelectronic camera, an endoscopic camera, etc. In some embodiments, the image acquisition device 250 may be fixed or variable in position, for example, an industrial camera fixed at a monitoring location or an endoscopic camera with adjustable position or posture. In some embodiments, the image acquisition device 250 may implement at least one of visible light imaging, infrared imaging, CT (Computed Tomography) imaging, and acoustic imaging. Depending on the type of image to be acquired, those skilled in the art may select different image acquisition devices as the image acquisition device 250.

[0037] In some embodiments, the control device 220 may receive a positioning image from the image acquisition device 250 and process the positioning image. For example, the control device 220 may identify multiple markers located on the actuator arm end 231 in the positioning image and determine the position and posture of the actuator arm end 231 relative to a reference coordinate system (e.g., a world coordinate system) as the actual position and posture of the actuator arm end 231.

[0038] In the present disclosure, the control device 220 can perform error detection on the actuator arm 230 based on the target posture and actual posture of the actuator arm end 231, determine whether the actuator arm end 231 has accurately moved to the position and posture corresponding to the operation of the main manipulator 210, and then determine whether the actuator arm 230 has failed and generate a corresponding control signal. In some embodiments, the control device 220 can also determine the target posture and actual posture of the actuator arm end 231 at a predetermined period, so as to perform error detection on the actuator arm 230 in real time through multiple detection cycles. Those skilled in the art should understand that the system 200 can be applied to special or general robot systems in multiple fields (for example, medical, industrial manufacturing, etc.), for example Figure 1 The robotic system 100 shown, or Figure 16 The robot system 1600 is shown. As an example, the system 200 can be applied to a robot system such as a surgical robot, and the end instrument 240 provided at the distal end of the end of the execution arm 231 can be, for example, a surgical implement.

[0039] Some embodiments of the present disclosure provide an error detection method for an actuator arm of a robotic system. Figure 3 FIG2 is a flow chart illustrating an error detection method 300 (hereinafter also referred to as “method 300”) according to some embodiments of the present disclosure. The method 300 may be implemented or executed by hardware, software, or firmware. In some embodiments, the method 300 may be performed by a robotic system (e.g., Figure 1 The robotic system 100 shown, or Figure 16 In some embodiments, the method 300 may be implemented as computer-readable instructions. These instructions may be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 1 The control device 120 shown, Figure 2 The control device 220 shown or Figure 16 The instructions may be read and executed by a control device 1620 as shown. For example, a control device for a robotic system may include a processor configured to execute method 300. In some embodiments, these instructions may be stored on a computer-readable medium.

[0040] See Figure 3 In step 301, the target posture of the end of the actuator arm is obtained. In some embodiments, the target posture of the end of the actuator arm can be determined according to the posture of the master manipulator based on the master-slave motion mapping relationship between the posture of the master manipulator and the posture of the end of the actuator arm. An exemplary method for obtaining the target posture of the end of the actuator arm includes the following steps: Figure 4 The method shown.

[0041] Figure 4 FIG. 4 is a flow chart showing a method 400 for determining a target pose of an end of an actuator arm according to some embodiments of the present disclosure. Figure 4 As shown, some or all of the steps in the method 400 may be performed by a control device (e.g., Figure 1 The control device 120 shown, Figure 2 The control device 220 shown or Figure 16 Some or all of the steps in method 400 may be implemented by software, firmware, and / or hardware. In some embodiments, method 400 may be performed by a robotic system (e.g., Figure 1 The robotic system 100 or Figure 16 In some embodiments, the method 400 may be implemented as computer-readable instructions. These instructions may be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 16 In some embodiments, the instructions may be stored on a computer readable medium.

[0042] Figure 5 Schematic diagram of coordinate system in master-slave motion mapping according to some embodiments of the present disclosure is shown. Figure 5 The definitions of the coordinate systems in are as follows: The base coordinate system of the actuator arm {Tb}, the origin of which is located at the base of the actuator arm or the exit of the sheath, It is consistent with the extension line of the base or the axial direction of the sheath, Direction Figure 5 As shown. The camera coordinate system {lens}, the origin is located at the center of the camera, the direction of the camera axis is Direction, after the field of vision is adjusted, the upper part is Direction. The end coordinate system {wm} of the actuator arm has its origin at the end of the actuator arm. In line with the axis direction of the end, Direction Figure 5 The reference coordinate system {w} can be the coordinate system of the space where the main manipulator, the actuator arm or the camera is located, such as the actuator arm base coordinate system {Tb}, or the world coordinate system, such as Figure 5 In some embodiments, the operator's body feeling can be used as a reference. When the operator sits in front of the main console, the body feeling upward is Direction, the forward direction of the body is The display coordinate system {Screen} has its origin at the center of the display and its vertical direction is Positive direction, the top of the screen is Direction. The main manipulator base coordinate system {CombX}, the coordinate axis direction is as follows Figure 5 The handle coordinate system {H} of the main manipulator has the following coordinate axis directions: Figure 5 shown.

[0043] Below is Figure 5 The coordinate system shown is used as an example to describe the method 400 for determining the target position of the end of the actuator arm. However, those skilled in the art will appreciate that other coordinate systems can be used to define and implement the method 400 for determining the target position of the end of the actuator arm.

[0044] See Figure 4 In step 401, the current posture of the master manipulator may be determined, the current posture including the current position and the current posture. In some embodiments, the current posture of the master manipulator is a posture relative to the master manipulator base coordinate system {CombX}. For example, the posture of the master manipulator is a posture of a coordinate system defined by a handle of the master manipulator or a portion thereof relative to the master manipulator base coordinate system {CombX} (e.g., a coordinate system defined by a support or base on which the master manipulator is located, or a world coordinate system). In some embodiments, determining the current position of the master manipulator includes determining the current position of the handle of the master manipulator relative to the master manipulator base coordinate system {CombX}, and determining the current posture of the master manipulator includes determining the current posture of the handle of the master manipulator relative to the master manipulator base coordinate system {CombX}.

[0045] In some embodiments, the current pose of the master manipulator can be determined based on a coordinate transformation. For example, the current pose of the handle can be determined based on the transformation relationship between the coordinate system {H} of the master manipulator's handle and the master manipulator's base coordinate system {CombX}. Typically, the master manipulator's base coordinate system {CombX} can be set on the support or base on which the master manipulator is mounted, and during teleoperation, the master manipulator's base coordinate system {CombX} remains unchanged.

[0046] In some embodiments, the current posture of the main manipulator can be determined based on the main manipulator sensor. In some embodiments, the current joint information of at least one joint of the main manipulator is received, and based on the current joint information of at least one joint, the current posture of the main manipulator is determined. For example, the current posture of the main manipulator is determined based on the current joint information of at least one joint obtained by the main manipulator sensor. The main manipulator sensor is arranged at at least one joint position of the main manipulator. For example, the main manipulator includes at least one joint, and at least one main manipulator sensor is arranged at at least one joint. The joint information (position or angle) of the corresponding joint is obtained based on the main manipulator sensor, and the current posture of the main manipulator is calculated. For example, the current position and current posture of the main manipulator are calculated based on the forward kinematics algorithm.

[0047] In some embodiments, the master manipulator includes at least one posture joint for controlling the posture of a handle. Determining the current posture of the master manipulator's handle includes obtaining joint information of at least one posture joint and determining the current posture of the master manipulator based on the joint information of the at least one posture joint. The master manipulator includes a robotic arm, which includes position joints and posture joints. The posture joints adjust the posture of the master manipulator, controlling the master manipulator to achieve a target posture via one or more posture joints. The position joints adjust the position of the master manipulator, controlling the master manipulator to achieve a target position via one or more position joints. Sensors of the master manipulator are disposed at the posture joints and position joints of the robotic arm to obtain joint information (position or angle) corresponding to the posture joints and position joints. Based on the obtained joint information, the current posture of the master manipulator's handle relative to the master manipulator's base coordinate system {CombX} can be determined. For example, the master manipulator may include seven joints, of which joints 5, 6, and 7 are posture joints used to adjust the posture of the master manipulator's handle. The current posture of the master manipulator is calculated based on the joint information (e.g., angles) obtained by the main manipulator sensors of the posture joints and a forward kinematics algorithm. Joints 1, 2, and 3 are positional joints used to adjust the position of the main manipulator's handle. The current position of the main manipulator is calculated based on joint information (such as position) obtained by the main manipulator's sensors and a forward kinematics algorithm.

[0048] In step 403, the target pose of the end of the actuator arm can be determined based on the current pose of the master manipulator and the pose relationship between the master manipulator and the end of the actuator arm. For example, a master-slave mapping relationship is established between the master manipulator and the end of the actuator arm, and the pose of the end of the actuator arm is controlled by remotely operating the master manipulator. The pose relationship includes the relationship between the pose of the end of the actuator arm relative to the reference coordinate system {w} and the pose of the master manipulator relative to the reference coordinate system {w}. The reference coordinate system {w} includes the coordinate system of the space where the master manipulator, the actuator arm, or the camera is located, or the world coordinate system.

[0049] In some embodiments, the posture relationship between the main manipulator and the end of the actuator arm may include a relationship between the posture change of the main manipulator and the posture change of the end of the actuator arm, such as being equal or proportional. Determining the target posture of the end of the actuator arm includes: determining the previous posture of the main manipulator, determining the starting posture of the end of the actuator arm, and determining the target posture of the end of the actuator arm based on the previous posture and current posture of the main manipulator and the starting posture of the end of the actuator arm. The previous posture and the current posture of the main manipulator can be the posture of the handle of the main manipulator relative to the main manipulator base coordinate system {CombX}. The starting posture and the target posture of the end of the actuator arm can be the posture of the end of the actuator arm relative to the actuator base coordinate system {Tb}.

[0050] The posture of the end of the actuator arm may include the posture of the end coordinate system {wm} of the actuator arm relative to the base coordinate system {Tb} of the actuator arm. The base coordinate system {Tb} of the actuator arm may be the coordinate system of the base on which the actuator arm is installed, the coordinate system of the sheath through which the end of the actuator arm passes (for example, the coordinate system of the sheath outlet), the coordinate system of the proximal motion center point (Remote Center of Motion, RCM) of the actuator arm, etc. For example, the base coordinate system {Tb} of the actuator arm may be set at the sheath outlet position, and during the remote operation process, the base coordinate system {Tb} of the actuator arm remains unchanged. The starting posture of the end of the actuator arm may be subjected to a coordinate system transformation to obtain a posture relative to other coordinate systems (for example, a reference coordinate system).

[0051] In some embodiments, previous joint information of at least one joint of the master manipulator can be received, and based on the previous joint information of the at least one joint, a previous position and current position of the master manipulator's handle can be determined. For example, based on the joint information of the master manipulator read by the master manipulator sensor at previous and current times, a previous position and current position of the master manipulator's handle can be determined. Based on the previous position and current position of the handle relative to the master manipulator's base coordinate system {CombX}, a position change of the master manipulator's handle can be determined. Based on the previous posture and current posture of the handle relative to the master manipulator's base coordinate system {CombX}, a posture change of the master manipulator's handle can be determined.

[0052] In some embodiments, the actual posture of the end of the executing arm obtained in the previous detection cycle can be received as the starting posture of the end of the executing arm in the current detection cycle. For example, in each detection cycle, the camera can take a positioning image of the end of the executing arm, and the positioning image can be used to identify multiple markers located on the end of the executing arm, and then determine the actual posture of the end of the executing arm (described in detail later). The actual posture can be used as the starting posture of the end of the executing arm in the next detection cycle. For example, for the first detection cycle, the initial posture of the end of the executing arm (for example, the zero position of the executing arm) can be used as the starting posture of the first detection cycle.

[0053] In some embodiments, a change in the posture of the master manipulator can be determined based on a previous posture and a current posture of the master manipulator. A change in the posture of the distal end of the actuator arm can be determined based on the change in the posture of the master manipulator and the posture relationship between the master manipulator and the distal end of the actuator arm. A target posture of the distal end of the actuator arm can be determined based on the starting posture of the distal end of the actuator arm and the change in the posture of the distal end of the actuator arm.

[0054] The posture relationship may include a position relationship and a posture relationship. The position relationship between the master manipulator and the end of the actuator arm may include a relationship between a position change of the master manipulator and a position change of the end of the actuator arm, such as being equal or proportional. The posture relationship between the master manipulator and the end of the actuator arm may include a relationship between a posture change of the master manipulator and a posture change of the end of the actuator arm, such as being equal or proportional.

[0055] In some embodiments, method 400 further includes determining a current position of a handle of the master manipulator relative to a master manipulator base coordinate system, determining a previous position of the handle relative to the master manipulator base coordinate system, determining a starting position of the end of the actuator arm relative to the actuator arm base coordinate system, and determining a target position of the end of the actuator arm relative to the actuator arm base coordinate system based on the previous and current positions of the handle relative to the master manipulator base coordinate system, the transformation relationship between the actuator arm base coordinate system and the master manipulator base coordinate system, and the starting position of the end of the actuator arm relative to the actuator arm base coordinate system. For example, the previous position of the master manipulator is determined based on joint information corresponding to the master manipulator at a previous time read by the master manipulator sensor, and the current position of the master manipulator is determined based on joint information corresponding to the master manipulator at a current time read by the master manipulator sensor. The position change of the master manipulator is determined based on the previous and current positions of the handle relative to the master manipulator base coordinate system {CombX}. The starting position of the end of the actuator arm is determined based on the actual pose of the end of the actuator arm obtained in the previous detection cycle. The position change of the end of the actuator arm is determined based on the position change of the master manipulator and the pose relationship between the master manipulator and the end of the actuator arm. The target position of the distal end of the actuator arm is determined based on the initial position of the distal end of the actuator arm and the position change amount of the distal end of the actuator arm.

[0056] In some embodiments, method 400 further includes determining a current pose of a handle of the master manipulator relative to a master manipulator base coordinate system, determining a previous pose of the handle relative to the master manipulator base coordinate system, determining a starting pose of the end of the actuator arm relative to the actuator arm base coordinate system, and determining a target pose of the end of the actuator arm relative to the actuator arm base coordinate system based on the previous and current poses of the handle relative to the master manipulator base coordinate system, the transformation relationship between the actuator arm base coordinate system and the master manipulator base coordinate system, and the starting pose of the end of the actuator arm relative to the actuator arm base coordinate system. For example, the previous pose of the master manipulator is determined based on joint information corresponding to the master manipulator at a previous time read by the master manipulator sensor, and the current pose of the master manipulator is determined based on joint information corresponding to the master manipulator at a current time read by the master manipulator sensor. A change in the pose of the master manipulator is determined based on the previous and current poses of the handle relative to the master manipulator base coordinate system {CombX}. The starting pose of the end of the actuator arm is determined based on the actual pose of the end of the actuator arm obtained in the previous detection cycle. The change in the pose of the end of the actuator arm is determined based on the pose change of the master manipulator and the pose relationship between the master manipulator and the end of the actuator arm. The target posture of the distal end of the actuator arm is determined based on the initial posture of the distal end of the actuator arm and the posture change amount of the distal end of the actuator arm.

[0057] In some embodiments, the posture relationship includes: the position change of the end of the actuator arm in the reference coordinate system {w} is proportional to the position change of the main manipulator in the reference coordinate system {w}, which can be expressed as:

[0058] w ΔP wm =k· w ΔP H (1)

[0059] In formula (1), the left side w ΔP wm Indicates the position change of the end of the actuator arm relative to the reference coordinate system {w}. w ΔP H represents the position change of the main manipulator relative to the reference coordinate system {w}. And, w ΔP wm and w ΔP H It is a proportional relationship, and the proportional coefficient is k.

[0060] In some embodiments, the master manipulator may be positioned relative to the reference coordinate system {w} based on its previous position. w P H(t0) and current location w P H Determine the position change of the main operator w ΔPH For example, at time t0 in teleoperation, the previous position of the master manipulator's handle relative to the reference coordinate system {w} can be determined based on the master manipulator's joint information obtained by the master manipulator sensor. w P H(t0) At time t1 in teleoperation, the current position of the master manipulator's handle relative to the reference coordinate system {w} can be determined based on the master manipulator's joint information obtained by the master manipulator sensor. w P H Based on the previous position of the master operator at time t0 w P H(t0) and the current position of the main operator at time t1 w P H , get the position change of the main manipulator w ΔP H In some embodiments, the period between time t0 and time t1 may include multiple control cycles of the actuator arm, time t0 may be the time when the teleoperation instruction is triggered or the time when multiple control cycles begin, and time t1 may be the time when the teleoperation instruction ends or the time when multiple control cycles complete.

[0061] In some embodiments, the starting position of the end of the actuator arm relative to the reference coordinate system {w} can be w P wmS and target location w P wmT , determine the position change of the end of the actuator arm w ΔP wm In some embodiments, the detection cycle of the execution arm (e.g., t0 to t1) may cover multiple control cycles of the execution arm. For example, the last detection cycle of the execution arm may end at time t0, and the current detection cycle of the execution arm may start at time t0 and end at time t1. In some embodiments, the actual position of the end of the execution arm obtained in the last detection cycle (e.g., time t0) may be w P wmR(t0) , which is determined as the starting position of the end of the actuator arm relative to the reference coordinate system {w} in the current detection cycle w P wmS . Can be based on the position change of the handle w ΔP H and the starting position of the end of the actuator relative to the reference coordinate system {w} w P wmS , determine the target position of the end of the actuator relative to the reference coordinate system {w} w P wmT .

[0062] In formula (1), the position change of the end of the actuator arm relative to the reference coordinate system {w} is w ΔP wm The target position of the end of the actuator arm relative to the reference coordinate system {w} can be w P wmT and the starting position of the end of the actuator arm (e.g., at time t0) relative to the reference coordinate system {w} w P wmS The difference between , as shown in formula (2),

[0063] w ΔP wm = w P wmT - w P wmS (2)

[0064] In formula (1), the position change of the main manipulator relative to the reference coordinate system {w} is w ΔP H The current position of the master manipulator (for example, at time t1) relative to the reference coordinate system {w} w P H and the previous position of the master manipulator (e.g., at time t0) relative to the reference frame {w} w P H(t0) The difference between , as shown in formula (3),

[0065] w ΔP H = w P H - w P H(t0) (3)

[0066] In some embodiments, the left and right sides of formula (1) are multiplied by the same matrix Tb R w , based on formula (1) to formula (3), we get formula (4),

[0067] Tb R w ( w P wmT - w P wmS )=k· Tb R w ( w P H - w P H(t0) ) (4)

[0068] Based on the left side of formula (4), we get formula (5):

[0069] Tb R w ( w P wmT - w P wmS )= Tb P wmT - Tb P wmS (5)

[0070] Based on the right side of formula (4), we get formula (6):

[0071] k. Tb R w ( w P H - w P H(t0) )=k· Tb R CombX ( CombX P H - CombX P H(t0) ) (6)

[0072] Based on formula (5) and formula (6), formula (7) is obtained:

[0073] Tb P wmT =k· Tb R CombX ( CombX P H - CombX P H(t0) )+ Tb P wmS (7)

[0074] Based on formula (7), in some embodiments, the previous position of the handle relative to the main manipulator base coordinate system {CombX} can be CombX P H(t0) and current location CombX P H The current position of the end of the actuator arm relative to the actuator arm base coordinate system {Tb} Tb P wmS , the transformation relationship between the main manipulator base coordinate system {CombX} and the actuator base coordinate system {Tb} Tb R CombX , determine the target position of the end of the actuator arm relative to the actuator arm base coordinate system {Tb} Tb P wmT .

[0075] In some embodiments, the posture of the end of the actuator arm in the reference coordinate system {w} is consistent with the posture of the master manipulator in the reference coordinate system {w}. In some embodiments, the change in posture of the end of the actuator arm relative to the reference coordinate system {w} is consistent with the change in posture of the master manipulator relative to the reference coordinate system {w}, which can be expressed as:

[0076] w R wmS-wmT = w R H(t0)-H (8)

[0077] In formula (8), the left side w R wmS-wmT Indicates the change in the posture of the end of the actuator arm relative to the reference coordinate system {w}. w R H(t0)-H Indicates the change in the master manipulator's posture relative to the reference coordinate system {w}.

[0078] In some embodiments, the master manipulator may be positioned relative to the reference coordinate system {w} based on its previous pose. w R H(t0) and current posture w R H Determine the posture change of the main manipulator w R H(t0)-H For example, at time t0 in teleoperation, the previous posture of the master manipulator's handle relative to the reference coordinate system {w} can be determined based on the joint information of the master manipulator obtained by the master manipulator sensor. w R H( t0 ) At time t1 in teleoperation, the current posture of the master manipulator's handle relative to the reference coordinate system {w} can be determined based on the joint information of the master manipulator obtained by the master manipulator sensor. w R H . Based on the previous posture of the master manipulator at time t0 w R H(t0) And the current posture of the main manipulator at time t1 w R H , get the posture change of the main manipulator w R H(t0)-H Similarly, in some embodiments, time t0 to time t1 may correspond to a single detection cycle, which may include multiple control cycles of the actuator arm. Time t0 may be the moment when the teleoperation instruction is triggered or the moment when the detection cycle begins, and time t1 may be the moment when the teleoperation instruction ends or the moment when the detection cycle is completed.

[0079] In some embodiments, the starting posture of the end of the actuator arm relative to the reference coordinate system {w} can be w RwmS and target posture w R wmT , determine the posture change of the end of the actuator arm w R wmS-wmT . Similarly, in some embodiments, the detection cycle of the execution arm (e.g., t0 to t1) can cover multiple control cycles of the execution arm. For example, the previous detection cycle of the execution arm can end at time t0, and the current detection cycle of the execution arm can start at time t0 and end at time t1. In some embodiments, the actual posture of the end of the execution arm obtained in the previous detection cycle (e.g., time t0) can be w R wmR(t0) , which is determined as the starting posture of the end of the actuator arm relative to the reference coordinate system {w} in the current detection cycle w R wmS . Can be based on the amount of change in the handle's posture w R H(t0)-H And the starting posture of the end of the actuator relative to the reference coordinate system {w} w R wmS , determine the target posture of the end of the actuator arm relative to the reference coordinate system {w} w R wmT .

[0080] In formula (8), the attitude change of the end of the actuator arm relative to the reference coordinate system {w} is w R wmS-wmT The starting posture of the end of the actuator arm relative to the reference coordinate system {w} can be w P wmS and the target posture of the end of the actuator relative to the reference coordinate system {w} w R wmT Determine the attitude change of the main manipulator relative to the reference coordinate system {w} w R H(t0)-H The previous pose of the handle (e.g., at time t0) relative to the reference frame {w} can be w R H(t0) and the current pose of the handle relative to the reference frame {w} (e.g., at time t1) w R H Determine. See formula (9) for details.

[0081] w R wmT ( w R wmS ) T = w R H ( w R H(t0) ) T (9)

[0082] In some embodiments, the left and right sides of formula (9) are multiplied by the same matrix Tb R w ( Tb R w ) T , based on formula (9), we get formula (10),

[0083] Tb R w w R wmT ( w R wmS ) T ( Tb R w ) T = Tb R w w R H ( w R H(t0) ) T ( Tb R w ) T (10)

[0084] Based on the left side of formula (10), we get formula (11):

[0085] Tb R w w R wmT ( w R wmS ) T ( Tb R w ) T =( Tb R w w R wmT )( Tb R w w R wmS ) T = Tb R wmT ( Tb R wmS ) T (11)

[0086] Based on the right side of formula (10), we get formula (12):

[0087] Tb R w w R H ( w RH(t0) ) T ( Tb R w ) T = Tb R H ( Tb R H(t0) ) T =( Tb R CombX CombX R H )( Tb R CombX CombX R H(t0) ) T (12)

[0088] Combining formulas (8) to (12), we can get the target posture of the end of the actuator arm during teleoperation: Tb R wmT The expression is as shown in formula (13),

[0089] Tb R wmT = Tb R CombX ( CombX R H ( CombX R H(t0) ) T ) CombX R Tb Tb R wmS (13)

[0090] Based on formula (13), in some embodiments, the previous posture of the handle relative to the main manipulator base coordinate system {CombX} can be CombX R H(t0) and current posture CombX R H , the starting posture of the end of the actuator arm relative to the actuator arm base coordinate system {Tb} w R wmS And the transformation relationship between the base coordinate system of the actuator arm {Tb} and the base coordinate system of the main manipulator {CombX} CombX R Tb , determine the target posture of the end of the actuator arm relative to the actuator arm base coordinate system {Tb} Tb R wmT .

[0091] In some embodiments, the transformation relationship between the base coordinate system of the actuator arm {Tb} and the base coordinate system of the main manipulator {CombX} is CombX R Tb It can be based on the transformation relationship between the execution arm base coordinate system {Tb} and the camera coordinate system {lens}lens R Tb , the transformation relationship between the camera coordinate system {lens} and the display coordinate system {Screen} Screen R lens , the transformation relationship between the display coordinate system {Screen} and the main operator base coordinate system {CombX} CombX R Screen Sure.

[0092] In some embodiments, the transformation relationship between the main operator and the display can be predetermined. For example, the main operator and the display can be fixedly set on the main control trolley respectively, and the display coordinate system {Screen} and the main operator base coordinate system {CombX} have a predetermined transformation relationship. In some embodiments, the execution arm base coordinate system {Tb} and the camera coordinate system {lens} have a predetermined transformation relationship. In some embodiments, the camera can be set at the end of the visual tool. Before the operator performs the operation, the visual tool has finished moving, and the transformation relationship between the execution arm base coordinate system {Tb} and the camera coordinate system {lens} is lens R Tb No longer changes.

[0093] In some embodiments, the display coordinate system {Screen} and the camera coordinate system {Lens} define the same field of view. Therefore, the positional change of the image of the end of the actuator arm on the display relative to the display coordinate system {Screen} is consistent with the positional change of the end of the actuator arm relative to the camera coordinate system {Lens}. Thus, when an operator grasps the handle of the main manipulator and operates it, the positional change of the image of the actuator at the end of the actuator arm and the positional change of the main manipulator handle perceived by the operator maintain a predetermined transformation relationship.

[0094] In some embodiments, the target pose of the end of the actuator arm relative to the reference coordinate system {w} can be based on the target pose of the end of the actuator arm relative to the actuator arm base coordinate system {Tb} and the transformation relationship between the actuator arm base coordinate system {Tb} and the reference coordinate system {w} w R Tb In some embodiments, the arm base coordinate system {Tb} and the reference coordinate system {w} have a predetermined transformation relationship. Specifically, as shown in formula (14),

[0095]

[0096] Those skilled in the art will appreciate that the actuator arm base coordinate system {Tb} may be used as the reference coordinate system {w}.

[0097] In some embodiments, a plurality of identifiers are distributed on the actuator arm (for example, on the actuator arm end 231). In some embodiments, a plurality of identifiers are provided on the outer surface of the columnar portion of the actuator arm 230. For example, a plurality of identifiers are circumferentially distributed on the actuator arm end 231, for example, circumferentially provided on the outer surface of the columnar portion of the actuator arm end 231. In some embodiments, a positioning tag 232 including a plurality of identifiers is provided on the outer surface of the columnar portion of the actuator arm end 231, and the plurality of identifiers may include a plurality of posture identifiers for identifying postures and at least one composite identifier for identifying postures and angles (for example, angles around an axis or roll angles). In some embodiments, a positioning tag 232 including a plurality of identifiers is provided on the outer surface of the columnar portion of the actuator arm end (for example, Figure 6 The label 600 shown in the figure) may include a plurality of identification patterns distributed on the positioning label along the circumference of the columnar portion and a plurality of identification pattern corner points in the identification pattern. The plurality of identification patterns include a plurality of different composite identification patterns and a plurality of posture identification patterns, and the plurality of posture identification patterns may be the same. The composite identification pattern and the pattern corner points therein may be used to identify the posture and angle, and the posture identification pattern and the pattern corner points therein may be used to identify the posture. In some embodiments, the plurality of different composite identification patterns and the plurality of posture identification patterns are located in the same pattern distribution band, such as Figure 6 or Figure 7 In some embodiments, N consecutive identification patterns in the plurality of identification patterns include at least one composite identification pattern, where 2≤N≤4, and the composite identification pattern in the N consecutive identification patterns is different from the posture identification pattern. For example, the plurality of identification patterns can be evenly distributed on the outer surface of the columnar portion, and the plurality of composite identification patterns can be evenly spaced among the plurality of posture identification patterns, for example, a composite identification pattern is inserted into every three posture identification patterns, as shown in FIG. Figure 6 shown.

[0098] In some embodiments, the identification pattern may be provided on a label on the end of the actuator arm, or may be printed on the end of the actuator arm, or may be a pattern formed by the physical structure of the end of the actuator arm itself, for example, it may include depressions or protrusions and a combination thereof. In some embodiments, the identification pattern may include a pattern formed by brightness, grayscale, color, etc. In some embodiments, the identification pattern may include a pattern that actively (for example, self-luminous) or passively (for example, reflected light) provides information detected by the image acquisition device. It will be understood by those skilled in the art that in some embodiments, the posture of the identification or the posture of the identification pattern may be represented by the posture of the coordinate system of the corner points of the identification pattern. In some embodiments, the identification pattern is provided on the end of the actuator arm in an area suitable for image acquisition by the image acquisition device, for example, an area that can be covered by the field of view of the image acquisition device during operation or an area that is not easily interfered with or blocked during operation.

[0099] Figure 6 A schematic diagram of a tag 600 including multiple logos is shown according to some embodiments. Figure 7 FIG. 7 is a schematic diagram showing a cylindrical label 700 disposed on the peripheral side of the end of the actuator arm. It is understood that for simplicity, the label 600 and the label 700 may include the same identification pattern.

[0100] See Figure 6 The multiple identifications include multiple posture identification patterns 610 and multiple posture identification pattern corner points P 610 , and the composite logo pattern 620 and the composite logo pattern corner point R therein 620 In some embodiments, as Figure 6 As shown, multiple pose identification patterns 610 and composite identification patterns 620 are arranged in the same pattern distribution band. In this disclosure, the "0" symbol represents the pose identification pattern corner point, and the "△" symbol represents the composite identification pattern corner point. In some embodiments, the pose identification pattern 610 or the pose identification pattern corner point P can be identified by 610 Determine the pose identification by identifying the composite identification pattern 620 or the composite identification pattern corner point R 620 Determine the composite identity.

[0101] See Figure 7 In the circumferential setting state, the tag 600 becomes a tag 700 with a spatial structure of a cylinder. In some embodiments, the rotation angle or roll angle of each marker can be represented by the rotation angle of the marker pattern or the corner point of the marker pattern, wherein the marker pattern includes the posture marker pattern 710 and the composite marker pattern 720. The rotation angle of each marker pattern or the corner point of the marker pattern is known or predetermined. In some embodiments, based on the distribution of multiple markers (marker patterns or marker pattern corner points), the rotation angle identified by each marker can be determined. In some embodiments, the multiple markers can be evenly distributed (for example, the marker pattern corner points in the tag 600 are evenly spaced, and the marker pattern corner points in the tag 700 are evenly distributed). In some embodiments, based on the distribution of multiple markers, each marker can be used to identify a specific rotation angle, and each marker has a one-to-one correspondence with the identified rotation angle. In the present disclosure, the rotation angle or roll angle refers to the angle around the Z axis (for example, the Z axis of the end coordinate system of the actuator arm or the marker coordinate system). In some embodiments, the Z axis can be a tangential direction along the end of the actuator arm.

[0102] like Figure 7 As shown, multiple identification patterns in the label 700 are evenly distributed along the circumference of the cylindrical structure, and multiple identification pattern corner points are evenly distributed on the cross-sectional circle 730. Then, the distribution angles (for example, angle α0) of any adjacent identification pattern corner points are equal.701 , P 701 As a reference point for marking the 0° angle around the axis (marking pattern corner point P 701 The logo pattern where it is located is used as a reference pattern), then according to the angle between any logo pattern corner point and the logo pattern corner point P 701 The positional relationship determines the angle around the axis of the corner point mark of the marking pattern.

[0103] In some embodiments, the corner points of the identification pattern are in a set coordinate system (eg, Figure 7 The identification coordinate system shown in {wm0}≡[X wm0 Y wm0 Z wm0 ] T The angle around the axis indicated in ) can be determined based on the following formula (15):

[0104] α m =α0(m-1) (15)

[0105] Among them, α m The selected marking pattern corner point (for example, the marking pattern corner point P 701 ) as the first identification pattern corner point, and the angle of the mth identification pattern corner point around the axis in the clockwise direction of the cross-sectional circle 730.

[0106] In some embodiments, the multiple posture identification patterns can be the same pattern or different patterns. In some embodiments, the multiple composite identification patterns are different patterns, each composite identification pattern can be used to identify a specific rotation angle, and each composite identification pattern has a one-to-one correspondence with the identified rotation angle.

[0107] Figure 8 Schematic diagram of an implementation scenario 800 according to some embodiments of the present disclosure is shown. Figure 8 As shown, the actuator arm 840 includes a distal end 830 and an actuator 860 at the distal end, and a plurality of markers (eg, a posture marker pattern 810 and a composite marker pattern 820) can be circumferentially arranged on the distal end 830. Figure 6 The label 600 is circumferentially arranged on the end of the actuator arm 830. A plurality of identification pattern corner points are distributed on the cross-sectional circle 831 of the end of the actuator arm 830. In some embodiments, based on the identified identification, an identification coordinate system {wm0}≡[X wm0 Y wm0 Z wm0 ] T The origin of the marker coordinate system {wm0} is the center of the cross-section circle 831, and the X-axis direction is from the origin to one of the marker pattern corners (for example, the pattern corner P corresponding to one of the identified pose markers). 801), the direction of the Z axis is parallel to the axial direction of the actuator arm end 830, and the Y axis is perpendicular to the XZ plane.

[0108] In some embodiments, the end coordinate system {wm}≡[X wm Y wm Z wm ] T The origin of the end coordinate system {wm} of the actuator arm is the center of the cross-section circle 831, and the X axis points to the corner point R of the composite logo pattern. 801 , the Z axis is parallel to or coincides with the axial direction of the actuator arm end 830, and the Y axis is perpendicular to the XZ plane. In some embodiments, the distribution of multiple composite identification patterns can be determined based on the distribution of multiple composite identification patterns, such as the distribution of the remaining composite identification patterns and the composite identification pattern corner point R 801 The positional relationship of the corresponding composite logo patterns is used to determine the angle around the axis of the composite logo pattern corner point logo included in the composite logo pattern.

[0109] Continue to read Figure 3 In step 303, a positioning image is acquired. In some embodiments, the positioning image includes multiple identifiers on the end of the actuator arm. In some embodiments, the multiple identifiers include multiple posture identifiers for identifying postures and at least one composite identifier for identifying postures and angles. In some embodiments, the positioning image can be obtained from Figure 2 The image acquisition device 250 shown receives the positioning image. For example, the control device 220 can receive the positioning image actively sent by the image acquisition device 250. Alternatively, the control device 220 can send an image request instruction to the image acquisition device 250, and the image acquisition device 250 sends the positioning image to the control device 220 in response to the image request instruction.

[0110] Continue to read Figure 3 In step 305, a plurality of markers located on the distal end of the actuator arm are identified in the positioning image. For example, an exemplary method of identifying a plurality of markers located on the distal end of the actuator arm may include: Figure 11 and Figure 13The method shown. In some embodiments, the control device 220 may identify part or all of the identifiers in the positioning image through an image processing algorithm. In some embodiments, the image processing algorithm may include a feature recognition algorithm, and the image processing algorithm may extract or identify the features of the identifier. For example, the image processing algorithm may include a corner detection algorithm for detecting the corners of the identification pattern. The corner detection algorithm may be one including but not limited to grayscale image-based corner detection, binary image-based corner detection, and contour curve-based corner detection. For example, the image processing algorithm may be a color feature extraction algorithm for detecting color features in the identification pattern. For another example, the image processing algorithm may be a contour detection algorithm for detecting contour features of the identification pattern. In some embodiments, the control device may identify part or all of the identifiers in the positioning image through a recognition model.

[0111] Continue to read Figure 3 In step 307, the actual pose of the end of the actuator arm is determined based on the at least one composite identifier and the multiple pose identifiers. In some embodiments, the pose of the coordinate system of the end of the actuator arm relative to the reference coordinate system can be determined based on the two-dimensional coordinates of the at least one composite identifier and the multiple pose identifiers in the positioning image and the three-dimensional coordinates in the coordinate system of the end of the actuator arm, as the actual pose of the end of the actuator arm.

[0112] In some embodiments, method 300 may further include determining the two-dimensional coordinates of multiple identifiers in the positioning image. In some embodiments, the coordinates of the identifiers may be represented by the coordinates of the corner points of the identifier pattern. For example, the two-dimensional coordinates of the identifier in the positioning image and the three-dimensional coordinates in the end coordinate system of the execution arm may be represented by the coordinates of the corner points of the identifier pattern. In some embodiments, determining the two-dimensional coordinates of multiple identifiers in the positioning image may include determining the two-dimensional coordinates of at least one composite identifier and multiple pose identifiers in the positioning image. In some embodiments, method 500 may further include determining the three-dimensional coordinates of at least one composite identifier and multiple pose identifiers in the end coordinate system of the execution arm based on the at least one composite identifier.

[0113] In some embodiments, method 300 may further include determining the pose of the actuator arm's end coordinate system relative to the reference coordinate system based on the two-dimensional coordinates of at least one composite marker pattern corner point and multiple pose marker pattern corner points in the positioning image, their three-dimensional coordinates in the actuator arm's end coordinate system, and the transformation relationship between the camera coordinate system and the reference coordinate system. In some embodiments, the transformation relationship between the camera coordinate system and the reference coordinate system may be known. For example, if the reference coordinate system is a world coordinate system, the transformation relationship between the camera coordinate system and the world coordinate system may be determined based on the pose of the camera. In other embodiments, depending on actual needs, the reference coordinate system may also be the camera coordinate system itself. In some embodiments, based on camera imaging principles and projection models, the pose of the actuator arm's end coordinate system relative to the camera coordinate system is determined based on the two-dimensional coordinates of at least one composite marker pattern corner point and multiple pose marker pattern corner points in the positioning image, as well as their three-dimensional coordinates in the actuator arm's end coordinate system. Based on the pose of the actuator arm's end coordinate system relative to the camera coordinate system and the transformation relationship between the camera coordinate system and the reference coordinate system, the pose of the actuator arm's end coordinate system relative to the reference coordinate system can be obtained.

[0114] In some embodiments, the camera's intrinsic parameters may also be considered. For example, the camera's intrinsic parameters may be Figure 2 The camera intrinsic parameters of the image acquisition device 250 shown. The intrinsic parameters of the camera can be known or obtained through calibration. In some embodiments, the camera coordinate system can be understood as a coordinate system established with the camera origin. For example, a coordinate system established with the optical center of the camera as the origin or a coordinate system established with the lens center of the camera as the origin. When the camera is a binocular camera, the origin of the camera coordinate system can be the center of the left lens of the camera, or the center of the right lens, or any point on the line connecting the centers of the left and right lenses (for example, the midpoint of the line).

[0115] In some embodiments, the pose of the end coordinate system {wm} of the actuator arm relative to the reference coordinate system (e.g., the world coordinate system) {w} can be determined based on the following formula (16):

[0116] w R wm = w R lens lens R wm

[0117] w P wm = w R lens ( lens R wm + lens P wm )+ w P lens (16)

[0118] in, w R wm is the posture of the end coordinate system {wm} of the actuator arm relative to the reference coordinate system, w P wm is the position of the end coordinate system of the actuator arm relative to the reference coordinate system, w R lens is the posture of the camera coordinate system relative to the reference coordinate system, w P lens is the position of the camera coordinate system relative to the reference coordinate system, lens R wm is the posture of the end coordinate system of the actuator arm relative to the camera coordinate system, lens P wm is the position of the end coordinate system of the actuator arm relative to the camera coordinate system.

[0119] Figure 9 FIG. 9 is a flow chart showing a method 900 for determining the actual posture of the end of an actuator arm according to some embodiments of the present disclosure. Figure 9 As shown, some or all of the steps in the method 900 may be performed by a control device (e.g., Figure 1 The control device 120 shown, Figure 2 The control device 220 shown, or Figure 16 Some or all of the steps in method 900 may be implemented by software, firmware, and / or hardware. In some embodiments, method 900 may be performed by a robotic system (e.g., Figure 1 The robotic system 100 or Figure 16 In some embodiments, the method 900 may be implemented as computer-readable instructions. These instructions may be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 16 In some embodiments, the instructions may be stored on a computer readable medium.

[0120] See Figure 9 In step 901, the three-dimensional coordinates of at least one composite marker and multiple pose markers in the marker coordinate system are determined. In some embodiments, the three-dimensional coordinates of each marker pattern corner point in the marker coordinate system {wm0} can be determined based on the following formula (17):

[0121] C m =[r·cosα m r·sinα m 0] T (17)

[0122] Among them, C mTo use the selected marker pattern corner point as the first marker pattern corner point (for example, the pose marker pattern corner point P 801 ), in the clockwise direction of the cross-sectional circle 831, the three-dimensional coordinates of the m-th corner point of the identification pattern in the identification coordinate system, where r is the radius.

[0123] In some embodiments, the angle α of the mth corner point of the identification pattern is determined based on formula (15): m , then the angle α around the axis is determined based on formula (15) m The three-dimensional coordinates C of the mth corner point of the logo pattern in the logo coordinate system {wm0} are determined by formula (17): m .

[0124] See Figure 9 In step 903, based on at least one composite identifier, the roll angle of the identification coordinate system relative to the end coordinate system of the actuator arm is determined. In some embodiments, a first rotation angle identified by one of the at least one composite identifier in the end coordinate system of the actuator arm can be determined, and a second rotation angle identified by the composite identifier in the identification coordinate system can be determined. Based on the first rotation angle and the second rotation angle, the roll angle of the identification coordinate system relative to the end coordinate system of the actuator arm can be determined. In some embodiments, see Figure 8 The roll angle Δα may refer to the rotation angle of the identification coordinate system {wm0} relative to the end coordinate system {wm} of the actuator arm around the Z axis. In some embodiments, the roll angle Δα may be determined based on the following formula (18):

[0125] Δα=α1-α2 (18)

[0126] Where α1 is the first rotation angle, and α2 is the second rotation angle. The first rotation angle is the composite logo pattern corner point (for example, the composite logo pattern corner point R 802 ) is an angle around the axis marked in the end coordinate system of the actuator arm, and the second angle around the axis is a composite identification pattern corner point (for example, composite identification pattern corner point R 802 ) is the angle around the axis identified in the identified coordinate system.

[0127] In some embodiments, the X-axis of the identification coordinate system {wm0} points to the composite identification pattern corner point (eg, composite identification pattern corner point R 802 ), method 900 may further include determining a first rotation angle identified by the composite marker in the end coordinate system of the actuator arm as a roll angle of the marker coordinate system relative to the end coordinate system of the actuator arm. In some embodiments, the first rotation angle may be determined based on a pattern included in the composite marker.

[0128] See Figure 9In step 905, based on the roll angle of the identification coordinate system relative to the end coordinate system of the actuator arm and the three-dimensional coordinates of the at least one composite identification and the multiple pose identifications in the identification coordinate system, the three-dimensional coordinates of the at least one composite identification and the multiple pose identifications in the end coordinate system of the actuator arm are determined. It will be understood that, given the roll angle of the identification coordinate system relative to the end coordinate system of the actuator arm, the three-dimensional coordinates of the multiple identification pattern corner points (e.g., the composite identification pattern corner points and the pose identification pattern corner points) in the identification coordinate system can be transformed into three-dimensional coordinates in the end coordinate system of the actuator arm according to a coordinate transformation.

[0129] See Figure 9 In step 907, based on the two-dimensional coordinates of the at least one composite identifier and the multiple pose identifiers in the positioning image and the three-dimensional coordinates in the coordinate system of the end of the actuator arm, the pose of the end coordinate system of the actuator arm relative to the reference coordinate system is determined as the actual pose of the end of the actuator arm. In some embodiments, step 907 in method 900 can be implemented similarly to determining the actual pose of the end of the actuator arm in method 300.

[0130] Figure 10 A flowchart of a method 1000 for determining the actual posture of the end of an actuator arm according to other embodiments of the present disclosure is shown. The method 1000 may be Figure 9 An alternative embodiment of method 900 is provided. Figure 10 As shown, some or all of the steps in the method 1000 may be controlled by a control device (e.g., Figure 1 The control device 120 shown, Figure 2 The control device 220 shown, or Figure 16 Some or all of the steps in method 1000 may be implemented by software, firmware, and / or hardware. In some embodiments, method 1000 may be performed by a robotic system (e.g., Figure 1 The robotic system 100 or Figure 16 In some embodiments, the method 1000 may be implemented as computer-readable instructions. These instructions may be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 16 In some embodiments, the instructions may be stored on a computer-readable medium.

[0131] See Figure 10 In step 1001, the pose of the marker coordinate system relative to the reference coordinate system is determined based on the two-dimensional coordinates of the at least one composite marker and the multiple pose markers in the positioning image and the three-dimensional coordinates in the marker coordinate system. In some embodiments, the three-dimensional coordinates of the at least one composite marker and the multiple pose markers in the marker coordinate system can be implemented similarly to step 901 in method 900.

[0132] See Figure 10 In step 1003, based on the at least one composite identifier, a roll angle of the identifier coordinate system relative to the distal end coordinate system of the actuator arm is determined. In some embodiments, determining the roll angle of the identifier coordinate system relative to the distal end coordinate system of the actuator arm can be implemented similarly to step 903 in method 900.

[0133] See Figure 10 In step 1005, based on the roll angle of the identification coordinate system relative to the end coordinate system of the actuator arm and the posture of the identification coordinate system relative to the reference coordinate system, the posture of the end coordinate system of the actuator arm relative to the reference coordinate system is determined as the actual posture of the end of the actuator arm.

[0134] For example, the pose of the end coordinate system {wm} of the actuator arm relative to the reference coordinate system (e.g., the world coordinate system) {w} can be determined based on the following formula (19):

[0135]

[0136] in, w R wm is the posture of the end coordinate system of the actuator relative to the reference coordinate system, w P wm is the position of the end coordinate system of the actuator arm relative to the reference coordinate system, w R wm0 To identify the posture of the coordinate system relative to the reference coordinate system, w P wm0 To identify the position of the coordinate system relative to the reference coordinate system, rot z (Δα) represents the roll angle Δα around the Z-axis of the end coordinate system of the actuator arm.

[0137] Figure 11 FIG. 1 is a flow chart showing a method 1100 for identifying an identifier according to some embodiments of the present disclosure. Figure 11 As shown, some or all of the steps in the method 1100 may be controlled by a control device (e.g., Figure 1 The control device 120 shown, Figure 2 The control device 220 shown, or Figure 16 Some or all of the steps in method 1100 may be implemented by software, firmware, and / or hardware. In some embodiments, method 1100 may be performed by a robotic system (e.g., Figure 1 The robotic system 100 or Figure 16 In some embodiments, the method 1100 may be implemented as computer-readable instructions. These instructions may be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 16In some embodiments, the instructions may be stored on a computer readable medium.

[0138] See Figure 11 In step 1101, multiple candidate markers are determined from the positioning image. In some embodiments, the markers may include marker pattern corner points within the marker pattern. The coordinates or coordinate system origins of the candidate markers may be represented by the candidate marker pattern corner points. In some embodiments, the candidate marker pattern corner points may be possible marker pattern corner points obtained through preliminary processing or preliminary identification of the positioning image.

[0139] In some embodiments, method 1100 may include determining a region of interest (ROI) in the positioning image. For example, the ROI may be first intercepted from the positioning image, and multiple candidate markers may be determined from the ROI. The ROI may be the entire image of the positioning image or a partial area. For example, the ROI of the current frame may be intercepted based on an area within a certain range of multiple marker pattern corner points determined in a previous frame image (e.g., a positioning image of a previous image processing cycle). For positioning images other than the first frame, the ROI may be an area within a certain distance range centered on a virtual point formed by the coordinates of multiple marker pattern corner points of the previous image processing cycle. The certain distance range may be a fixed multiple of the average spacing distance of the marker pattern corner points, such as twice. It should be understood that the predetermined multiple may also be a variable multiple of the average spacing distance of multiple candidate marker pattern corner points in the previous image processing cycle.

[0140] In some embodiments, method 1100 may include determining a corner likelihood (CL) value for each pixel in the positioning image. In some embodiments, the corner likelihood value of a pixel may be a numerical value that characterizes the likelihood of the pixel being a feature point (e.g., a corner point). In some embodiments, the positioning image may be preprocessed before calculating the corner likelihood value of each pixel, and then the corner likelihood value of each pixel in the preprocessed image may be determined. Image preprocessing may, for example, include: image grayscale conversion, image denoising, and image enhancement. For example, image preprocessing may include: extracting a ROI from the positioning image and converting the ROI into a corresponding grayscale image.

[0141] In some embodiments, the method for determining the corner likelihood value of each pixel in the ROI may include, for example, performing a convolution operation on each pixel within the ROI to obtain the first-order and / or second-order derivatives of each pixel. The first-order and / or second-order derivatives of each pixel within the ROI are used to calculate the corner likelihood value of each pixel. Exemplarily, the corner likelihood value of each pixel can be determined based on the following formula (20):

[0142] CL=max(c xy ,c 45 )

[0143]

[0144] Wherein, τ is a set constant, for example, set to 2; I x , I 45 , I y , I n45 are the first-order derivatives of the pixel points in the directions of 0, π / 4, π / 2, and -π / 4; I xy and I 45_45 are the second-order derivatives of the pixel in the directions of 0, π / 2, π / 4, and -π / 4 respectively.

[0145] In some embodiments, method 1100 may include dividing the ROI into multiple sub-regions. For example, a non-maximum suppression method may be used to evenly segment the ROI into multiple sub-images. In some embodiments, the ROI may be evenly segmented into multiple sub-images of 5×5 pixels. The above embodiments are exemplary and non-limiting. It should be understood that the positioning image or ROI may also be segmented into multiple sub-images of other sizes, for example, into multiple sub-images of 9×9 pixels.

[0146] In some embodiments, method 1100 may include determining the pixel with the maximum corner likelihood value in each sub-region to form a pixel set. For example, the pixel with the maximum corner likelihood value in each sub-image may be determined, and the pixel with the maximum corner likelihood value in each sub-image may be compared with a first threshold to determine a set of pixels having a corner likelihood value greater than the first threshold. In some embodiments, the first threshold may be set to 0.06. It should be understood that the first threshold may also be set to other values.

[0147] See Figure 11 In step 1103, a first identifier among the multiple identifiers is identified from the multiple candidate identifiers. In some embodiments, the first identifier is identified based on an identifier pattern matching template. In some embodiments, the identifier pattern matching template includes at least one pose identifier pattern matching template and a plurality of composite identifier pattern matching templates with different patterns. In some embodiments, a composite identifier is identified based on a plurality of composite identifier pattern matching templates with different patterns. For example, in the case where the identifier patterns of the pose identifiers are the same, the pose identifier pattern matching template can be matched with the candidate identifier first. If the match fails, the plurality of different composite identifier pattern matching templates can be matched with the candidate identifier one by one until the match is successful.

[0148] In some embodiments, a marking pattern matching template is used to match the pattern at the corner point of the candidate marking pattern to identify the first marking. For example, a candidate marking pattern corner point that meets a preset pose pattern matching standard is determined to be a first marking pattern corner point. In some embodiments, the marking pattern matching template and the pattern in the area near the marking pattern corner point have the same or similar features. If the matching degree between the marking pattern matching template and the pattern in the area near the candidate marking pattern corner point meets a preset pattern matching standard (for example, the matching degree is higher than a threshold), it can be considered that the pattern in the area near the candidate marking pattern corner point has the same or similar features as the marking pattern matching template, and the current candidate marking pattern corner point can be considered to be the marking pattern corner point.

[0149] In some embodiments, the pixel with the largest CL value in the pixel set is determined as a candidate marker pattern corner point. For example, all pixels in the pixel set can be sorted from largest to smallest CL value, and the pixel with the largest CL value is selected as the candidate marker pattern corner point. In some embodiments, after the candidate marker pattern corner point is determined, a marker pattern matching template is used to match the pattern at the candidate marker pattern corner point. If a preset pattern matching criterion is met, the candidate marker pattern corner point is determined to be the identified first marker pattern corner point.

[0150] In some embodiments, method 1100 may further include, in response to a matching failure, determining a pixel with the largest corner likelihood value among the remaining pixels in the pixel set as a candidate marker pattern corner point. For example, if the candidate marker pattern corner point does not meet a preset matching criterion, a pixel with a secondary CL value (a pixel with the second largest CL value) is selected as a candidate marker pattern corner point, and a marker pattern matching template is used to match the pattern at the candidate marker pattern corner point, and so on, until the first marker pattern corner point is identified.

[0151] In some embodiments, the identification pattern may be a black and white checkerboard pattern, so the identification pattern matching template may be the same checkerboard pattern, and the grayscale distribution G of the identification pattern matching template is used. M Grayscale distribution G of the pixel neighborhood corresponding to the corner point of the candidate logo pattern image The grayscale distribution G of the pixel neighborhood is used to match the pixel. image is the grayscale distribution of pixels within a certain range (e.g., 10×10 pixels) centered on the pixel point. The correlation coefficient can be determined based on the following formula (21):

[0152]

[0153] Wherein, Var() is the variance function, and Cov() is the covariance function. In some embodiments, when the correlation coefficient is less than 0.8, the grayscale distribution within the pixel area has a low correlation with the identification pattern matching template, and the candidate identification pattern corner point with the maximum corner point likelihood value is determined not to be a identification pattern corner point; otherwise, the candidate identification pattern corner point with the maximum corner point likelihood value is determined to be a identification pattern corner point.

[0154] In some embodiments, the method 1100 may further include determining the edge direction of the candidate identification pattern corner point. Figure 12 As shown, the candidate pose identification pattern corner point is the corner point P in the pose identification pattern 1200. 1201 , then the corner point P 1201 The edge direction can refer to the corner point P 1201 The direction of the edge, such as Figure 12 The dotted arrow indicates the direction.

[0155] In some embodiments, the edge direction can be obtained by calculating the first-order derivative value (I) of each pixel in a certain range of neighborhood (e.g., 10×10 pixels) centered at the corner point of the candidate identification pattern in the X direction and the Y direction of the plane coordinate system. x and I y ). For example, the edge direction can be determined based on the following formula (22):

[0156]

[0157] Among them, the first-order derivative (I x and I y ) can be obtained by performing a convolution operation on each pixel point within a certain range of neighborhood. In some embodiments, by calculating the edge direction I of the pixel points within each range of neighborhood, angle and the corresponding weight I weight Perform clustering calculation to obtain the edge direction of the pixel point and select weight I weight The class with the largest proportion corresponds to I angle As the edge direction. It should be noted that if there are multiple edge directions, the weight I is selected. weight I corresponding to the largest number of classes angle as the edge direction.

[0158] In some embodiments, the method used for clustering calculation can be any one of the K-means method, BIRCH (Balanced Iterative Reducing and Clustering using Hierarchies) method, DBSCAN (Density-Based Spatial Clustering of Applications with Noise) method, and GMM (Gaussian Mixed Model) method.

[0159] In some embodiments, method 1100 may include rotating the identification pattern matching template based on the edge orientation. Rotating the identification pattern matching template based on the edge orientation may align the identification pattern matching template with the image at the corner of the candidate identification pattern. The edge orientation of the corner of the candidate identification pattern may be used to determine the orientation of the image at the corner of the candidate identification pattern in the positioning image. In some embodiments, rotating the identification pattern matching template based on the edge orientation may adjust the identification pattern matching template to have the same or nearly the same orientation as the image at the corner of the candidate identification pattern to facilitate image matching.

[0160] See Figure 11 In step 1105, the first identifier is used as a starting point to search for other identifiers. In some embodiments, in response to identifying the composite identifier, other identifiers are identified based on the pose identifier pattern matching template. In some embodiments, the other identifiers include pose identifiers or composite identifiers.

[0161] Figure 13 FIG. 1 is a flow chart showing a method 1300 for searching for an identifier according to some embodiments of the present disclosure. Figure 13 As shown, some or all of the steps in the method 1300 may be controlled by a control device (e.g., Figure 1 The control device 120 shown, Figure 2 The control device 220 shown, or Figure 16 Some or all of the steps in method 1300 may be implemented by software, firmware, and / or hardware. In some embodiments, method 1300 may be performed by a robotic system (e.g., Figure 1 The robotic system 100 or Figure 16 In some embodiments, the method 1300 may be implemented as computer-readable instructions. These instructions may be executed by a general-purpose processor or a special-purpose processor (e.g., Figure 16 In some embodiments, the instructions may be stored on a computer-readable medium.

[0162] See Figure 13 In step 1301, the second marker is determined with the first marker as the starting point. In some embodiments, the second marker pattern corner point is searched for in a set search direction with the first marker pattern corner point as the starting point. In some embodiments, the set search direction may include at least one of the following directions: directly in front of the first marker pattern corner point (corresponding to a 0° angle direction), directly behind (corresponding to a 120° angle direction), directly above (at a 90° angle direction), directly below (at a -90° angle direction), and obliquely (e.g., at a ±45° angle direction).

[0163] In some embodiments, the number of search directions is n, for example, searching in 8 directions, each search direction v sn It can be determined based on the following formula (23):

[0164] v sn =[cos(n·π / 4)sin(n·π / 4)], (n=1,2,…,8) (23)

[0165] In some embodiments, the search direction set in the current step can be determined based on the deviation angle between adjacent identification pattern corner points among the multiple identification pattern corner points determined in the previous frame. Exemplarily, the predetermined search direction can be determined based on the following formula (24):

[0166]

[0167] Among them, (x j ,y j ) are the two-dimensional coordinates of multiple corner points of the identification pattern determined in the previous frame (or the previous image processing cycle); n last The number of multiple identification pattern corner points determined in the previous frame; v s1 The search direction for the first setting; v s2 The search direction for the second setting.

[0168] In some embodiments, as Figure 14 As shown, the first identification pattern corner point P 1401 The coordinate position of the second identification pattern is used as the search starting point, and the second identification pattern corner point P is searched in the set search direction. 1402 For example, the first identification pattern corner point P 1401 The coordinate position of the search is used as the search starting point, and the search box (for example, Figure 14 The dotted box in the figure) searches in the set search direction V with a certain search step size. 1401 Search for the corner points of the identified pattern.

[0169] In some embodiments, if there is at least one candidate marker in the search box, the candidate marker pattern corner point with the largest corner likelihood value in the search box is preferentially selected as the second marker pattern corner point P 1402 When the search box is limited to a suitable size, the first corner point P of the identification pattern is used. 1401 The coordinate position of the second pattern corner point P is used as the search starting point 1402 During the search, the candidate identification pattern corner point with the largest likelihood value among the candidate identifications in the search box is more likely to be the identification pattern corner point. Therefore, it can be considered that the candidate identification pattern with the largest likelihood value in the search box is the second identification pattern corner point P 1402 In order to improve the data processing speed. In other embodiments, in order to improve the accuracy of identification of the corner point of the identification pattern, the candidate identification pattern corner point with the largest corner likelihood value among the candidate identifications appearing in the search box is selected for corner point identification to determine whether the candidate identification pattern corner point with the largest corner likelihood value is the identification pattern corner point. For example, a pose identification pattern matching template or a composite identification pattern matching template can be used to match the image within a certain range of the candidate identification pattern corner point with the largest corner likelihood value. The candidate identification pattern corner point that meets the preset pattern matching standard can be considered as the searched second identification pattern corner point P. 1402 .

[0170] In some embodiments, continue to see Figure 14 , the size of the search box can be gradually increased, thereby gradually increasing the search range. The search step size can be changed synchronously with the side length of the search box. In other embodiments, the size of the search box can also be a fixed size.

[0171] In some embodiments, the identification pattern may be a black and white pattern, and pattern matching may be performed based on the correlation coefficient in formula (21). If the correlation coefficient is greater than a threshold, the candidate identification pattern corner point with the largest corner likelihood value is considered to be the identification pattern corner point and is recorded as the second identification pattern corner point.

[0172] See Figure 13 In step 1303, based on the first identifier and the second identifier, a search direction is determined. In some embodiments, the search direction includes: a first search direction and a second search direction. The first search direction can be a direction starting from the coordinate position of the corner point of the first identifier pattern and away from the corner point of the second identifier pattern. The second search direction can be a direction starting from the coordinate position of the corner point of the second identifier pattern and away from the corner point of the first identifier pattern. For example, Figure 14 The search direction V shown in 1402 .

[0173] In step 1305, the first marker or the second marker is used as a starting point to search for the marker in the search direction. In some embodiments, if the first marker pattern corner point is used as a new starting point, the first search direction in the above embodiment can be used as the search direction to search for the marker pattern corner point. If the second marker pattern corner point is used as a new search starting point, the second search direction in the above embodiment can be used as the search direction to search for the marker pattern corner point. In some embodiments, the search for the new marker pattern corner point (for example, Figure 14 The third identification pattern corner point P in 1403 ) can be performed similarly to step 1301. In some embodiments, the search step size can be the first identification pattern corner point P 1401 and the second identification pattern corner point P 1402 The distance between them is L1.

[0174] In some embodiments, in response to the search distance being greater than a search distance threshold, the pixel with the largest corner likelihood value among the remaining pixels in the pixel set is determined as a candidate identification pattern corner point; and the identification pattern matching template is matched with the identification pattern at the position of the candidate identification pattern corner point to identify the first identification. In some embodiments, after determining the pixel with the largest corner likelihood value among the remaining pixels in the pixel set as a new candidate identification pattern corner point, a new first identification can be identified based on a method similar to step 1103. In some embodiments, the search distance being greater than the search distance threshold can be understood as the search distance in some or all search directions being greater than the search distance threshold. In some embodiments, the search distance threshold can include a set multiple of the distance between the N-1th pose identification pattern corner point and the N-2th pose identification pattern corner point, where N≥3. For example, the search distance threshold is twice the distance between the first two identification pattern corner points. In this way, the maximum search distance for searching the third marker pattern corner point is twice the distance between the first marker pattern corner point and the second marker pattern corner point. If the marker pattern corner point is not found after reaching this search distance in the search direction, the pixel with the maximum corner likelihood value among the remaining pixels in the pixel set is determined as the new candidate pose marker pattern corner point, and a new first marker is identified, and the current search process is terminated accordingly. In some embodiments, similar to method 1100, a new first marker pattern corner point can be re-determined, and similar to method 1300, the remaining marker pattern corner points can be searched using the new marker pattern corner point as the search starting point.

[0175] In some embodiments, in response to the number of identified markers being greater than or equal to a marker number threshold, the position of the end of the actuator arm relative to the reference coordinate system can be determined based on the identified markers, and the search for the markers can be stopped accordingly. For example, in response to the number of identified marker pattern corner points being greater than or equal to a marker number threshold, the search for the marker pattern corner points can be stopped. For example, when four marker pattern corner points are identified, the search for the marker pattern corner points can be stopped.

[0176] In some embodiments, in response to the number of identified identifiers being less than a threshold identifier number, the pixel with the maximum corner likelihood value among the remaining pixels in the pixel set is determined as a candidate identifier pattern corner point; and the identifier pattern matching template is matched with the identifier pattern at the location of the candidate identifier pattern corner point to identify the first identifier. In some embodiments, if the total number of identified identifier pattern corner points is less than the threshold identifier number, the search based on the first identifier pattern in the above steps is deemed to have failed. In some embodiments, if all identified identifiers do not include a composite identifier, for example, if the identified identifier pattern corner points do not include a composite identifier pattern corner point, the search based on the first identifier pattern in the above steps is deemed to have failed. In some embodiments, if the search fails, the pixel with the maximum corner likelihood value among the remaining pixels in the pixel set is determined as a new candidate identifier pattern corner point, and a new first identifier can then be identified using a method similar to step 1103. In some embodiments, similar to method 1100, a new first identifier pattern corner point can be re-determined, and similar to method 1300, the search for the remaining identifier pattern corner points can be performed using the new identifier pattern corner point as the search starting point.

[0177] In some embodiments, if the identified identifier includes a composite identifier, the identifier type of the remaining identifiers searched for may not be determined (it should be understood that the identifier type includes a pose identifier and a composite identifier). For example, if the first identifier is a composite identifier, it may not be determined whether the second identifier is a pose identifier or a composite identifier.

[0178] In some embodiments, if the identified identifiers do not include a composite identifier, the type of the newly searched identifier is determined. For example, if the first identifier is not a composite identifier, it is necessary to determine whether the second identifier is a pose identifier or a composite identifier. If neither the first identifier nor the second identifier is a composite identifier, it is necessary to determine whether the third identifier is a pose identifier or a composite identifier, and so on.

[0179] In some embodiments, after searching for or identifying the corner points of the marker pattern, sub-pixel positioning may be performed on the determined corner points of the marker pattern to improve the position accuracy of the corner points of the marker pattern.

[0180] In some embodiments, the CL values of the pixels can be fitted based on a model to determine the coordinates of the corner points of the identification pattern after sub-pixel positioning. For example, the fitting function of the CL value of each pixel in the ROI can be a quadratic surface function, and the extreme point of the function is the sub-pixel point. The fitting function can be determined based on the following formulas (25) and (26):

[0181] S(x,y)=ax 2 +by 2 +cx+dy+exy+f (25)

[0182]

[0183] Among them, S(x, y) is the CL value fitting function of all pixels in each ROI, a, b, c, d, e, f are coefficients; x c is the x coordinate of the pose marker, y c is the y coordinate of the pose identifier.

[0184] Continue to read Figure 3 , in step 309, in response to the target posture and the actual posture satisfying the error detection condition, a control signal related to the fault is generated. After obtaining the target posture and the actual posture of the end of the actuator arm, the control device determines the posture error of the end of the actuator arm to determine whether the actuator arm has correctly reached the position and posture expected by the operator, and then determines whether the robot system has a fault. In some embodiments, when the target posture and the actual posture of the actuator arm meet the error detection condition (for example, greater than or equal to the error threshold), the control device determines that the actuator arm has not correctly moved to the position and posture corresponding to the main manipulator, and sends a control signal related to the fault. For example, the control device can send a first alarm signal, which indicates that the control of the actuator arm has a fault.

[0185] In some embodiments, the control device can obtain multiple sets of target and actual poses of the actuator arm in real time during teleoperation, and comprehensively determine the operating status of the actuator arm based on these multiple sets of target and actual poses. In some embodiments, the control device can determine the target and actual poses of the actuator arm end at a predetermined period, perform error detection on the actuator arm through multiple detection cycles, analyze the multiple sets of errors using mathematical statistics, and issue a fault-related control signal when error detection conditions are met.

[0186] For example, in the kth error detection cycle, the pose difference can be expressed as follows:

[0187]

[0188] in, is the position difference of the actuator arm during the kth error detection cycle, is the angle difference of the actuator arm during the kth error detection cycle, P t k is the target position of the actuator arm during the kth error detection cycle, R t k is the target posture of the actuator arm during the k-th error detection cycle, is the actual position of the actuator arm during the kth error detection cycle, R r k is the actual posture of the actuator arm during the kth error detection cycle, express and The corner between.

[0189] In some embodiments, the control device may store the errors obtained in multiple detection cycles in a memory and accumulate these errors. When the accumulated value of the errors meets the error detection condition (for example, exceeds a threshold), a control signal related to the fault is issued.

[0190] In some embodiments, method 300 also includes receiving status information of at least one driving device for driving the execution arm in response to the target posture and the actual posture satisfying an error detection condition, and issuing a second alarm signal in response to the status information and the driving information of at least one driving device satisfying a fault detection condition, the second alarm signal indicating that a fault has occurred in the driving device of the execution arm.

[0191] In some embodiments, the drive device is provided with a drive device sensor, which is coupled to the drive device and used to obtain status information of the drive device. For example, the drive device may include at least one drive motor, and the drive device sensor may include a potentiometer or an encoder, and the drive device sensor is coupled to the drive motor to record and output the status information of the motor. The control device sends drive information to at least one drive device based on the target posture of the end of the actuator arm, and receives status information of at least one drive device for driving the actuator arm through the drive device sensor. When the status information and the drive information meet the fault detection condition (for example, greater than or equal to the error threshold), a second alarm signal is issued to indicate that a fault has occurred in at least one drive device driving the actuator arm.

[0192] In some embodiments of the present disclosure, the present disclosure further provides a computer device, the computer device including a memory and a processor. The memory may be used to store at least one instruction, and the processor is coupled to the memory and is used to execute at least one instruction to perform some or all steps in the method of the present disclosure, such as Figure 3 、 Figure 4 、 Figure 9 、 Figure 10 、 Figure 11 and Figure 13Some or all of the steps in the method disclosed in.

[0193] Figure 15 FIG1 shows a schematic block diagram of a computer device 1500 according to some embodiments of the present disclosure. Figure 15 The computer device 1500 may include a central processing unit (CPU) 1501, a system memory 1504 including a random access memory (RAM) 1502 and a read-only memory (ROM) 1503, and a system bus 1505 connecting the various components. The computer device 1500 may also include an input / output system and a mass storage device 1507 for storing an operating system 1513, application programs 1514, and other program modules 1515. The input / output device includes an input / output controller 1510 mainly composed of a display 1508 and input devices 1509.

[0194] The mass storage device 1507 is connected to the central processing unit 1501 through a mass storage controller (not shown) connected to the system bus 1505. The mass storage device 1507 or computer-readable medium provides non-volatile storage for the computer device. The mass storage device 1507 may include a computer-readable medium (not shown) such as a hard disk or a Compact Disc Read-Only Memory (CD-ROM) drive.

[0195] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, flash memory or other solid-state storage technologies, CD-ROM, or other optical storage, magnetic cassettes, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media is not limited to the aforementioned types. The above-mentioned system memory and mass storage devices may be collectively referred to as memory.

[0196] The computer device 1500 can be connected to a network 1512 via a network interface unit 1511 connected to the system bus 1505 .

[0197] The system memory 1504 or the mass storage device 1507 is further configured to store one or more instructions. The central processing unit 1501 implements all or part of the steps of the method in some embodiments of the present disclosure by executing the one or more instructions.

[0198] In some embodiments of the present disclosure, the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, and the at least one instruction is executed by a processor to enable a computer to perform some or all steps of the method of some embodiments of the present disclosure, such as Figure 3 、 Figure 4 、 Figure 9 、 Figure 10 、 Figure 11 and Figure 13 Examples of computer-readable storage media include storage for computer programs (instructions), such as read-only memory (ROM), random access memory (RAM), compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage devices.

[0199] Figure 16 Schematic diagram 1600 of a robotic system according to some embodiments of the present disclosure is shown. Figure 16 As shown, a robotic system 1600 includes a master manipulator 1610, a control device 1620, a drive device 1660, a slave tool 1650, and an image acquisition device 1670. The master manipulator 1610 includes a robotic arm, a handle disposed on the robotic arm, and at least one master manipulator sensor disposed at at least one joint of the robotic arm. The at least one master manipulator sensor is used to obtain joint information of the at least one joint. In some embodiments, the master manipulator 1610 comprises a six-degree-of-freedom robotic arm, with a master manipulator sensor disposed at each joint of the six-degree-of-freedom robotic arm. Joint information (e.g., joint angle data) is generated by the master manipulator sensor at each joint. In some embodiments, the master manipulator sensor utilizes a potentiometer and / or encoder. An actuator arm 1640 is disposed on the slave tool 1650. In some embodiments, the actuator arm 1640 comprises a multi-segmented, continuous-body deformable arm. Multiple identifiers may be formed or disposed on the distal end 1630 of the actuator arm 1640, including multiple position identifiers and at least one composite identifier. An actuator may be disposed at the distal end of the distal end 1630. The image acquisition device 1670 can be used to acquire positioning images of the actuator arm 1640. The driving device 1660 is used to drive the actuator arm 1640. At least one driving device sensor is coupled to at least one driving device and is used to obtain driving information. The control device 1620 is in communication with the main operator 1610, at least one driving device 1660, and the image acquisition device 1670, and is configured to perform some or all of the steps in the method of some embodiments of the present disclosure, such as Figure 3 、 Figure 4 、 Figure 9 、 Figure 10 、 Figure 11 and Figure 13 Some or all of the steps in the method disclosed in.

[0200] Robots have very high requirements for operational accuracy and human-computer interaction experience. During the operation of the robot system, if the actuator arm cannot move to the target position and posture accurately and quickly, it will reduce the operator's operating experience and may even cause operation failure, creating unnecessary risks. In the embodiments of the present disclosure, by detecting the actual posture of the operating arm and comparing it with the target posture of the actuator arm expected by the operator in real time, the existing fault risk can be discovered. The embodiments of the present disclosure can improve the operability and safety of the robot system and reduce the operational risks caused by the posture error of the actuator arm during the operation of the robot system.

[0201] Note that the above are only exemplary embodiments of the present disclosure and the technical principles used. Those skilled in the art will understand that the present disclosure is not limited to the specific embodiments herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present disclosure. Therefore, although the present disclosure has been described in more detail through the above embodiments, the present disclosure is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present disclosure, and the scope of the present disclosure is determined by the scope of the appended claims.

Claims

1. An error detection method, characterized in that: include: Obtain the target pose of the end of the actuator arm; Acquire positioning images; In the positioning image, a plurality of markers located on the distal end of the actuator arm are identified, wherein the plurality of markers include a plurality of posture markers for identifying postures and at least one composite marker for identifying posture and angle; determining an actual posture of the end of the actuator arm based on the at least one composite identifier and the plurality of posture identifiers; as well as In response to the target posture and the actual posture satisfying an error detection condition, generating a control signal related to a fault, wherein the control signal related to the fault includes a first alarm signal, and the first alarm signal indicates that a fault occurs in the control of the actuator arm; The generation of a control signal related to the fault further includes: In response to the target posture and the actual posture satisfying an error detection condition, receiving status information of at least one driving device for driving the actuator arm; and In response to the state information and the driving information of the at least one driving device satisfying a fault detection condition, issuing a second alarm signal, wherein the second alarm signal indicates that a fault has occurred in the driving device of the actuator arm; Determining the actual position of the end of the actuator arm includes: Determining three-dimensional coordinates of the at least one composite identifier and the plurality of pose identifiers in an identifier coordinate system; determining, based on the at least one composite identifier, a roll angle of the identifier coordinate system relative to the end coordinate system of the actuator arm; Determining the three-dimensional coordinates of the at least one composite identifier and the multiple pose identifiers in the end coordinate system of the actuator arm based on the roll angle of the identifier coordinate system relative to the end coordinate system of the actuator arm and the three-dimensional coordinates of the at least one composite identifier and the multiple pose identifiers in the identifier coordinate system; and Based on the two-dimensional coordinates of the at least one composite identifier and the multiple posture identifiers in the positioning image and the three-dimensional coordinates in the end coordinate system of the execution arm, the posture of the end coordinate system of the execution arm relative to the reference coordinate system is determined as the actual posture.

2. The method according to claim 1, wherein Obtaining the target pose of the end of the actuator arm includes: Determine the current pose of the master manipulator; and Based on the current posture of the master manipulator and the posture relationship between the master manipulator and the end of the actuator arm, a target posture of the end of the actuator arm is determined.

3. The method according to claim 2, wherein The posture relationship includes at least one of the following: The position change of the end of the actuator arm in the reference coordinate system is proportional to the position change of the main manipulator in the reference coordinate system; or The posture change amount of the end of the actuator arm in the reference coordinate system is consistent with the posture change amount of the main manipulator in the reference coordinate system; or The posture of the end of the actuator arm in the reference coordinate system is consistent with the posture of the main manipulator in the reference coordinate system.

4. The method according to claim 2, characterized in that Also includes: determining a previous pose of the master manipulator; Determining a starting position of the end of the actuator arm; as well as The target pose is determined based on the previous pose and the current pose of the master manipulator and the starting pose of the end of the actuator arm.

5. The method according to claim 2, characterized in that Also includes: Determining the current position of the handle of the master manipulator relative to the master manipulator base coordinate system; determining a previous position of the handle relative to a base coordinate system of the master manipulator; Determining a starting position of the end of the actuator arm relative to the actuator arm base coordinate system; as well as Based on the previous position and current position of the handle relative to the main manipulator base coordinate system, the transformation relationship between the execution arm base coordinate system and the main manipulator base coordinate system, and the starting position of the end of the execution arm relative to the execution arm base coordinate system, the target position of the end of the execution arm relative to the execution arm base coordinate system is determined.

6. The method according to claim 2, characterized in that Also includes: Determining a current posture of the handle of the master manipulator relative to a base coordinate system of the master manipulator; determining a previous pose of the handle relative to a base coordinate system of the master manipulator; Determining a starting posture of the end of the actuator arm relative to the actuator arm base coordinate system; as well as Based on the previous posture and current posture of the handle relative to the main manipulator base coordinate system, the transformation relationship between the execution arm base coordinate system and the main manipulator base coordinate system, and the starting posture of the end of the execution arm relative to the execution arm base coordinate system, the target posture of the end of the execution arm relative to the execution arm base coordinate system is determined.

7. The method according to claim 5 or 6, characterized in that Also includes: Based on the transformation relationship between the actuator arm base coordinate system and the camera coordinate system, the transformation relationship between the camera coordinate system and the display coordinate system, and the transformation relationship between the display coordinate system and the main manipulator base coordinate system, the transformation relationship between the actuator arm base coordinate system and the main manipulator base coordinate system is determined.

8. The method according to claim 7, characterized in that The execution arm base coordinate system and the camera coordinate system have a predetermined transformation relationship.

9. The method according to claim 2, wherein Also includes: receiving current joint information of at least one joint of the master manipulator; as well as Based on current joint information of at least one joint of the master manipulator, a current posture of the master manipulator is determined.

10. The method according to claim 5, characterized in that Also includes: receiving previous joint information of at least one joint of the master manipulator; determining a previous pose of the master manipulator based on previous joint information of at least one joint of the master manipulator; as well as The actual posture of the end of the execution arm obtained in the last detection cycle is received as the starting posture of the end of the execution arm.

11. The method according to claim 1, wherein Also includes: determining a plurality of candidate markers from the positioning image; identifying a first identifier among the plurality of identifiers from the plurality of candidate identifiers; as well as Using the first identifier as a starting point, search for other identifiers.

12. The method according to claim 11, characterized in that Also includes: In response to identifying the composite identifier, other identifiers are identified based on a posture identifier pattern matching template.

13. The method according to claim 11, characterized in that The identification includes an identification pattern and identification pattern corner points in the identification pattern, and the method further includes: determining a region of interest in the scout image; Dividing the region of interest into a plurality of sub-regions; Determine the pixel with the largest corner likelihood value in each of the sub-regions to form a pixel set; Determine a pixel with the largest corner point likelihood value among the multiple candidate markers as a candidate marker pattern corner point; and The identification pattern matching template is matched with the identification pattern at the corner point position of the candidate identification pattern to identify the first identification.

14. The method according to claim 13, characterized in that Also includes: In response to a matching failure, a pixel with a maximum corner point likelihood value among the remaining pixels in the pixel set is determined as a candidate marker pattern corner point.

15. The method according to claim 13, characterized in that Also includes: Using the first identifier as a starting point, searching for the second identifier; Determining a search direction based on the first identifier and the second identifier; as well as The first identifier or the second identifier is used as a starting point to search for identifiers in the search direction.

16. The method according to claim 15, wherein Also includes: In response to the search distance being greater than the search distance threshold, determining a pixel having a maximum corner likelihood value among the remaining pixels in the pixel set as a candidate marker pattern corner point; as well as The identification pattern matching template is matched with the identification pattern at the corner point position of the candidate identification pattern to identify the first identification.

17. The method according to claim 15, characterized in that Also includes: In response to the number of recognized markers being greater than or equal to a marker number threshold, the actual posture is determined based on the recognized markers.

18. The method according to claim 15, wherein Also includes: In response to the number of identified markers being less than a marker number threshold, determining a pixel having a maximum corner likelihood value among the remaining pixels in the pixel set as a candidate marker pattern corner point; as well as The identification pattern matching template is matched with the identification pattern at the corner point position of the candidate identification pattern to identify the first identification.

19. According to the method described in any one of claims 1, 11-18, a positioning label is provided on the outer surface of the columnar part at the end of the execution arm, and the positioning label includes a plurality of identification patterns, and the plurality of identification patterns include a plurality of different composite identification patterns and a plurality of posture identification patterns, and the plurality of different composite identification patterns and the plurality of posture identification patterns are located in the same pattern distribution band. 20 . The method according to claim 19 , wherein the N consecutive identification patterns in the plurality of identification patterns include at least one composite identification pattern, wherein the composite identification pattern is different from the posture identification pattern, and 2≤N≤4.

21. The method according to any one of claims 1-6, 8-18, and 20, characterized in that Also includes: The target posture and actual posture of the end of the actuator arm are determined in a predetermined period, so as to perform error detection on the actuator arm in real time through multiple detection cycles.

22. A computer device comprising: A memory for storing at least one instruction; as well as A processor is coupled to the memory and configured to execute the at least one instruction to perform the error detection method according to any one of claims 1-21.

23. A computer-readable storage medium, configured to store at least one instruction, wherein when the at least one instruction is executed by a computer, the computer is caused to perform the error detection method according to any one of claims 1 to 21.

24. A robotic system comprising: a main manipulator, comprising a robotic arm, a handle provided on the robotic arm, and at least one main manipulator sensor provided at at least one joint of the robotic arm, the at least one main manipulator sensor being configured to obtain joint information of the at least one joint; An execution arm, wherein a plurality of identifiers are provided at the end of the execution arm, wherein the plurality of identifiers include a plurality of posture identifiers and at least one composite identifier; at least one driving device for driving the actuator arm; at least one drive device sensor coupled to the at least one drive device and configured to obtain status information of the at least one drive device; An image acquisition device, used for acquiring a positioning image of the execution arm; as well as A control device is configured to be connected to the main operator, the at least one driving device, the at least one driving device sensor, and the image acquisition device to execute the error detection method according to any one of claims 1-21.

Images