A method, apparatus, device, and medium for attitude control of a target control object.

Abstract

Embodiments of the present application provide a target control object posture control method, device, equipment and medium, the target control object includes at least one joint axis, each joint axis is hung on one control joint axis on the mechanical arm, and the method comprises: when each control joint axis completes rotation control on the corresponding joint axis of the target control object based on the angle control instruction of the current time, obtaining the binocular endoscope image of the target control object under the current pose and the encoder value corresponding to each control joint axis;Based on the binocular endoscope image and each encoder value, determine the actual observation angle corresponding to each joint axis;Based on the actual observation angle, adjust the angle control instruction of the control joint axis at the next time, so that the control joint axis controls the corresponding joint axis to rotate based on the adjusted angle control instruction, improve the positioning accuracy of the target control object.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a method, apparatus, device, and medium for controlling the posture of a target controlled object. Background Technology

[0002] With the advancement of science and technology, surgical robotic arms have gradually been developed in the field of surgery. Surgical scenarios place extremely high demands on the absolute positioning accuracy of surgical instruments controlled by these robotic arms. Especially in complex surgical procedures, even minute deviations in the end effector can have a significant impact on the surgical outcome. Therefore, the precision of end-effector posture control is crucial to the success rate of surgery and the safety of the patient.

[0003] Currently, in surgical robot systems, the posture control of surgical instruments at the end effector mainly relies on parameters fed back from the robotic arm's sensors. However, due to various factors such as the rigidity of the robotic arm, joint clearance, and the decay of the surgical instrument's wire life, errors often exist in the actual control process, resulting in significant deviations in the posture of the surgical instrument's end effector, making it difficult to meet the high-precision positioning requirements of surgical instruments. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and medium for attitude control of a target controlled object, which eliminates error accumulation and improves the positioning accuracy of the target controlled object by using an attitude control compensation mechanism based on attitude errors fed back from binocular endoscope images and attitude errors fed back from joint encoder values.

[0005] In a first aspect, embodiments of the present invention provide a method for posture control of a target control object, wherein the target control object includes at least one joint axis, and each joint axis is mounted on a control joint axis of a robotic arm, the method comprising:

[0006] When each control joint axis completes rotation control of the corresponding joint axis of the target controlled object based on the angle control command at the current moment, the binocular endoscope image of the target controlled object in the current pose and the encoder value corresponding to each control joint axis are acquired.

[0007] Based on the binocular endoscope images and the encoder values, the actual observation angle corresponding to each joint axis is determined.

[0008] Based on the actual observation angle, the angle control command of the control joint axis at the next moment is adjusted so that the control joint axis controls the corresponding joint axis to rotate based on the adjusted angle control command.

[0009] Secondly, embodiments of the present invention also provide a target control object posture control device, wherein the target control object includes at least one joint axis, and each joint axis is mounted on a control joint axis on a robotic arm, the device comprising:

[0010] The data acquisition module is used to acquire the binocular endoscope image of the target controlled object in its current pose and the encoder value corresponding to each control joint axis when each control joint axis completes rotation control of the corresponding joint axis of the target controlled object based on the angle control command at the current moment.

[0011] The observation angle determination module is used to determine the actual observation angle corresponding to each joint axis based on the binocular endoscope image and each encoder value;

[0012] The instruction adjustment module is used to adjust the angle control instruction of the control joint axis at the next moment based on the actual observation angle, so that the control joint axis controls the corresponding joint axis to rotate based on the adjusted angle control instruction.

[0013] Thirdly, embodiments of the present invention also provide an electronic device, the electronic device comprising:

[0014] One or more processors;

[0015] Storage device for storing one or more programs.

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the target control object attitude control method as described in any embodiment of the present invention.

[0017] Fourthly, embodiments of the present invention also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the target control object attitude control method as described in any of the embodiments of the present invention.

[0018] The technical solution of this invention includes a target control object comprising at least one joint axis, each joint axis being mounted on a control joint axis of a robotic arm. The method includes: when each control joint axis completes rotational control of the corresponding joint axis of the target control object based on the current angle control command, acquiring a binocular endoscopic image of the target control object in its current pose and encoder values ​​corresponding to each control joint axis; furthermore, determining the actual observation angle corresponding to each joint axis based on the binocular endoscopic image and encoder values; and then adjusting the angle control command of the control joint axis for the next moment based on the actual observation angle, so that the control joint axis controls the corresponding joint axis to rotate based on the adjusted angle control command. The technical solution provided in this embodiment can detect in real time the posture errors caused by joint clearance and instrument wire lifespan decay of the target control object through the binocular endoscopic image of the target control object in its current pose. Furthermore, the posture control compensation mechanism based on the posture errors fed back from the binocular endoscopic image and the posture errors fed back from the joint encoder values ​​can eliminate error accumulation, improving the positioning accuracy of the target control object. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of exemplary embodiments of the present invention, the accompanying drawings used in describing the embodiments are briefly introduced below. Obviously, the accompanying drawings described are only a portion of the drawings of the embodiments to be described in this invention, and not all of the drawings. For those skilled in the art, other drawings can be obtained from these drawings without any creative effort.

[0020] Figure 1 A flowchart illustrating a target control object attitude control method provided in an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram illustrating the specific implementation process of a target control object attitude control method according to an embodiment of the present invention;

[0022] Figure 3 A flowchart illustrating another target control object attitude control method provided in an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram illustrating the process of determining the disparity map and key point location information in an embodiment of the present invention;

[0024] Figure 5 This is a schematic diagram of the parallax depth relationship of the binocular endoscope involved in the embodiments of the present invention;

[0025] Figure 6 This is a schematic diagram of the structure of a target control object attitude control device provided in an embodiment of the present invention;

[0026] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0027] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0028] Example 1

[0029] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0030] Before introducing the embodiments of the present invention, the application scenarios of the embodiments of the present invention can be described first. For different surgical procedures, the operating postures of the surgical robotic arm vary widely, and the surgical scenario requires extremely high absolute positioning accuracy of the surgical robot. In this case, the posture control of the target control object driven by the surgical robotic arm can be performed based on the technical method provided in the embodiments of the present invention to improve the positioning accuracy of the target control object.

[0031] Figure 1 This is a flowchart illustrating a target control object posture control method provided in an embodiment of the present invention. This embodiment is applicable to situations where the posture of a target control object needs to be precisely controlled. The method can be executed by a target control object posture control device, which can be implemented in the form of software and / or hardware. The hardware can be an electronic device such as a server, and the electronic device can execute the target control object posture control method provided in this technical solution.

[0032] In this embodiment, the target control object includes at least one joint axis, and each joint axis is mounted on a control joint axis on the robotic arm.

[0033] A robotic arm is a device that can mimic the movements of a human arm, and it includes multiple control joint axes. The target controlled object is the instrument held at the end effector of the robotic arm. For example, if surgery is to be performed using a robotic arm, the target controlled object could be the surgical instrument held at the end effector of the surgical robotic arm. The target controlled object includes multiple joint axes. These joint axes can include at least one of yaw, pitch, and rotation axes. In practical applications, the joint axes on the target controlled object can be mounted on the control joint axes of the robotic arm. The joint axes can rotate under the drive of the control joint axes, allowing the target controlled object to perform specific tasks, such as object grasping, suturing, or cutting.

[0034] like Figure 1 As shown, the attitude control method for the target controlled object includes:

[0035] S110. When each control joint axis completes rotation control of the corresponding joint axis of the target controlled object based on the angle control command at the current moment, acquire the binocular endoscope image of the target controlled object in the current pose and the encoder value corresponding to each control joint axis.

[0036] Angle control commands are used to precisely control the rotation of control joint axes on a robotic arm to a specified angle. This control typically involves the rotation control of a motor, achieving precise control of the motor's rotation angle through an inner current loop, an outer speed loop, and an outermost position loop. In this embodiment, upon receiving an operation control command requiring the target controlled object to perform a certain operation, the operation control command can be decomposed into angle control commands corresponding to each control joint axis. Therefore, the control joint axis can execute the corresponding operation based on the angle control commands. Consequently, the joint axis of the target controlled object, mounted on the control joint axis, can complete the corresponding operation under the drive of the control joint axis.

[0037] The current pose refers to the position and orientation of the target controlled object after completing the corresponding operation based on the angle control command at the current moment. A binocular endoscopic image refers to an image acquired through a binocular endoscopic imaging system. This imaging system includes a light source module, two imaging modules, and two camera modules. The light source module illuminates the object under test (i.e., the target controlled object), while the two camera modules simulate two human eyes to acquire different images, using parallax to calculate depth. This system can provide two different viewpoints, obtaining the object's depth information by calculating the difference between the two viewpoints, thereby achieving high-precision measurement.

[0038] In this embodiment, a joint encoder is configured in each control joint axis of the robotic arm. The encoder values ​​are data extracted from the joint encoder. A joint encoder is a sensor used to measure the rotational angle of a robot joint; it can accurately measure the position and movement of the joint. The joint encoder measures the joint position by generating pulse signals as the joint rotates. These pulse signals can then be used to calculate the position and movement of the robotic arm's end effector (i.e., the target controlled object).

[0039] In this embodiment, the robotic arm's end effector holds the target control object, which can be moved to a designated position by the robotic arm. This process is the process of the robotic arm rotating the target control object. During operation, the robotic arm receives control commands that correspond to the angle control commands of each control joint axis. Each control joint axis then performs a rotational action according to the corresponding angle control commands. The angle control commands can be issued to the robotic arm by the main controller.

[0040] Specifically, during the rotational control of the target object by the robotic arm, the robotic arm can receive angle control commands corresponding to the current moment at different control times. When each control joint axis of the robotic arm completes the rotational control of the corresponding joint axis of the target object based on the angle control command at the current moment, the pose of the target object has changed relative to the previous control cycle. At this time, a binocular endoscope image of the target object in the current pose can be acquired through binocular endoscopy, and the corresponding encoder value can be obtained from the joint encoder of each control joint axis.

[0041] For example, the target controlled object is a surgical instrument, which includes two joint axes, namely joint axis 1 and joint axis 2. Joint axis 1 is attached to the control joint axis A of the robotic arm, and joint axis 2 is attached to the control joint axis B of the robotic arm. At the current time T0, the angle control command corresponding to control joint axis A is θ01, and the angle control command corresponding to control joint axis B is θ02. When control joint axis A completes the rotation control of joint axis 1 based on θ01, and control joint axis B completes the rotation control of joint axis 2 based on θ02, a binocular endoscope is used to acquire binocular endoscopic images of the surgical instrument in the current pose, and the encoder value δ1 corresponding to control joint axis A and the encoder value δ2 corresponding to control joint axis B are obtained.

[0042] S120. Based on the binocular endoscope images and the values ​​of each encoder, determine the actual observation angle corresponding to each joint axis.

[0043] The actual observation angle is the actual angle value corresponding to the joint axis of the target control object, which is determined by using a specific method.

[0044] Specifically, binocular endoscopic images can determine the true pose of the target controlled object after the current control cycle. From this true pose, the first actual observation angle corresponding to each joint axis of the target controlled object can be calculated. By establishing the angular mapping relationship between each control joint axis of the robotic arm and the corresponding joint axis of the target controlled object, and based on the encoder values ​​of the control joint axes, the second actual observation angle corresponding to each joint axis of the target controlled object can be calculated. Furthermore, the first and second actual observation angles can be fused to obtain the actual observation angle corresponding to each joint axis.

[0045] Based on the above example, the first actual observation angle α1 corresponding to joint axis 1 and the first actual observation angle α2 corresponding to joint axis 2 in the target controlled object are calculated using binocular endoscopic images. The second actual observation angle β1 corresponding to joint axis 1 and the second actual observation angle β2 corresponding to joint axis 2 in the target controlled object are calculated using encoder values ​​controlling joint axis A and joint axis B. Therefore, the actual observation angle γ1 corresponding to joint axis 1 in the target controlled object can be determined based on α1 and β1; and the actual observation angle γ2 corresponding to joint axis 2 in the target controlled object can be determined based on α2 and β2.

[0046] S130. Adjust the angle control command of the corresponding joint axis at the next moment based on the actual observation angle, so that the joint axis controls the corresponding joint axis to rotate based on the adjusted angle control command.

[0047] Based on the above embodiments, after each control joint axis completes rotational control of the corresponding joint axis of the target controlled object based on the angle control command at the current moment, each control joint axis will then control the joint axis of the target controlled object according to the angle control command at the next moment. Before controlling the joint axis of the target controlled object according to the angle control command at the next moment, the angle control command at the next moment can be adjusted based on the actual observed angle. This allows the control joint axis to control the rotation of the corresponding joint axis according to the adjusted angle control command, thereby achieving the goal of matching the actual position information of the target controlled object with the desired position information.

[0048] Based on the above example, the angle control command for joint axis A at the next moment is θ11. This angle control command θ11 can be adjusted using proportional-integral-derivative (PID) control based on the actual observed angle γ1 corresponding to joint axis 1, resulting in the adjusted angle control command θ11. The angle control command for joint axis B at the next moment is θ12. This angle control command θ12 can be adjusted using proportional-integral-derivative (PID) control based on the actual observed angle γ2 corresponding to joint axis 2, resulting in the adjusted angle control command θ12. Thus, joint axis A is controlled to rotate based on θ'11, and joint axis B is controlled to rotate based on θ12.

[0049] In this embodiment, before the control joint axis on the robotic arm rotates the joint axis of the target controlled object based on the angle control command at the next moment, the angle control command at the next moment is adjusted according to the actual observed angle of the joint axis. The adjusted angle control command is then sent to the robotic arm so that the robotic arm can drive the target controlled object to rotate based on the angle control command. This is because if the robotic arm performs rotation control based on the unadjusted angle control command, the target controlled object should eventually reach the target position A. However, due to the influence of various factors such as the rigidity of the robotic arm, joint clearance, and the decay of the lifespan of the surgical instrument wire, the target controlled object may only reach position A1 if rotation control is performed based on the unadjusted angle control command. Therefore, in order to make the final position A1 of the target controlled object as close as possible to the target position A, error compensation can be performed on the angle control command beforehand.

[0050] The following example illustrates the target control object attitude control method provided in this embodiment. A schematic diagram of the specific implementation process of the target control object attitude control method can be found below. Figure 2 .like Figure 2As shown, the process involves five steps: First, the robotic arm controls the rotation of each joint axis of the target object based on the current angle control command. Second, as each joint axis of the target object completes its rotation control, a binocular endoscopic image of the target object in its current pose is acquired, and simultaneously, the encoder value corresponding to each control joint axis is acquired. Third, the binocular endoscopic image is processed using a pose recognition algorithm to obtain the first actual observation angle corresponding to each joint axis. Simultaneously, based on the encoder value and a preset angle mapping relationship, the second actual observation angle corresponding to each joint axis is obtained. Fourth, based on the first and second actual observation angles, the actual observation angle corresponding to each joint axis in the target object is determined. Fifth, the angle control command for the next control joint axis is adjusted according to the actual observation angle, so that the control joint axis rotates based on the adjusted angle control command.

[0051] The technical solution of this invention includes a target control object comprising at least one joint axis, each joint axis being mounted on a control joint axis of a robotic arm. The method includes: when each control joint axis completes rotational control of the corresponding joint axis of the target control object based on the current angle control command, acquiring a binocular endoscopic image of the target control object in its current pose and encoder values ​​corresponding to each control joint axis; furthermore, determining the actual observation angle corresponding to each joint axis based on the binocular endoscopic image and encoder values; and then adjusting the angle control command of the control joint axis for the next moment based on the actual observation angle, so that the control joint axis controls the corresponding joint axis to rotate based on the adjusted angle control command. The technical solution provided in this embodiment can detect in real time the posture errors caused by joint clearance and instrument wire lifespan decay of the target control object through the binocular endoscopic image of the target control object in its current pose. Furthermore, the posture control compensation mechanism based on the posture errors fed back from the binocular endoscopic image and the posture errors fed back from the joint encoder values ​​can eliminate error accumulation, improving the positioning accuracy of the target control object.

[0052] Example 2

[0053] Figure 3 This is a flowchart illustrating a target control object attitude control method provided in an embodiment of the present invention. Based on the foregoing embodiments, the specific implementation of 120 is described in detail. For specific implementation methods, please refer to the technical solution of this embodiment. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here.

[0054] like Figure 3 As shown, the method specifically includes the following steps:

[0055] S210. When each control joint axis completes rotation control of the corresponding joint axis of the target controlled object based on the angle control command at the current moment, acquire the binocular endoscope image of the target controlled object in the current pose and the encoder value corresponding to each control joint axis.

[0056] S220. Based on the binocular endoscope image and the initial pose 3D model of the target control object, determine the first actual observation angle corresponding to each joint axis.

[0057] The initial pose 3D model refers to the 3D data model of the target control object in its initial pose. In the initial pose, the angle values ​​of each joint axis of the target control object are initial values.

[0058] In this embodiment, the specific implementation of determining the initial pose 3D model of the target control object may include: when the target control object is in the initial pose, scanning the target control object with a scanner to obtain a 3D structural model of the target control object; when the target control object is within the field of view of the binocular endoscope, determining the 3D position coordinates of each geometric body in the 3D model structure based on the homogeneous transformation matrix of the target control object relative to the binocular endoscope coordinate system; and constructing the initial pose 3D model based on the 3D structural model and the 3D position coordinates of each geometric body.

[0059] Among them, a three-dimensional structural model refers to a physical entity (i.e., the target control object) represented by a series of geometric objects (such as structural points, triangles, lines, arcs, cubes, etc.) connected in three-dimensional space.

[0060] In this embodiment, with the target control object in its initial pose, a scanner, such as a 3D scanner, can be used to scan the target control object to obtain a 3D structural model corresponding to the target control object. Further, the target control object is attached to a robotic arm, and the target control object in its initial pose is moved into the field of view of a binocular endoscope. Then, the pose data of the target control object is generated using the Monte Carlo method. The homogeneous transformation matrix of the target control object relative to the binocular endoscope coordinate system is calculated using the kinematic equations of the robotic arm. Based on the homogeneous transformation matrix of the target control object, the 3D position coordinates (i.e., spatial coordinates in the coordinate system of the binocular endoscope) corresponding to each geometric object in the 3D structural model are derived. Based on this, for the 3D structural model, the 3D position coordinates corresponding to each geometric object are assigned to the corresponding geometric object, thus constructing the initial pose 3D model.

[0061] Specifically, the current pose of the target control object can be determined based on the binocular endoscopic images. Based on the relative positional relationship between key components in the initial pose 3D model and the current pose of the target control object, the target 3D model of the target control object in the current pose can be derived. Then, based on the spatial coordinates of multiple key points in the target 3D model, the first actual observation angle corresponding to each joint axis in the target control object can be calculated.

[0062] Optionally, the specific implementation method for determining the first actual observation angle corresponding to each joint axis based on the binocular endoscopic image and the initial pose 3D model of the target control object may include:

[0063] S2201. Based on binocular endoscopic images, determine the homogeneous transformation matrix corresponding to at least one preset key part of the target control object in its current pose.

[0064] The preset key components include the joint axes and edge vertices of the target control object. The binocular endoscopic images include a left-eye endoscopic image and a right-eye endoscopic image.

[0065] Specifically, based on binocular endoscopic images, the homogeneous transformation matrix corresponding to at least one preset key part of the target control object in its current pose is determined, including:

[0066] (1) Input the left and right endoscope images into the pre-trained disparity estimation model to obtain the disparity map of the target control object in the current pose.

[0067] (2) Input the disparity map into the pre-trained key point recognition model to obtain the key point position information corresponding to at least one preset key part of the target control object in the current pose.

[0068] In this embodiment, a schematic diagram illustrating the process of determining the disparity map and key point location information is provided. Figure 4 ,like Figure 4As shown, the disparity estimation model includes a feature extraction layer and a disparity estimation layer. First, the left and right endoscopic images are input into the feature extraction layer, where 2D convolutional kernels are used to extract features. Then, the images are fed into the disparity estimation layer, where a 3D convolutional layer is used to perform cost matching calculations to reconstruct the disparity map from the features. Optionally, this disparity map is coarse. This coarse disparity map can be input into a refinement network to obtain a refined disparity map, which is then used to update the original disparity map. The refinement network uses hierarchical and edge-sensitive weights to optimize and adjust the input coarse disparity map. For example, bilinear upsampling and convolution can be used to construct the refinement network. Meanwhile, based on the obtained disparity map, the disparity map can be input into a pre-trained key point recognition model to obtain the position information of multiple key points (i.e., the x-axis and y-axis coordinates of the key points) corresponding to each preset key part of the target control object in the current pose. The key point recognition model can use a pose recognition network model, such as deepPose or openPose, to mark the x-axis and y-axis coordinates of the key points in the image.

[0069] (3) Based on the disparity map and the key point position information corresponding to each preset key part, determine the homogeneous transformation matrix corresponding to at least one preset key part of the target control object in the current pose.

[0070] Among them, the key point location information is the two-dimensional coordinates of the key points in the target fused image, and the target fused image is a fused image of the left endoscope image and the right endoscope image.

[0071] More specifically, determining the homogeneous transformation matrix corresponding to at least one preset key part of the target control object in the current pose, based on the disparity map and the key point position information corresponding to each preset key part, may include: determining the depth map corresponding to the disparity map based on the disparity map and the parameters of the binocular endoscope device; determining the depth information corresponding to each key point based on the depth map and the two-dimensional coordinates of each key point; for each key point, determining the three-dimensional spatial coordinates corresponding to each key point based on the two-dimensional coordinates of the key point and the corresponding depth information; and determining the homogeneous transformation matrix corresponding to each preset key part based on the three-dimensional spatial coordinates of at least four key points belonging to the same preset key part.

[0072] The parameters of the binocular endoscope device include the distance between the camera origins of the left and right cameras, the camera focal length, the first field corresponding to the left camera, and the second field corresponding to the right camera.

[0073] Based on this, and using the disparity map and binocular endoscope parameters, a depth map corresponding to the disparity map is determined, including: for each pixel in the disparity map, determining the first imaging point of the current pixel in the first field and the second imaging point of the current pixel in the second field; determining the disparity distance based on the first distance between the first imaging point and the left boundary of the first field, and the second distance between the second imaging point and the left boundary of the second field; determining the depth value corresponding to the current pixel based on the disparity distance of the current pixel, the distance from the camera origin, and the camera focal length; and determining the depth map corresponding to the disparity map based on the depth values ​​of each pixel.

[0074] In this embodiment, after obtaining the disparity map, it needs to be converted into a depth map to calculate the depth information of key points. The specific determination method is as follows: See the schematic diagram of the disparity-depth relationship of the binocular endoscope. Figure 5 ,like Figure 5 As shown, P is a point within the field of view (equivalent to a pixel in the disparity map), D is the depth of point P, and O... l With O r Let B be the origin of the coordinate system for the left and right cameras of the binocular endoscope, respectively. The distance between the camera origins of the left and right cameras is denoted as B, f is the camera focal length, and P is the distance between the camera origins of the left and right cameras. l The current pixel P is the first imaging point in the first field. r Let x be the second imaging point of the current pixel P in the second field. l x is the first distance between the first imaging point and the left boundary of the first field. r The second distance between the second imaging point and the left boundary of the second field can be expressed as S = x. l -x r In the disparity map, the pixel value of each pixel is S. The depth value of point P is calculated according to the principle of similar triangles, as shown in Formula 1.

[0075]

[0076] Using the same method, the depth value corresponding to each pixel in the disparity map can be determined, and thus the depth map corresponding to the disparity map can be determined based on the depth value of each pixel.

[0077] Furthermore, based on the obtained depth map, for a given keypoint, its two-dimensional coordinates can be used to determine which pixel in the depth map it corresponds to. The depth value of this pixel can then be used as the depth information of the keypoint. Using the same method, the depth information of each keypoint can be determined. Therefore, for each keypoint, based on its corresponding two-dimensional coordinates (x and y coordinates) and depth information (equivalent to z coordinates), its three-dimensional spatial coordinates (x, y, z) can be determined. Specifically, these three-dimensional spatial coordinates are relative to the coordinate system under the binocular endoscope.

[0078] Furthermore, since it is easy to determine which key points correspond to the preset key parts, the homogeneous transformation matrix of the coordinate system corresponding to each preset key part can be determined based on the three-dimensional spatial coordinates of at least four key points belonging to the same preset key part.

[0079] S2202. Based on the relative positional relationships between each preset key part and each homogeneous transformation matrix in the initial pose 3D model of the target control object, determine the target 3D model of the target control object in the current pose.

[0080] In this embodiment, the relative positional relationship between the preset key parts can be determined based on the three-dimensional position coordinates of the geometric objects corresponding to each preset key part in the initial pose three-dimensional model. For a specific target control object, the relative positional relationship between some preset key parts is determined. Based on this, the target three-dimensional model of the target control object in the current pose can be determined according to the homogeneous transformation matrix corresponding to each preset key part and the relative positional relationship between each preset key part in the initial pose three-dimensional model. That is, it can be understood as transforming the initial pose three-dimensional model of the target control object to the binocular endoscope coordinate system according to the current pose to obtain the target three-dimensional model.

[0081] S2203. Based on the three-dimensional position coordinates of each preset key part in the target three-dimensional model, determine the first actual observation angle corresponding to each joint axis.

[0082] In this embodiment, the phase position relationship between each preset key part can be calculated based on the three-dimensional position coordinates of key points in each preset key part of the target three-dimensional model, and then the first actual observation angle corresponding to each joint axis can be calculated.

[0083] S230. Based on the encoder values ​​and the preset angle mapping relationship, determine the second actual observation angle corresponding to each joint axis.

[0084] The preset angle mapping relationship is the angle mapping relationship between the joint axis of the target controlled object and the control joint axis of the robotic arm.

[0085] In this embodiment, the encoder value can represent the actual observed value of the control joint axis of the robotic arm. Therefore, based on a preset angle mapping relationship, the angle calculation value corresponding to the encoder value can be calculated. For a certain joint axis, this angle calculation value is the corresponding second actual observed angle. Using the same method, the second actual observed angle corresponding to each joint axis of the target controlled object can be determined.

[0086] S240. For each joint axis, the first actual observation angle and the second actual observation angle are fused to obtain the actual observation angle corresponding to the joint axis.

[0087] In this embodiment, for a given joint axis, the first and second actual observation angles can be fused using an extended Kalman filter to obtain the actual observation angle corresponding to the joint axis. Based on the same determination method, the actual observation angle corresponding to each joint axis can be obtained.

[0088] S250. Based on the actual observation angle, adjust the angle control command of the corresponding joint axis for the next moment, so that the joint axis controls the corresponding joint axis to rotate based on the adjusted angle control command.

[0089] The technical solution of this embodiment, when determining the actual observation angle corresponding to each joint axis based on binocular endoscopic images and encoder values, determines the first actual observation angle corresponding to each joint axis based on the binocular endoscopic images and the initial pose 3D model of the target controlled object, and determines the second actual observation angle corresponding to each joint axis based on encoder values ​​and a preset angle mapping relationship. The preset angle mapping relationship is the angle mapping relationship between the joint axis and the control joint axis. Thus, the first and second actual observation angles are fused for each joint axis to obtain the actual observation angle corresponding to the joint axis. The technical solution provided in this embodiment can eliminate error accumulation based on the attitude error feedback from the binocular endoscopic images and the attitude error feedback from the joint encoder values ​​through an attitude control compensation mechanism, thereby improving the positioning accuracy of the target controlled object.

[0090] Example 3

[0091] Figure 6 This is a schematic diagram of the structure of a target control object posture control device provided in an embodiment of the present invention. The target control object includes at least one joint axis, and each joint axis is mounted on a control joint axis on a robotic arm. The target control object posture control device includes: a data acquisition module 310, an observation angle determination module 320, and a command adjustment module 330.

[0092] The data acquisition module 310 is used to acquire the binocular endoscope image of the target control object in the current pose and the encoder value corresponding to each control joint axis when each control joint axis completes rotation control of the corresponding joint axis of the target control object based on the angle control command at the current moment.

[0093] The observation angle determination module 320 is used to determine the actual observation angle corresponding to each joint axis based on the binocular endoscope image and each encoder value.

[0094] The instruction adjustment module 330 is used to adjust the angle control instruction of the control joint axis at the next moment based on the actual observation angle, so that the control joint axis controls the corresponding joint axis to rotate based on the adjusted angle control instruction.

[0095] Based on the above embodiments, the observation angle determination module 320 includes:

[0096] The first angle determination submodule is used to determine the first actual observation angle corresponding to each joint axis based on the binocular endoscope image and the initial pose 3D model of the target control object.

[0097] The second angle determination submodule is used to determine the second actual observation angle corresponding to each joint axis based on each encoder value and a preset angle mapping relationship; wherein, the preset angle mapping relationship is the angle mapping relationship between the joint axis of the target controlled object and the control joint axis of the robotic arm;

[0098] The observation angle determination submodule is used to fuse the first actual observation angle and the second actual observation angle for each joint axis to obtain the actual observation angle corresponding to the joint axis.

[0099] Based on the above embodiments, the first angle determination submodule includes:

[0100] The transformation matrix determination unit is used to determine, based on the binocular endoscope image, the homogeneous transformation matrix corresponding to at least one preset key part of the target control object in the current pose; wherein, the preset key part includes each joint axis and edge vertex of the target control object;

[0101] The 3D model conversion unit is used to determine the target 3D model of the target control object in the current pose based on the relative positional relationship between each preset key part in the initial pose 3D model of the target control object and each homogeneous transformation matrix.

[0102] The first angle determination unit is used to determine the first actual observation angle corresponding to each joint axis based on the three-dimensional position coordinates corresponding to each preset key part in the target three-dimensional model.

[0103] Based on the above embodiments, the binocular endoscopic images include a left-eye endoscopic image and a right-eye endoscopic image. The transformation matrix determination unit based on the binocular endoscopic images includes:

[0104] The disparity map determination subunit is used to input the left endoscope image and the right endoscope image into a pre-trained disparity estimation model to obtain the disparity map of the target control object in the current pose.

[0105] The key point location determination subunit is used to input the disparity map into a pre-trained key point recognition model to obtain the key point location information corresponding to at least one preset key part of the target control object in the current pose;

[0106] The transformation matrix determination subunit is used to determine the homogeneous transformation matrix corresponding to at least one preset key part of the target control object in the current pose, based on the disparity map and the key point position information corresponding to each preset key part.

[0107] Based on the above embodiments, the key point location information is the two-dimensional coordinates of the key point in the target fused image, and the target fused image is the fused image of the left endoscope image and the right endoscope image. The transformation matrix determination subunit is specifically used to determine the depth map corresponding to the disparity map based on the disparity map and the parameters of the binocular endoscope device; determine the depth information corresponding to each key point based on the depth map and the two-dimensional coordinates of each key point; for each key point, determine the three-dimensional spatial coordinates corresponding to each key point based on the two-dimensional coordinates of the key point and the corresponding depth information; and determine the homogeneous transformation matrix corresponding to each preset key part based on the three-dimensional spatial coordinates of at least four key points belonging to the same preset key part.

[0108] Based on the above embodiments, the parameters of the binocular endoscope device include the distance between the camera origin of the left and right cameras, the camera focal length, the first field corresponding to the left camera, and the second field corresponding to the right camera. The transformation matrix determination subunit is specifically used to determine, for each pixel in the disparity map, the first imaging point of the current pixel in the first field and the second imaging point of the current pixel in the second field; determine the disparity distance based on the first distance between the first imaging point and the left boundary of the first field, and the second distance between the second imaging point and the left boundary of the second field; determine the depth value corresponding to the current pixel based on the disparity distance of the current pixel, the distance between the camera origin, and the camera focal length; and determine the depth map corresponding to the disparity map based on the depth values ​​of each pixel.

[0109] Based on the above embodiments, the observation angle determination module 320 further includes: a three-dimensional model determination submodule; the three-dimensional model determination submodule includes:

[0110] An initial model determination unit is used to scan the target control object with a scanner when the target control object is in an initial pose to obtain a three-dimensional structural model of the target control object; wherein the three-dimensional structural model includes multiple geometric bodies;

[0111] The position coordinate determination unit is used to determine the three-dimensional position coordinates of each geometric body in the three-dimensional model structure based on the homogeneous transformation matrix of the target control object relative to the binocular endoscope coordinate system when the target control object is located within the field of view of the binocular endoscope.

[0112] The three-dimensional model determination unit is used to construct an initial pose three-dimensional model based on the three-dimensional structural model and the three-dimensional position coordinates corresponding to each of the geometric objects.

[0113] The technical solution of this invention includes a target control object comprising at least one joint axis, each joint axis being mounted on a control joint axis of a robotic arm. The method includes: when each control joint axis completes rotational control of the corresponding joint axis of the target control object based on the current angle control command, acquiring a binocular endoscopic image of the target control object in its current pose and encoder values ​​corresponding to each control joint axis; furthermore, determining the actual observation angle corresponding to each joint axis based on the binocular endoscopic image and encoder values; and then adjusting the angle control command of the control joint axis for the next moment based on the actual observation angle, so that the control joint axis controls the corresponding joint axis to rotate based on the adjusted angle control command. The technical solution provided in this embodiment can detect in real time the posture errors caused by joint clearance and instrument wire lifespan decay of the target control object through the binocular endoscopic image of the target control object in its current pose. Furthermore, the posture control compensation mechanism based on the posture errors fed back from the binocular endoscopic image and the posture errors fed back from the joint encoder values ​​can eliminate error accumulation, improving the positioning accuracy of the target control object.

[0114] The target control object attitude control device provided in the embodiments of the present invention can execute the target control object attitude control method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0115] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of the present invention.

[0116] Example 4

[0117] Figure 7 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0118] like Figure 7 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0119] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0120] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as target control object attitude control methods.

[0121] In some embodiments, the target object attitude control method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the target object attitude control method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the target object attitude control method by any other suitable means (e.g., by means of firmware).

[0122] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0123] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0124] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0125] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0126] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0127] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0128] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0129] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

[0130] Example 5

[0131] This invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform a target control object posture control method. The target control object includes at least one joint axis, and each joint axis is mounted on a control joint axis of a robotic arm. The method includes:

[0132] When each control joint axis completes rotation control of the corresponding joint axis of the target controlled object based on the angle control command at the current moment, the binocular endoscope image of the target controlled object in the current pose and the encoder value corresponding to each control joint axis are acquired.

[0133] Based on the binocular endoscope images and the encoder values, the actual observation angle corresponding to each joint axis is determined.

[0134] Based on the actual observation angle, the angle control command of the control joint axis at the next moment is adjusted so that the control joint axis controls the corresponding joint axis to rotate based on the adjusted angle control command.

[0135] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0136] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0137] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0138] Computer program code for performing the operations of embodiments of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A target control object posture control device, the target control object including at least one joint axis, each of the joint axes being mounted on one control joint axis on a robot arm, characterized by, The method comprises: a data acquisition module configured to acquire a binocular endoscope image of a target control object in a current pose and encoder values corresponding to control joint axes of the target control object when the control joint axes complete rotation control of corresponding joint axes of the target control object based on angle control instructions at a current time; an observation angle determination module configured to determine actual observation angles corresponding to the joint axes based on the binocular endoscope image and the encoder values; an instruction adjustment module configured to adjust angle control instructions of the control joint axes at a next time based on the actual observation angles, so that the control joint axes control corresponding joint axes to rotate based on the adjusted angle control instructions; wherein the observation angle determination module comprises: a first angle determination submodule configured to determine first actual observation angles corresponding to the joint axes based on the binocular endoscope image and an initial pose three-dimensional model of the target control object; a second angle determination submodule configured to determine second actual observation angles corresponding to the joint axes based on the encoder values and a preset angle mapping relationship, wherein the preset angle mapping relationship is an angle mapping relationship between joint axes of the target control object and the control joint axes of the robot arm; an observation angle determination submodule configured to perform fusion processing on the first actual observation angles and the second actual observation angles for each of the joint axes to obtain actual observation angles corresponding to the joint axes.

2. An electronic device, comprising: An electronic device comprises: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement a target control object pose control method, the target control object comprises at least one joint axis, each joint axis is mounted on a control joint axis of a robot arm, and the target control object pose control method comprises: acquiring a binocular endoscope image of a target control object in a current pose and encoder values corresponding to control joint axes of the target control object when the control joint axes complete rotation control of corresponding joint axes of the target control object based on angle control instructions at a current time; determining first actual observation angles corresponding to the joint axes based on the binocular endoscope image and an initial pose three-dimensional model of the target control object; determining second actual observation angles corresponding to the joint axes based on the encoder values and a preset angle mapping relationship, wherein the preset angle mapping relationship is an angle mapping relationship between joint axes of the target control object and the control joint axes of the robot arm; performing fusion processing on the first actual observation angles and the second actual observation angles for each of the joint axes to obtain actual observation angles corresponding to the joint axes; adjusting angle control instructions of the control joint axes at a next time based on the actual observation angles, so that the control joint axes control corresponding joint axes to rotate based on the adjusted angle control instructions; the determination of the first actual observation angles corresponding to the joint axes based on the binocular endoscope image and the initial pose three-dimensional model of the target control object comprises: determine, based on the binocular endoscope images, a homogeneous transformation matrix corresponding to at least one preset key part of the target control object in the current pose; wherein the preset key part includes each joint axis and edge vertex of the target control object; determine, based on the relative positional relationship between each preset key part in the initial pose three-dimensional model of the target control object and each homogeneous transformation matrix, a target three-dimensional model of the target control object in the current pose; determine, based on the three-dimensional position coordinates corresponding to each preset key part in the target three-dimensional model, a first actual observation angle corresponding to each joint axis; The binocular endoscope images include left and right endoscope images. The method for determining the homogeneous transformation matrix corresponding to at least one preset key part of the target control object in the current pose based on the binocular endoscope images comprises: inputting the left and right endoscope images into a pre-trained disparity estimation model to obtain a disparity map of the target control object in the current pose; inputting the disparity map into a pre-trained key point recognition model to obtain key point position information corresponding to at least one preset key part of the target control object in the current pose; determining, based on the disparity map and the key point position information corresponding to each preset key part, a homogeneous transformation matrix corresponding to at least one preset key part of the target control object in the current pose; The key point position information is the two-dimensional coordinates of the key points in the target fusion image, and the target fusion image is a fusion image of the left and right endoscope images. The method for determining the homogeneous transformation matrix corresponding to at least one preset key part of the target control object in the current pose based on the disparity map and the key point position information corresponding to each preset key part comprises: determining, based on the disparity map and the binocular endoscope device parameters, a depth map corresponding to the disparity map; determining, based on the depth map and the two-dimensional coordinates of each key point, depth information corresponding to each key point; for each key point, determining a three-dimensional space coordinate corresponding to each key point based on the two-dimensional coordinates and the corresponding depth information of the key point; determining a homogeneous transformation matrix corresponding to each preset key part based on the three-dimensional space coordinates of at least four key points belonging to the same preset key part; The binocular endoscope device parameters include the camera origin distance, camera focal length, first field of view corresponding to the left camera, and second field of view corresponding to the right camera. The method for determining the depth map corresponding to the disparity map based on the disparity map and the binocular endoscope device parameters comprises: for each pixel point in the disparity map, determining a first imaging point of the current pixel point in the first field of view and a second imaging point of the current pixel point in the second field of view; determining a disparity distance based on a first distance between the first imaging point and the left boundary of the first field of view, and a second distance between the second imaging point and the left boundary of the second field of view; determine a depth value corresponding to the current pixel point based on the parallax distance of the current pixel point, the camera origin distance, and the camera focal length; determine a depth map corresponding to the disparity map based on the depth values of the pixel points; The target control object posture control method further includes determining an initial pose three-dimensional model of the target control object, including: When the target control object is in an initial pose, a scanner is used to scan the target control object to obtain a three-dimensional structure model of the target control object; wherein the three-dimensional structure model includes a plurality of geometric bodies; When the target control object is located within the field of view of the binocular endoscope, based on the homogeneous transformation matrix of the target control object relative to the binocular endoscope coordinate system, the three-dimensional position coordinates corresponding to each geometric body in the three-dimensional model structure are determined. Based on the three-dimensional structure model and the three-dimensional position coordinates corresponding to each geometric body, an initial pose three-dimensional model is constructed.

3. A storage medium containing computer-executable instructions, wherein: The computer executable instructions, when executed by a computer processor, are used to execute a target control object posture control method, the target control object including at least one joint axis, each joint axis being mounted on a control joint axis on a mechanical arm, the target control object posture control method including: When each control joint axis completes rotation control of the corresponding joint axis of the target control object based on the angle control instruction at the current time, binocular endoscope images of the target control object at the current pose and encoder values corresponding to each control joint axis are obtained; Based on the binocular endoscope images and the initial pose three-dimensional model of the target control object, first actual observation angles corresponding to each joint axis are determined; Based on each encoder value and a preset angle mapping relationship, second actual observation angles corresponding to each joint axis are determined; wherein the preset angle mapping relationship is an angle mapping relationship between the joint axes of the target control object and the control joint axes of the mechanical arm; For each joint axis, the first actual observation angle and the second actual observation angle are fused to obtain an actual observation angle corresponding to the joint axis; Based on the actual observation angle, the angle control instruction of the control joint axis at the next time is adjusted, so that the control joint axis controls the corresponding joint axis to rotate based on the adjusted angle control instruction; The determination of the first actual observation angle corresponding to each joint axis based on the binocular endoscope images and the initial pose three-dimensional model of the target control object includes: Based on the binocular endoscope images, homogeneous transformation matrices corresponding to at least one preset key part of the target control object at the current pose are determined; wherein the preset key part includes each joint axis and an edge vertex of the target control object; Based on the relative positional relationship between each preset key part in the initial pose three-dimensional model of the target control object and each homogeneous transformation matrix, a target three-dimensional model of the target control object at the current pose is determined. determine a first actual observation angle corresponding to each joint axis based on the three-dimensional position coordinates corresponding to each preset key part in the target three-dimensional model; The binocular endoscope image includes a left eye endoscope image and a right eye endoscope image, and the determination of the homogeneous transformation matrix corresponding to at least one preset key part of the target control object in the current pose based on the binocular endoscope image includes: inputting the left eye endoscope image and the right eye endoscope image into a pre-trained disparity estimation model to obtain a disparity map of the target control object in the current pose; inputting the disparity map into a pre-trained key point recognition model to obtain key point position information corresponding to at least one preset key part of the target control object in the current pose; determine the homogeneous transformation matrix corresponding to at least one preset key part of the target control object in the current pose based on the disparity map and the key point position information corresponding to each preset key part; The key point position information is the two-dimensional coordinates of the key points in the target fusion image, and the target fusion image is the fusion image of the left eye endoscope image and the right eye endoscope image. The determination of the homogeneous transformation matrix corresponding to at least one preset key part of the target control object in the current pose based on the disparity map and the key point position information corresponding to each preset key part includes: determine the depth map corresponding to the disparity map based on the disparity map and the binocular endoscope device parameters; determine the depth information corresponding to each key point based on the depth map and the two-dimensional coordinates of each key point; For each key point, determine the three-dimensional space coordinates corresponding to each key point based on the two-dimensional coordinates and the corresponding depth information of the key point; determine the homogeneous transformation matrix corresponding to each preset key part based on the three-dimensional space coordinates of at least four key points belonging to the same preset key part; The binocular endoscope device parameters include the camera origin distance, the camera focal length, the first field corresponding to the left eye camera, and the second field corresponding to the right eye camera. The determination of the depth map corresponding to the disparity map based on the disparity map and the binocular endoscope device parameters includes: For each pixel point in the disparity map, determine the first imaging point of the current pixel point in the first field and the second imaging point of the current pixel point in the second field; determine the disparity distance based on the first distance between the first imaging point and the left boundary of the first field, and the second distance between the second imaging point and the left boundary of the second field; determine the depth value corresponding to the current pixel point based on the disparity distance of the current pixel point, the camera origin distance, and the camera focal length; determine the depth map corresponding to the disparity map based on the depth values of each pixel point; The target control object pose control method further includes determining an initial pose three-dimensional model of the target control object, including: In the case that the target control object is in an initial pose, a scanner is used to scan the target control object to obtain a three-dimensional structure model of the target control object; wherein the three-dimensional structure model comprises a plurality of geometric bodies; When the target control object is located in the field of view of the binocular endoscope, based on a homogeneous transformation matrix of the target control object relative to the binocular endoscope coordinate system, the three-dimensional position coordinates corresponding to each geometric body in the three-dimensional model structure are determined; Based on the three-dimensional structure model and the three-dimensional position coordinates corresponding to each geometric body, an initial pose three-dimensional model is constructed.

Images