Target control object attitude control method, apparatus and device, and medium

Abstract

The embodiment of the invention provides a target control object attitude control method, device and equipment and a medium, a target control object comprises at least one joint shaft, and each joint shaft is mounted on a control joint shaft on a mechanical arm. The method comprises the following steps: when each control joint shaft completes rotation control on a joint shaft corresponding to a target control object based on an angle control instruction at the current moment, acquiring a binocular endoscope image of the target control object under the current pose and an encoder value corresponding to each control joint shaft; determining an actual observation angle corresponding to each joint shaft based on the binocular endoscope image and each encoder value; and the angle control instruction of the control joint shaft at the next moment is adjusted based on the actual observation angle, so that the control joint shaft controls the corresponding joint shaft to rotate based on the adjusted angle control instruction, and the positioning precision of the target control object is improved.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a method, device, equipment and medium for controlling the posture of a target control object. Background Art

[0002] With the improvement of science and technology, surgical robotic arms have gradually developed in the field of surgery. The surgical scene requires extremely high absolute positioning accuracy of surgical instruments controlled by surgical robotic arms. Especially in complex surgical procedures, a slight deviation at the end of the surgical instrument may have a significant impact on the outcome of the operation. Therefore, the end posture control accuracy of the surgical instrument is crucial to the success rate of the operation and the safety of the patient.

[0003] At present, in surgical robot systems, the posture control of the end of the surgical instrument mainly depends on the feedback parameters of the robot arm sensor. However, due to the influence of various factors such as the rigidity of the robot arm, the gap between the joints, and the life attenuation of the surgical instrument wire, there are often errors in the actual control process, resulting in large deviations in the posture of the end of the surgical instrument, making it difficult to meet the high-precision positioning requirements of the surgical instrument. Summary of the invention

[0004] The embodiments of the present invention provide a method, device, equipment and medium for controlling the posture of a target control object, so as to eliminate error accumulation and improve the positioning accuracy of the target control object by a posture control compensation mechanism based on the posture error fed back by a binocular endoscope image and the posture error fed back by a joint encoder value.

[0005] In a first aspect, an embodiment of the present invention provides a method for controlling a posture of a target control object, wherein the target control object includes at least one joint axis, each of the joint axes being mounted on a control joint axis on a robotic arm, and the method includes:

[0006] When each control joint axis completes rotation control of the joint axis corresponding to the target control object based on the angle control instruction at the current moment, a binocular endoscope image of the target control object in the current posture and encoder values ​​corresponding to each control joint axis are obtained;

[0007] Determining an actual observation angle corresponding to each of the joint axes based on the binocular endoscopic image and each of the encoder values;

[0008] The angle control instruction for controlling the joint axis at the next moment is adjusted based on the actual observation angle, so that the controlled joint axis controls the corresponding joint axis to rotate based on the adjusted angle control instruction.

[0009] In a second aspect, an embodiment of the present invention further provides a target control object posture control device, wherein the target control object includes at least one joint axis, each of the joint axes is mounted on a control joint axis on a robotic arm, and the device includes:

[0010] A data acquisition module, used to acquire a binocular endoscope image of the target control object in a current position and an encoder value corresponding to each of the control joint axes when each control joint axis completes rotation control of the corresponding joint axis of the target control object based on the angle control instruction at the current moment;

[0011] An observation angle determination module, used to determine an actual observation angle corresponding to each of the joint axes based on the binocular endoscopic image and each of the encoder values;

[0012] The instruction adjustment module is used to adjust the angle control instruction of the control joint axis at the next moment based on the actual observation angle, so that the control joint axis controls the corresponding joint axis to rotate based on the adjusted angle control instruction.

[0013] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0014] one or more processors;

[0015] a storage device for storing one or more programs,

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the target control object posture control method as described in any one of the embodiments of the present invention.

[0017] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer executable instructions, wherein the computer executable instructions are used to execute the target control object posture control method as described in any of the embodiments of the present invention when executed by a computer processor.

[0018] The technical solution of the embodiment of the present invention, the target control object includes at least one joint axis, each joint axis is mounted on a control joint axis on the robot arm, and the method includes: when each control joint axis completes the rotation control of the corresponding joint axis of the target control object based on the angle control instruction at the current moment, the binocular endoscope image of the target control object in the current posture and the encoder value corresponding to each control joint axis are obtained, and then, based on the binocular endoscope image and each encoder value, the actual observation angle corresponding to each joint axis is determined, so that the angle control instruction of the control joint axis at the next moment is adjusted based on the actual observation angle, so that the control joint axis controls the corresponding joint axis to rotate based on the adjusted angle control instruction. The technical solution provided by this embodiment can detect the posture error of the target control object caused by the joint gap and the life attenuation of the instrument wire in real time through the binocular endoscope image of the target control object in the current posture, and further, the posture control compensation mechanism based on the posture error feedback from the binocular endoscope image and the posture error feedback from the joint encoder value can eliminate the error accumulation, thereby improving the positioning accuracy of the target control object. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required to describe the embodiments. Obviously, the drawings introduced are only drawings of a part of the embodiments to be described in the present invention, rather than all the drawings. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A schematic diagram of a flow chart of a method for controlling a posture of a target control object provided by an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of a specific implementation process of a target control object posture control method involved in an embodiment of the present invention;

[0022] Figure 3 A schematic flow chart of another method for controlling the posture of a target control object provided by an embodiment of the present invention;

[0023] Figure 4 Schematic diagram of a process for determining a disparity map and key point position information according to an embodiment of the present invention;

[0024] Figure 5 A schematic diagram of the parallax depth relationship of a binocular endoscope according to an embodiment of the present invention;

[0025] Figure 6 A schematic diagram of the structure of a target control object posture control device provided by an embodiment of the present invention;

[0026] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0028] Embodiment 1

[0029] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0030] Before introducing the embodiments of the present invention, the application scenarios of the embodiments of the present invention can be described first. For different surgical procedures, the operating postures of the surgical robot arm vary, and the surgical scenarios have extremely high requirements for the absolute positioning accuracy of the surgical robot. In this case, the posture of the target control object driven by the surgical robot arm can be controlled based on the technical method provided by the embodiments of the present invention to improve the positioning accuracy of the target control object.

[0031] Figure 1 A flow chart of a method for controlling the posture of a target control object provided in an embodiment of the present invention is provided. This embodiment can be applied to situations where the posture of a target control object is to be precisely controlled. The method can be executed by a target control object posture control device, which can be implemented in the form of software and / or hardware. The hardware can be an electronic device such as a server, which can execute the method for controlling the posture of a target control object provided by this technical solution.

[0032] In this embodiment, the target control object includes at least one joint axis, and each joint axis is mounted on a control joint axis on the robot arm.

[0033] Among them, the robotic arm is a device that can imitate the movements of a human arm, and the robotic arm includes multiple control joint axes. The target control object is the instrument held at the end of the robotic arm. For example, if surgery is required based on a robotic arm, the target control object may be a surgical instrument held at the end of the surgical robotic arm. The target control object includes multiple joint axes. The joint axes of the target control object may include at least one of a left and right yaw axis, a pitch axis, and a rotation axis. In a specific application process, the joint axis on the target control object can be mounted on the control joint axis of the robotic arm, and the joint axis can rotate driven by the control joint axis, so that the target control object can complete specific tasks, such as object grabbing tasks, suturing tasks, cutting tasks, etc.

[0034] like Figure 1 As shown, the target control object posture control method includes:

[0035] S110. When each control joint axis completes rotation control of the corresponding joint axis of the target control object based on the angle control instruction at the current moment, a binocular endoscope image of the target control object in the current posture and encoder values ​​corresponding to each control joint axis are obtained.

[0036] Among them, the angle control instruction is an instruction for accurately controlling the control joint axis on the robot arm to rotate to a specified angle. This control usually involves the rotation control of the motor, and the precise control of the motor rotation angle is achieved through the inner current loop, the outer speed loop and the outermost position loop. In this embodiment, when an operation control instruction that requires the target control object to complete a certain operation is received, the angle control instruction corresponding to each control joint axis can be decomposed according to the operation control instruction, so that the control joint axis can perform the corresponding operation based on the angle control instruction. Based on this, the joint axis of the target control object mounted on the control joint axis can complete the corresponding operation under the drive of the control joint axis.

[0037] Among them, the current posture refers to the position and posture of the target control object after the corresponding operation is completed based on the angle control instruction at the current moment. Binocular endoscopic image refers to the image obtained by a binocular endoscopic imaging system. This imaging system includes a light source module, two imaging modules and two camera modules. The light source module is used to illuminate the object to be measured (i.e., the target control object), while the two camera modules simulate the two eyes of a person to obtain different images and use parallax to calculate the depth. This system can provide two different perspectives, and obtain the depth information of the object by calculating the difference between the two perspectives, thereby achieving high-precision measurement.

[0038] In this embodiment, a joint encoder is configured in each control joint axis of the robot arm. The encoder value is the data extracted from the joint encoder. The joint encoder is a sensor used to measure the rotation angle of the robot joint, which can accurately measure the position and movement of the joint. The joint encoder can measure the joint position by generating pulse signals when the joint rotates, and these pulse signals can then be used to calculate the position and movement of the end effector of the robot arm (i.e., the target control object).

[0039] In this embodiment, the end of the robot arm holds the target control object, and the target control object can reach the specified position under the drive of the robot arm. This process is the process of the robot arm performing rotation control on the target control object. During the operation of the robot arm, the control instructions received are the angle control instructions corresponding to each control joint axis, so that each control joint axis performs a rotation action according to the corresponding angle control instruction, wherein the angle control instruction can be sent from the main controller to the robot arm.

[0040] Specifically, during the process of the robot arm performing rotation control on the target control object, the robot arm can receive the angle control instruction corresponding to the current moment at different control moments. When each control joint axis of the robot arm completes the rotation control of the corresponding joint axis of the target control object based on the angle control instruction at the current moment, the position and posture of the target control object has changed relative to the previous control cycle. At this time, the binocular endoscope image of the target control object in the current position and posture can be collected through the binocular endoscope, and the corresponding encoder value can be obtained from the joint encoder of each control joint axis.

[0041] Exemplarily, the target control object is a surgical instrument, which includes two joint axes, namely joint axis 1 and joint axis 2. Joint axis 1 is hung on the control joint axis A of the robot arm, and joint axis 2 is hung on the control joint axis B of the robot arm. At the current time T0, the angle control instruction corresponding to the control joint axis A is θ01, and the angle control instruction corresponding to the control joint axis A is θ02. When the control joint axis A completes the rotation control of joint axis 1 based on θ01, and the control joint axis B completes the rotation control of joint axis 2 based on θ02, a binocular endoscope is used to collect a binocular endoscope image of the surgical instrument in the current posture, and the encoder value δ1 corresponding to the control joint axis A and the encoder value δ2 corresponding to the control joint axis B are obtained.

[0042] S120. Determine the actual observation angle corresponding to each joint axis based on the binocular endoscope image and each encoder value.

[0043] The actual observation angle is the actual angle value corresponding to the joint axis of the target control object measured by using a specific method.

[0044] Specifically, the binocular endoscope image can be used to determine the real posture of the target control object after the current control cycle, so that the first actual observation angle corresponding to each joint axis of the target control object can be calculated through the real posture. Through the angle mapping relationship between each control joint axis of the robot arm and the corresponding joint axis of the target control object, the second actual observation angle corresponding to each joint axis of the target control object can be calculated on the basis of the encoder value of the control joint axis. Furthermore, the first actual observation angle and the second actual observation angle can be fused to obtain the actual observation angle corresponding to each joint axis.

[0045] Based on the above examples, the first actual observation angle corresponding to the joint axis 1 in the target control object is calculated as α1, and the first actual observation angle corresponding to the joint axis 2 is α2 through the binocular endoscope image; the second actual observation angle corresponding to the joint axis 1 in the target control object is calculated as β1, and the second actual observation angle corresponding to the joint axis 2 is β2 through the encoder values ​​of the control joint axis A and the control joint axis B. Thus, the actual observation angle γ1 corresponding to the joint axis 1 in the target control object can be determined according to α1 and β1; the actual observation angle γ2 corresponding to the joint axis 2 in the target control object can be determined according to α2 and β2.

[0046] S130, adjusting the angle control instruction of the corresponding control joint axis at the next moment based on the actual observation angle, so that the control joint axis controls the corresponding joint axis to rotate based on the adjusted angle control instruction.

[0047] On the basis of the above-mentioned embodiment, after each controlled joint axis completes the rotation control of the corresponding joint axis of the target controlled object based on the angle control instruction at the current moment, each controlled joint axis will control the joint axis of the target controlled object according to the angle control instruction at the next moment. Before controlling the joint axis of the target controlled object according to the angle control instruction at the next moment, the angle control instruction at the next moment can be adjusted based on the actual observation angle, so that the controlled joint axis can control the corresponding joint axis for rotation control according to the adjusted angle control instruction, based on which, the purpose of matching the actual position information of the target controlled object with the expected position information can be achieved.

[0048] On the basis of the above exemplary embodiment, the angle control instruction for controlling joint axis A at the next moment is θ11. The angle control instruction θ11 can be adjusted by the actual observation angle γ1 corresponding to joint axis 1 using proportional-integral-derivative (PID) control, and the adjusted angle control instruction is θ11; the angle control instruction for controlling joint axis B at the next moment is θ12. The angle control instruction θ12 can be adjusted by the actual observation angle γ2 corresponding to joint axis 2 using proportional-integral-derivative (PID) control, and the adjusted angle control instruction is θ12; and then, joint axis A is controlled to rotate joint axis 1 based on θ'11, and joint axis B is controlled to rotate joint axis 2 based on θ12.

[0049] In this embodiment, before the control joint axis on the robot arm drives the joint axis of the target control object to rotate based on the angle control instruction at the next moment, the angle control instruction at the next moment is adjusted according to the actual observation angle of the joint axis, and the adjusted angle control instruction is sent to the robot arm, so that the robot arm drives the target control object to rotate based on the angle control instruction. This is because if the robot arm performs rotation control based on the unadjusted angle control instruction, the target control object should reach the target position A in the end. However, due to the influence of various factors such as the rigidity of the robot arm, the joint clearance, and the attenuation of the life of the surgical instrument wire, the target control object can only reach position A1 in the end if the rotation control is actually performed based on the unadjusted angle control instruction. Therefore, in order to make the final position A1 of the target control object as close to the target position A as possible, the angle control instruction can be error compensated first.

[0050] Next, a specific example is used to illustrate the target control object posture control method provided by this embodiment. For a schematic diagram of the specific implementation process of the target control object posture control method, see Figure 2 .like Figure 2As shown, in the first step, each control joint axis of the robot arm performs rotation control on each joint axis of the target control object based on the angle control instruction at the current moment; in the second step, when each joint axis of the target control object completes the rotation control, a binocular endoscopic image of the target control object in the current posture is obtained, and at the same time, the encoder value corresponding to each control joint axis is obtained; in the third step, the binocular endoscopic image is processed by a posture recognition algorithm to obtain a first actual observation angle corresponding to each joint axis, and at the same time, based on the mapping relationship between each encoder value and the preset angle, the second actual observation angle corresponding to each joint axis is obtained; in the fourth step, based on the first actual observation angle and the second actual observation angle, the actual observation angle corresponding to each joint axis in the target control object is determined; in the fifth step, the angle control instruction for controlling the joint axis at the next moment is adjusted according to the actual observation angle, so that the control joint axis controls the corresponding joint axis to rotate based on the adjusted angle control instruction.

[0051] The technical solution of the embodiment of the present invention, the target control object includes at least one joint axis, each joint axis is mounted on a control joint axis on the robot arm, and the method includes: when each control joint axis completes the rotation control of the corresponding joint axis of the target control object based on the angle control instruction at the current moment, the binocular endoscope image of the target control object in the current posture and the encoder value corresponding to each control joint axis are obtained, and then, based on the binocular endoscope image and each encoder value, the actual observation angle corresponding to each joint axis is determined, so that the angle control instruction of the control joint axis at the next moment is adjusted based on the actual observation angle, so that the control joint axis controls the corresponding joint axis to rotate based on the adjusted angle control instruction. The technical solution provided by this embodiment can detect the posture error of the target control object caused by the joint gap and the life attenuation of the instrument wire in real time through the binocular endoscope image of the target control object in the current posture, and further, the posture control compensation mechanism based on the posture error feedback from the binocular endoscope image and the posture error feedback from the joint encoder value can eliminate the error accumulation, thereby improving the positioning accuracy of the target control object.

[0052] Embodiment 2

[0053] Figure 3 The flowchart of a method for controlling the posture of a target control object provided by an embodiment of the present invention is a detailed description of the specific implementation method 120 based on the above embodiment, and its specific implementation method can refer to the technical solution of this embodiment. Among them, the technical terms that are the same or corresponding to the above embodiment are not repeated here.

[0054] like Figure 3 As shown, the method specifically comprises the following steps:

[0055] S210. When each control joint axis completes rotation control of the corresponding joint axis of the target control object based on the angle control instruction at the current moment, a binocular endoscope image of the target control object in the current posture and encoder values ​​corresponding to each control joint axis are obtained.

[0056] S220. Determine a first actual observation angle corresponding to each joint axis based on the binocular endoscope image and the initial posture three-dimensional model of the target controlled object.

[0057] The initial posture three-dimensional model refers to the three-dimensional data model corresponding to the target control object in the initial posture. When the target control object is in the initial posture, the angle value of each joint axis is the initial value.

[0058] In this embodiment, a specific implementation method for determining the initial posture three-dimensional model of the target control object may include: when the target control object is in the initial posture, scanning the target control object with a scanner to obtain a three-dimensional structural model of the target control object; when the target control object is located within the field of view of the binocular endoscope, based on the homogeneous transformation matrix of the target control object relative to the binocular endoscope coordinate system, determining the three-dimensional position coordinates corresponding to each geometric body in the three-dimensional model structure; based on the three-dimensional structural model and the three-dimensional position coordinates corresponding to each geometric body, constructing an initial posture three-dimensional model.

[0059] Among them, the three-dimensional structural model refers to a series of geometric bodies (such as structural points, triangles, lines, arcs, cubes, etc.) connected in three-dimensional space to represent a physical entity (i.e., the target control object).

[0060] In this embodiment, when the target control object is in an initial posture, a scanner, for example, a 3D scanner, can be used to scan the target control object to obtain a three-dimensional structural model corresponding to the target control object; further, the target control object is mounted on a robotic arm, and the target control object in the initial posture is moved into the field of view of the binocular endoscope, and then, the posture data of the target control object is generated using the Monte Carlo method; the homogeneous transformation matrix of the target control object relative to the binocular endoscope coordinate system is calculated through the kinematic equation of the robotic arm; thus, the three-dimensional position coordinates corresponding to each geometric body in the three-dimensional structural model (i.e., the spatial coordinates in the coordinate system of the binocular endoscope) are derived according to the homogeneous transformation matrix of the target control object; based on this, for the three-dimensional structural model, the three-dimensional position coordinates corresponding to each geometric body are assigned to the corresponding geometric body, and the initial posture three-dimensional model is constructed.

[0061] Specifically, the current posture of the target controlled object can be determined based on the binocular endoscopic image, and the target three-dimensional model of the target controlled object in the current posture can be derived based on the relative position relationship between key components in the initial posture three-dimensional model and the current posture of the target controlled object. Then, based on the spatial coordinates of multiple key points in the target three-dimensional model, the first actual observation angle corresponding to each joint axis in the target controlled object can be calculated.

[0062] Optionally, based on the binocular endoscope image and the initial posture three-dimensional model of the target controlled object, a specific implementation method for determining the first actual observation angle corresponding to each joint axis may include:

[0063] S2201. Based on the binocular endoscopic image, determine the homogeneous transformation matrix corresponding to at least one preset key part of the target control object in the current posture.

[0064] The preset key parts include the joint axes and edge vertices of the target control object. The binocular endoscope image includes a left endoscope image and a right endoscope image.

[0065] Specifically, based on the binocular endoscopic image, determining the homogeneous transformation matrix corresponding to at least one preset key part of the target control object in the current posture includes:

[0066] (1) The left-eye endoscopic image and the right-eye endoscopic image are input into a pre-trained disparity estimation model to obtain a disparity map of the target controlled object in its current position.

[0067] (2) The disparity map is input into a pre-trained key point recognition model to obtain key point position information corresponding to at least one preset key part of the target control object in the current posture.

[0068] In this embodiment, the schematic diagram of the process of determining the disparity map and key point position information is shown in Figure 4 ,like Figure 4As shown, the disparity estimation model includes a feature extraction layer and a disparity estimation layer. First, the left-eye endoscopic image and the right-eye endoscopic image are input into the feature extraction layer, and features are extracted using a 2D convolution kernel. Then, the images enter the disparity estimation layer, and a 3D convolution layer is used to perform cost matching calculations to restore the disparity map from the features. Optionally, this disparity map is a rough image, and this rough disparity map can be input into a refinement network to obtain a refined disparity map, which is then used to update the disparity map. The refinement network uses hierarchical and edge-sensitive weights to optimize and adjust the input rough disparity map. For example, bilinear upsampling and convolution can be used to construct the refinement network. At the same time, based on the disparity map, the disparity map can be input into a pre-trained key point recognition model to obtain multiple key point position information corresponding to each preset key part of the target control object in the current posture (i.e., the x-axis and y-axis coordinates of the key points). Among them, the key point recognition model can use a posture recognition network model, such as deepPose or openPose, to mark the x-axis and y-axis coordinates of the key points in the image.

[0069] (3) Based on the disparity map and the key point position information corresponding to each preset key part, determine the homogeneous transformation matrix corresponding to at least one preset key part of the target control object in the current posture.

[0070] The key point position information is the two-dimensional coordinates of the key point in the target fusion image, and the target fusion image is a fusion image of the left endoscope image and the right endoscope image.

[0071] More specifically, based on the disparity map and the key point position information corresponding to each preset key part, determining the homogeneous transformation matrix corresponding to at least one preset key part of the target control object in the current posture may include: determining the depth map corresponding to the disparity map based on the disparity map and the binocular endoscope equipment parameters; determining the depth information corresponding to each key point based on the depth map and the two-dimensional coordinates of each key point; for each key point, determining the three-dimensional space coordinates corresponding to each key point based on the two-dimensional coordinates corresponding to the key point and the corresponding depth information; determining the homogeneous transformation matrix corresponding to each preset key part based on the three-dimensional space coordinates of at least four key points belonging to the same preset key part.

[0072] Among them, the binocular endoscope equipment parameters include the camera origin distance between the left camera and the right camera, the camera focal length, the first field corresponding to the left camera, and the second field corresponding to the right camera.

[0073] Based on this, based on the disparity map and the binocular endoscope equipment parameters, a depth map corresponding to the disparity map is determined, including: for each pixel in the disparity map, a first imaging point of the current pixel in the first field and a second imaging point of the current pixel in the second field are determined; based on a first distance between the first imaging point and the left boundary of the first field, and a second distance between the second imaging point and the left boundary of the second field, a disparity distance is determined; based on the disparity distance of the current pixel, the camera origin distance and the camera focal length, a depth value corresponding to the current pixel is determined; based on the depth values ​​of each pixel, a depth map corresponding to the disparity map is determined.

[0074] In this embodiment, after obtaining the disparity map, it is necessary to convert it into a depth map to calculate the depth information of the key points. The specific determination method is: See the schematic diagram of the disparity depth relationship of the binocular endoscope. Figure 5 ,like Figure 5 As shown, P is a point in the field of view (equivalent to a pixel in the disparity map), D is the depth of point P, and O l With O r are the coordinate origins of the left and right cameras of the binocular endoscope, respectively. The distance between the origins of the left and right cameras is represented by B, f is the focal length of the camera, and P l The first imaging point of the current pixel point P in the first field, P r is the second imaging point of the current pixel point P in the second field, x l is the first distance between the first imaging point and the left boundary of the first field, x r is the second distance between the second imaging point and the left boundary of the second field. The parallax distance can be expressed as S=x l -x r Among them, the pixel value of each pixel in the disparity map is S, and the depth value of point P is calculated according to the principle of similar triangles as shown in Formula 1:

[0075]

[0076] Based on the same method, the depth value corresponding to each pixel in the disparity map can be determined, so that the depth map corresponding to the disparity map can be determined according to the depth value of each pixel.

[0077] Further, based on the depth map, for a certain key point, the two-dimensional coordinates of the key point can be used to determine which pixel in the depth map the key point corresponds to, so that the depth value of the pixel can be determined as the depth information of the key point. In the same way, the depth information of each key point can be determined. Thus, for each key point, the three-dimensional space coordinates (x, y, z) corresponding to the key point can be determined based on the two-dimensional coordinates (i.e., the x-coordinate value and the y-coordinate value) and the corresponding depth information (equivalent to the z-coordinate value) corresponding to the key point. In particular, the three-dimensional space coordinates are coordinate data relative to the coordinate system under the binocular endoscope.

[0078] Furthermore, since it is easy to determine which key points the preset key parts correspond to, the homogeneous transformation matrix of the coordinate system corresponding to each preset key part can be determined based on the three-dimensional spatial coordinates of at least four key points belonging to the same preset key part.

[0079] S2202: Determine a target three-dimensional model of the target controlled object in the current posture based on the relative position relationship between the preset key parts in the initial posture three-dimensional model of the target controlled object and the homogeneous transformation matrices.

[0080] In this embodiment, the relative position relationship between the preset key parts can be determined according to the three-dimensional position coordinates of the geometric bodies corresponding to each preset key part in the initial pose three-dimensional model. For a specific target control object, the relative position relationship between a part of the preset key parts is determined. Based on this, the target three-dimensional model of the target control object in the current pose can be determined according to the homogeneous transformation matrix corresponding to each preset key part and the relative position relationship between each preset key part in the initial pose three-dimensional model. That is, it can be understood that the initial pose three-dimensional model of the target control object is transformed to the binocular endoscope coordinate system according to the current pose to obtain the target three-dimensional model.

[0081] S2203. Determine a first actual observation angle corresponding to each joint axis based on the three-dimensional position coordinates corresponding to each preset key part in the target three-dimensional model.

[0082] In this embodiment, the phase position relationship between each preset key part can be calculated based on the three-dimensional position coordinates of the key points in each preset key part in the target three-dimensional model, and then the first actual observation angle corresponding to each joint axis can be calculated.

[0083] S230. Determine a second actual observation angle corresponding to each joint axis based on a mapping relationship between each encoder value and a preset angle.

[0084] The preset angle mapping relationship is the angle mapping relationship between the joint axis of the target control object and the control joint axis of the robot arm.

[0085] In this embodiment, the encoder value can represent the actual observation value of the control joint axis of the robot arm, so that the angle calculation value corresponding to the encoder value can be calculated according to the preset angle mapping relationship. For a certain joint axis, this angle calculation value is the second actual observation angle corresponding thereto. Based on the same method, the second actual observation angle corresponding to each joint axis of the target control object can be determined.

[0086] S240 . For each joint axis, perform fusion processing on the first actual observation angle and the second actual observation angle to obtain the actual observation angle corresponding to the joint axis.

[0087] In this embodiment, for a certain joint axis, the first actual observation angle and the second actual observation angle can be fused by extended Kalman filtering to obtain the actual observation angle corresponding to the joint axis. Based on the same determination method, the actual observation angle corresponding to each joint axis can be obtained.

[0088] S250, adjusting the angle control instruction of the corresponding control joint axis at the next moment based on the actual observation angle, so that the control joint axis controls the corresponding joint axis to rotate based on the adjusted angle control instruction.

[0089] The technical solution of this embodiment, when determining the actual observation angle corresponding to each joint axis based on the binocular endoscopic image and each encoder value, determines the first actual observation angle corresponding to each joint axis based on the binocular endoscopic image and the initial posture three-dimensional model of the target control object, and determines the second actual observation angle corresponding to each joint axis based on each encoder value and the preset angle mapping relationship, wherein the preset angle mapping relationship is the angle mapping relationship between the joint axis and the control joint axis, thereby fusing the first actual observation angle and the second actual observation angle for each joint axis to obtain the actual observation angle corresponding to the joint axis. The technical solution provided by this embodiment can eliminate error accumulation based on the posture control compensation mechanism of the posture error feedback from the binocular endoscopic image and the posture error feedback from the joint encoder value, thereby improving the positioning accuracy of the target control object.

[0090] Embodiment 3

[0091] Figure 6 A structural schematic diagram of a target control object posture control device provided in an embodiment of the present invention, wherein the target control object includes at least one joint axis, each of which is mounted on a control joint axis on a robotic arm, and the target control object posture control device includes: a data acquisition module 310, an observation angle determination module 320 and an instruction adjustment module 330.

[0092] The data acquisition module 310 is used to acquire the binocular endoscope image of the target control object in the current posture and the encoder value corresponding to each of the control joint axes when each control joint axis completes the rotation control of the corresponding joint axis of the target control object based on the angle control instruction at the current moment;

[0093] An observation angle determination module 320, for determining an actual observation angle corresponding to each of the joint axes based on the binocular endoscopic image and each of the encoder values;

[0094] The instruction adjustment module 330 is used to adjust the angle control instruction of the control joint axis at the next moment based on the actual observation angle, so that the control joint axis controls the corresponding joint axis to rotate based on the adjusted angle control instruction.

[0095] Based on the above embodiment, the observation angle determination module 320 includes:

[0096] A first angle determination submodule, used for determining a first actual observation angle corresponding to each of the joint axes based on the binocular endoscopic image and the initial posture three-dimensional model of the target controlled object;

[0097] A second angle determination submodule is used to determine a second actual observation angle corresponding to each of the joint axes based on each of the encoder values ​​and a preset angle mapping relationship; wherein the preset angle mapping relationship is an angle mapping relationship between the joint axis of the target control object and the control joint axis of the robotic arm;

[0098] The observation angle determination submodule is used to fuse the first actual observation angle and the second actual observation angle for each joint axis to obtain the actual observation angle corresponding to the joint axis.

[0099] Based on the above embodiment, the first angle determination submodule includes:

[0100] A transformation matrix determination unit, used to determine, based on the binocular endoscope image, a homogeneous transformation matrix corresponding to at least one preset key part of the target control object in the current posture; wherein the preset key part includes each joint axis and edge vertex of the target control object;

[0101] A three-dimensional model conversion unit, used to determine a target three-dimensional model of the target control object in a current posture based on the relative position relationship between the preset key parts in the initial posture three-dimensional model of the target control object and the homogeneous transformation matrices;

[0102] The first angle determination unit is used to determine the first actual observation angle corresponding to each of the joint axes based on the three-dimensional position coordinates corresponding to each of the preset key parts in the target three-dimensional model.

[0103] On the basis of the above embodiment, the binocular endoscopic image includes a left-eye endoscopic image and a right-eye endoscopic image, and the transformation matrix determination unit based on the binocular endoscopic image includes:

[0104] A disparity map determination subunit, used for inputting the left-eye endoscopic image and the right-eye endoscopic image into a pre-trained disparity estimation model to obtain a disparity map of the target control object in a current posture;

[0105] A key point position determination subunit is used to input the disparity map into a pre-trained key point recognition model to obtain key point position information corresponding to at least one preset key part of the target control object in the current posture;

[0106] The transformation matrix determination subunit is used to determine the homogeneous transformation matrix corresponding to at least one preset key part of the target control object in the current posture based on the disparity map and the key point position information corresponding to each preset key part.

[0107] On the basis of the above embodiment, the key point position information is the two-dimensional coordinate of the key point in the target fusion image, and the target fusion image is a fusion image of the left-eye endoscopic image and the right-eye endoscopic image. The transformation matrix determination subunit is specifically used to determine the depth map corresponding to the disparity map based on the disparity map and the binocular endoscope device parameters; determine the depth information corresponding to each key point based on the depth map and the two-dimensional coordinates of each key point; for each key point, determine the three-dimensional space coordinates corresponding to each key point based on the two-dimensional coordinates corresponding to the key point and the corresponding depth information; determine the homogeneous transformation matrix corresponding to each preset key part based on the three-dimensional space coordinates of at least four key points belonging to the same preset key part.

[0108] On the basis of the above embodiment, the binocular endoscope device parameters include the camera origin distance between the left camera and the right camera, the camera focal length, the first field corresponding to the left camera, the second field corresponding to the right camera, and the transformation matrix determination subunit, which is specifically used to determine, for each pixel in the disparity map, the first imaging point of the current pixel in the first field and the second imaging point of the current pixel in the second field; determine the disparity distance based on the first distance between the first imaging point and the left boundary of the first field, and the second distance between the second imaging point and the left boundary of the second field; determine the depth value corresponding to the current pixel based on the disparity distance of the current pixel, the camera origin distance and the camera focal length; determine the depth map corresponding to the disparity map based on the depth values ​​of each of the pixels.

[0109] Based on the above embodiment, the observation angle determination module 320 further includes: a three-dimensional model determination submodule; the three-dimensional model determination submodule includes:

[0110] An initial model determination unit is used to scan the target control object with a scanner when the target control object is in an initial posture to obtain a three-dimensional structure model of the target control object; wherein the three-dimensional structure model includes a plurality of geometric bodies;

[0111] A position coordinate determination unit, used to determine the three-dimensional position coordinates corresponding to each of the geometric bodies in the three-dimensional model structure based on the homogeneous transformation matrix of the target control object relative to the binocular endoscope coordinate system when the target control object is located within the field of view of the binocular endoscope;

[0112] The three-dimensional model determination unit is used to construct an initial posture three-dimensional model based on the three-dimensional structure model and the three-dimensional position coordinates corresponding to each of the geometric bodies.

[0113] The technical solution of the embodiment of the present invention, the target control object includes at least one joint axis, each joint axis is mounted on a control joint axis on the robot arm, and the method includes: when each control joint axis completes the rotation control of the corresponding joint axis of the target control object based on the angle control instruction at the current moment, the binocular endoscope image of the target control object in the current posture and the encoder value corresponding to each control joint axis are obtained, and then, based on the binocular endoscope image and each encoder value, the actual observation angle corresponding to each joint axis is determined, so that the angle control instruction of the control joint axis at the next moment is adjusted based on the actual observation angle, so that the control joint axis controls the corresponding joint axis to rotate based on the adjusted angle control instruction. The technical solution provided by this embodiment can detect the posture error of the target control object caused by the joint gap and the life attenuation of the instrument wire in real time through the binocular endoscope image of the target control object in the current posture, and further, the posture control compensation mechanism based on the posture error feedback from the binocular endoscope image and the posture error feedback from the joint encoder value can eliminate the error accumulation, thereby improving the positioning accuracy of the target control object.

[0114] The target control object posture control device provided in the embodiment of the present invention can execute the target control object posture control method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0115] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present invention.

[0116] Embodiment 4

[0117] Figure 7 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0118] like Figure 7 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0119] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0120] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a target control object posture control method.

[0121] In some embodiments, the target control object posture control method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the target control object posture control method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to execute the target control object posture control method by any other appropriate means (e.g., by means of firmware).

[0122] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0123] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0124] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0125] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0126] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0127] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0128] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0129] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

[0130] Embodiment 5

[0131] An embodiment of the present invention further provides a storage medium containing computer executable instructions, wherein the computer executable instructions are used to execute a method for controlling a posture of a target control object when executed by a computer processor, wherein the target control object includes at least one joint axis, each of which is mounted on a control joint axis on a robotic arm, and the method includes:

[0132] When each control joint axis completes rotation control of the joint axis corresponding to the target control object based on the angle control instruction at the current moment, a binocular endoscope image of the target control object in the current posture and encoder values ​​corresponding to each control joint axis are obtained;

[0133] Determining an actual observation angle corresponding to each of the joint axes based on the binocular endoscopic image and each of the encoder values;

[0134] The angle control instruction for controlling the joint axis at the next moment is adjusted based on the actual observation angle, so that the controlled joint axis controls the corresponding joint axis to rotate based on the adjusted angle control instruction.

[0135] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0136] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0137] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0138] Computer program code for performing the operation of embodiments of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for controlling the posture of a target control object, wherein the target control object comprises at least one joint axis, each of which is mounted on a control joint axis on a robotic arm, characterized in that: include: When each control joint axis completes rotation control of the joint axis corresponding to the target control object based on the angle control instruction at the current moment, a binocular endoscope image of the target control object in the current posture and encoder values ​​corresponding to each control joint axis are obtained; Determining an actual observation angle corresponding to each of the joint axes based on the binocular endoscopic image and each of the encoder values; The angle control instruction for controlling the joint axis at the next moment is adjusted based on the actual observation angle, so that the controlled joint axis controls the corresponding joint axis to rotate based on the adjusted angle control instruction.

2. The method according to claim 1, characterized in that The determining of the actual observation angle corresponding to each of the joint axes based on the binocular endoscope image and each of the encoder values ​​comprises: Determine a first actual observation angle corresponding to each of the joint axes based on the binocular endoscope image and the initial posture three-dimensional model of the target controlled object; Based on each of the encoder values ​​and the preset angle mapping relationship, determining the second actual observation angle corresponding to each of the joint axes; wherein the preset angle mapping relationship is the angle mapping relationship between the joint axis of the target control object and the control joint axis of the robotic arm; For each of the joint axes, the first actual observation angle and the second actual observation angle are fused to obtain the actual observation angle corresponding to the joint axis.

3. The method according to claim 2, characterized in that The determining of the first actual observation angle corresponding to each joint axis based on the binocular endoscope image and the initial posture three-dimensional model of the target controlled object comprises: Based on the binocular endoscope image, determine the homogeneous transformation matrix corresponding to at least one preset key part of the target control object in the current posture; wherein the preset key part includes each joint axis and edge vertex of the target control object; Determining a target three-dimensional model of the target controlled object in a current posture based on the relative position relationship between the preset key parts in the initial posture three-dimensional model of the target controlled object and the homogeneous transformation matrices; Based on the three-dimensional position coordinates corresponding to each of the preset key parts in the target three-dimensional model, the first actual observation angle corresponding to each of the joint axes is determined.

4. The method according to claim 3, characterized in that The binocular endoscopic image includes a left-eye endoscopic image and a right-eye endoscopic image, and determining a homogeneous transformation matrix corresponding to at least one preset key part of the target control object in the current posture based on the binocular endoscopic image includes: Inputting the left-eye endoscopic image and the right-eye endoscopic image into a pre-trained disparity estimation model to obtain a disparity map of the target control object in the current position; Inputting the disparity map into a pre-trained key point recognition model to obtain key point position information corresponding to at least one preset key part of the target control object in the current posture; Based on the disparity map and the key point position information corresponding to each of the preset key parts, a homogeneous transformation matrix corresponding to at least one preset key part of the target control object in the current posture is determined.

5. The method according to claim 4, characterized in that The key point position information is the two-dimensional coordinates of the key point in the target fusion image, the target fusion image is the fusion image of the left endoscope image and the right endoscope image, and the method of determining the homogeneous transformation matrix corresponding to at least one preset key part of the target control object in the current posture based on the disparity map and the key point position information corresponding to each preset key part includes: Based on the disparity map and the binocular endoscope device parameters, determining a depth map corresponding to the disparity map; Determine depth information corresponding to each of the key points based on the depth map and the two-dimensional coordinates of each of the key points; For each of the key points, based on the two-dimensional coordinates corresponding to the key point and the corresponding depth information, determine the three-dimensional space coordinates corresponding to each of the key points; Based on the three-dimensional space coordinates of at least four key points belonging to the same preset key part, a homogeneous transformation matrix corresponding to each preset key part is determined.

6. The method according to claim 5, characterized in that The binocular endoscope device parameters include the camera origin distance between the left camera and the right camera, the camera focal length, the first field corresponding to the left camera, and the second field corresponding to the right camera. The determining of the depth map corresponding to the disparity map based on the disparity map and the binocular endoscope device parameters includes: For each pixel in the disparity map, determining a first imaging point of the current pixel in the first field and a second imaging point of the current pixel in the second field; Determine a parallax distance based on a first distance between the first imaging point and a left boundary of the first field, and a second distance between the second imaging point and a left boundary of the second field; Determine a depth value corresponding to the current pixel based on the parallax distance of the current pixel, the camera origin distance, and the camera focal length; Based on the depth values ​​of the pixels, a depth map corresponding to the disparity map is determined.

7. The method according to claim 2, characterized in that: Also includes: Determine an initial position and posture three-dimensional model of the target controlled object; Determining the initial posture three-dimensional model of the target controlled object includes: When the target control object is in an initial position, a scanner is used to scan the target control object to obtain a three-dimensional structure model of the target control object; wherein the three-dimensional structure model includes a plurality of geometric bodies; When the target control object is located within the field of view of the binocular endoscope, determining the three-dimensional position coordinates corresponding to each of the geometric bodies in the three-dimensional model structure based on the homogeneous transformation matrix of the target control object relative to the binocular endoscope coordinate system; An initial posture three-dimensional model is constructed based on the three-dimensional structure model and the three-dimensional position coordinates corresponding to each of the geometric bodies.

8. A target control object posture control device, wherein the target control object comprises at least one joint axis, each of which is mounted on a control joint axis on a robot arm, characterized in that: include: A data acquisition module, used to acquire a binocular endoscope image of the target control object in a current position and an encoder value corresponding to each of the control joint axes when each control joint axis completes rotation control of the corresponding joint axis of the target control object based on the angle control instruction at the current moment; an observation angle determination module, for determining an actual observation angle corresponding to each of the joint axes based on the binocular endoscopic image and each of the encoder values; The instruction adjustment module is used to adjust the angle control instruction of the control joint axis at the next moment based on the actual observation angle, so that the control joint axis controls the corresponding joint axis to rotate based on the adjusted angle control instruction.

9. An electronic device, characterized in that: Electronic equipment includes: one or more processors; a storage device for storing one or more programs, When one or more programs are executed by one or more processors, the one or more processors implement the target control object posture control method as claimed in any one of claims 1 to 7.

10. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to execute the target control object posture control method as described in any one of claims 1-7 when executed by a computer processor.

Images