Robot system control method, apparatus, system, and storage medium

Abstract

The application relates to a robot system control method, device, system, storage medium and computer program product. The method comprises the following steps: acquiring a real-time pose of a mechanical arm end; determining a reachable area of the mechanical arm end according to the real-time pose of the mechanical arm end and a configuration of the mechanical arm; in the case that a preset execution area is not located in the reachable area, controlling a support arm to move according to the real-time pose of the mechanical arm end and the preset execution area, updating the reachable area until the preset execution area is located in the updated reachable area; and controlling the support arm to stop moving and controlling the mechanical arm end to move in the preset execution area. The method can improve the control efficiency of mechanical arm positioning in a robot system.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to a robot system control method, apparatus, system, storage medium, and computer program product. Background Technology

[0002] In existing puncture surgical robot systems, the robotic arm is typically mounted in a fixed position. During the control of the robotic arm, there are situations where the area to be executed exceeds the robotic arm's range of motion. To control this area, the robotic arm's mounting position is often manually adjusted, allowing it to be positioned arbitrarily. However, manually positioning the robotic arm can lead to a mismatch between the planned surgical path and the robot's working range, resulting in the inability to perform the expected operation according to the planned path, and thus causing low efficiency in robotic arm positioning control. Summary of the Invention

[0003] Therefore, it is necessary to provide a robot system control method, device, system, computer-readable storage medium, and computer program product to address the above-mentioned technical problems, which can ensure that the planned path is within the working range of the robotic arm and improve the positioning control efficiency of the robotic arm.

[0004] Firstly, this application provides a robot system control method. The method includes:

[0005] Obtain the real-time pose of the robotic arm's end effector;

[0006] Based on the real-time pose of the robotic arm's end effector and the configuration of the robotic arm, the reachable area of ​​the robotic arm's end effector is determined.

[0007] If the preset execution area is not within the reachable area, the support arm is controlled to move according to the real-time pose of the robotic arm end and the preset execution area, and the reachable area is updated until the preset execution area is within the updated reachable area.

[0008] Control the support arm to stop moving, and control the end effector of the robotic arm to move within the preset execution area.

[0009] In one embodiment, the movement of the support arm is controlled based on the real-time pose of the robotic arm's end effector and a preset execution area, including:

[0010] Acquire the first pose of the fixed target in the tracker coordinate system, the second pose of the fixed target in the image coordinate system, and the real-time pose of the robotic arm end effector in the base coordinate system;

[0011] Based on the first pose of the fixed target in the tracker coordinate system and the second pose of the fixed target in the image coordinate system, register the image coordinate system and the tracker coordinate system.

[0012] Based on the registration relationship between the image coordinate system and the tracker coordinate system, the real-time pose of the robotic arm end effector, the pose of the support arm in the base coordinate system, and the real-time pose of the robotic arm end effector in the base coordinate system, the pose of the support arm in the image coordinate system is obtained.

[0013] Based on the pose of the support arm in the image coordinate system and the preset execution area, the movement of the support arm is controlled so that the preset execution area is located in the reachable area.

[0014] In one embodiment, controlling the end effector of the robotic arm to move within a preset execution area includes:

[0015] Based on the real-time pose of the robotic arm end effector in the image coordinate system and the preset planned trajectory, the robotic arm end effector is controlled to move within the preset execution area.

[0016] In one embodiment, the robot system control method further includes:

[0017] Obtain the safe zone corresponding to the target object;

[0018] Determine whether the reachable area is within a safe zone;

[0019] If the reachable area is not within the safe area, control the movement of the support arm based on the non-overlapping area between the reachable area and the safe area, update the reachable area until the updated reachable area is within the safe area; determine whether the preset execution area is within the updated reachable area;

[0020] If the reachable area is within the safe area, determine whether the preset execution area is within the reachable area.

[0021] In one embodiment, the step of determining the safe area includes:

[0022] The three-dimensional model of the target object is obtained by reconstructing the three-dimensional model based on the scanned image of the target object.

[0023] Extract the outline of the target object from the 3D model, and determine the safe area based on the outline.

[0024] In one embodiment, the movement of the support arm is controlled based on the real-time pose of the robotic arm's end effector and a preset execution area, including:

[0025] Obtain the drag force of the support arm;

[0026] Based on the drag force, the real-time pose of the robotic arm end effector, and the preset execution area, the movement of the support arm is controlled through gravity compensation.

[0027] In one embodiment, the robot system control method further includes:

[0028] The preset execution area and reachable area are displayed on a single screen.

[0029] Secondly, this application also provides a robot system control device. The device includes:

[0030] The acquisition module is used to acquire the real-time pose of the robotic arm's end effector.

[0031] The determination module is used to determine the reachable area of ​​the robotic arm end effector based on the real-time pose of the robotic arm end effector and the configuration of the robotic arm.

[0032] The control module is used to control the movement of the support arm and update the reachable area based on the real-time pose of the robotic arm end and the preset execution area when the preset execution area is not within the reachable area, until the preset execution area is within the updated reachable area.

[0033] When the preset execution area is within the updated reachable area, the control module is also used to control the support arm to stop moving and to control the end effector of the robotic arm to move within the preset execution area.

[0034] Thirdly, this application also provides a robot system, which includes: a robotic arm, a support arm, a fixed target, a tracker, an image scanning device, and a computer device; the robotic arm is fixedly connected to the support arm; the tracker is used to acquire the real-time pose of the end effector of the robotic arm and the first pose of the fixed target in the tracker's coordinate system; the image scanning device is used to scan the target object and the fixed target to obtain scanned images including the target object and the fixed target; the computer device includes a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the following steps:

[0035] Obtain the real-time pose of the robotic arm's end effector;

[0036] Based on the real-time pose of the robotic arm's end effector and the configuration of the robotic arm, the reachable area of ​​the robotic arm's end effector is determined.

[0037] If the preset execution area is not within the reachable area, the support arm is controlled to move according to the real-time pose of the robotic arm end and the preset execution area, and the reachable area is updated until the preset execution area is within the updated reachable area.

[0038] Control the support arm to stop moving, and control the end effector of the robotic arm to move within the preset execution area.

[0039] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0040] Obtain the real-time pose of the robotic arm's end effector;

[0041] Based on the real-time pose of the robotic arm's end effector and the configuration of the robotic arm, the reachable area of ​​the robotic arm's end effector is determined.

[0042] If the preset execution area is not within the reachable area, the support arm is controlled to move according to the real-time pose of the robotic arm end and the preset execution area, and the reachable area is updated until the preset execution area is within the updated reachable area.

[0043] Control the support arm to stop moving, and control the end effector of the robotic arm to move within the preset execution area.

[0044] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0045] Obtain the real-time pose of the robotic arm's end effector;

[0046] Based on the real-time pose of the robotic arm's end effector and the configuration of the robotic arm, the reachable area of ​​the robotic arm's end effector is determined.

[0047] If the preset execution area is not within the reachable area, the support arm is controlled to move according to the real-time pose of the robotic arm end and the preset execution area, and the reachable area is updated until the preset execution area is within the updated reachable area.

[0048] Control the support arm to stop moving, and control the end effector of the robotic arm to move within the preset execution area.

[0049] The aforementioned robot system control method, device, system, storage medium, and computer program product determine the reachable area of ​​the robot arm's end effector by acquiring its real-time pose and combining it with the robot arm's configuration. If the preset execution area is not within the reachable area, the system controls the movement of the support arm based on the real-time pose of the robot arm's end effector and the preset execution area, updating the reachable area until the preset execution area falls within the updated reachable area. This method of controlling the movement of the support arm when the preset execution area is not within the reachable area, ensuring that the robot arm's reachable area covers the preset execution area, improves the control efficiency of robot arm positioning in the robot system. Attached Figure Description

[0050] Figure 1 This is an application environment diagram of the robot system control method in one embodiment;

[0051] Figure 2 This is a flowchart illustrating a robot system control method in one embodiment;

[0052] Figure 3 This is a schematic diagram of the support arm in one embodiment;

[0053] Figure 4 This is a schematic diagram of the reachable area at the end of a robotic arm in one embodiment;

[0054] Figure 5 This is a schematic diagram of a sub-process of S206 in one embodiment;

[0055] Figure 6 This is a flowchart illustrating the robot system control method in another embodiment;

[0056] Figure 7 This is a schematic diagram of contour line extraction in one embodiment;

[0057] Figure 8 This is a schematic diagram of the support arm in one embodiment;

[0058] Figure 9 This is a structural schematic diagram of the support arm's X-axis in one embodiment;

[0059] Figure 10 This is a schematic diagram of the support arm's Z-axis structure in one embodiment;

[0060] Figure 11 This is a schematic diagram of the structure of the support arm Z1 axis in one embodiment;

[0061] Figure 12 This is a schematic diagram of the robotic arm in one embodiment;

[0062] Figure 13 This is a schematic diagram of the overall flow of a robot system control method in one embodiment;

[0063] Figure 14 This is a flowchart illustrating the robot system control method in yet another embodiment;

[0064] Figure 15 This is a schematic diagram of the movement of the robotic arm's end effector from an unreachable area to an reachable area in one embodiment.

[0065] Figure 16 This is a schematic diagram of multiple coordinate systems in one embodiment;

[0066] Figure 17 This is a schematic diagram illustrating the establishment of the coordinate system at the end effector of a robotic arm in one embodiment;

[0067] Figure 18 This is a schematic diagram of the coordinate system transformations in one embodiment;

[0068] Figure 19 This is a schematic diagram of automatic motion trajectory planning in one embodiment;

[0069] Figure 20 This is a flowchart illustrating the automatic planning of motion trajectories in one embodiment;

[0070] Figure 21 This is a schematic diagram of a gravity compensation method in one embodiment;

[0071] Figure 22 This is a structural block diagram of a robot system control device in one embodiment. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0073] The robot system control method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, the robot system control method is applied to a robot system including a robotic arm 101, a support arm 102, a fixed target 103, a tracker 104, an image scanning device 105, and a computer device 106. The robotic arm 101 is fixedly connected to the support arm 102. The tracker 104 is used to acquire the real-time pose of the robotic arm's end effector and the first pose of the fixed target in the tracker's coordinate system. The image scanning device 105 is used to scan the target object and the fixed target 103 to obtain scanned images including the fixed target 103 and the target object. The robot system control method is executed by the computer device 106. The computer device 106 acquires the real-time pose of the robotic arm's end effector; based on the real-time pose of the robotic arm's end effector and the configuration of the robotic arm, it determines the reachable area of ​​the robotic arm's end effector; if the preset execution area is not within the reachable area, based on the real-time pose of the robotic arm's end effector and the preset execution area, it controls the support arm 102 to move, updating the reachable area until the preset execution area is within the updated reachable area; it controls the support arm 102 to stop moving and controls the robotic arm's end effector to move within the preset execution area. The computer device 106 can be a terminal or a server. The robot system control method provided in this embodiment can be executed by the terminal or the server alone, or by the terminal and the server working together. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. The server can be a standalone server or a server cluster consisting of multiple servers.

[0074] In one embodiment, such as Figure 2 As shown, a robot system control method is provided, which can be applied to... Figure 1 Taking computer device 106 as an example, the following steps are included:

[0075] S202, obtain the real-time pose of the robotic arm's end effector.

[0076] A robotic arm is a programmable mechanical arm that is connected by joints and can rotate or translate. A robot system includes a robotic arm. Pose includes position and orientation; changes in pose involve both translation and rotation. Changes in position are translation, and changes in orientation are rotation. The movement of the robotic arm causes continuous changes in the pose of its end effector, and computer equipment acquires the real-time pose of the end effector.

[0077] S204, based on the real-time pose of the robotic arm's end effector and the configuration of the robotic arm, determine the reachable area of ​​the robotic arm's end effector.

[0078] The reachable area refers to the region encompassing all points that the robotic arm's end effector can reach. The robotic arm's configuration refers to its structure and connections within the robot system. In some embodiments, the robot system includes a robotic arm and a support arm, with the robotic arm fixedly connected to the support arm. The reachable area of ​​the robotic arm's end effector refers to its range of motion; as the support arm and the robotic arm move, the reachable area of ​​the robotic arm changes accordingly.

[0079] The robotic arm has multiple degrees of freedom. Based on the real-time pose of the robotic arm's end effector, a computer can determine the pose of the robotic arm's base using robot forward kinematics. Since the robotic arm is fixedly connected to the support arm, the relative poses of the robotic arm base and the support arm can be obtained through design parameters. From the relative poses of the robotic arm base and the support arm, and the pose of the robotic arm base, the real-time pose of the support arm can be determined.

[0080] The support arm has multiple degrees of freedom, but this application does not specifically limit the number of degrees of freedom. For example... Figure 3 The diagram shows a schematic of the support arm in one embodiment. The support arm's X-axis 301 direction consists of a lead screw and a guide rail, allowing horizontal movement; the Z-axis 302 direction also consists of a lead screw and a guide rail, allowing vertical movement; the Z1-axis 303 direction is directly driven by a motor, allowing rotation around the Z-axis. The support arm moves in multiple axial directions. The computer device determines the support arm's motion area based on the real-time pose of the support arm and the available space in multiple axial directions. Similarly, the computer device determines the robotic arm's motion area based on the real-time pose of the robotic arm's end effector and the available space in multiple axial directions. The robotic arm's own motion area, plus the support arm's motion area, yields the reachable area of ​​the robotic arm's end effector. Figure 4 The diagram shows the reachable area at the end of the robotic arm.

[0081] S206, when the preset execution area is not within the reachable area, control the movement of the support arm according to the real-time pose of the robotic arm end and the preset execution area, update the reachable area until the preset execution area is within the updated reachable area.

[0082] The preset execution area refers to the preset activity area of ​​the robotic arm's end effector. In some embodiments, the preset execution area can be the lesion area of ​​the target object, and the robotic arm's end effector moves within the lesion area according to a planned path. The computer device determines whether the preset execution area is within the reachable region. If the preset execution area is not within the reachable region, the computer device controls the movement of the support arm based on the real-time pose of the robotic arm's end effector and the preset execution area. Since the preset execution area is not within the reachable region, the computer device moves the support arm towards the preset execution area. As the support arm moves, the reachable region of the robotic arm's end effector is continuously updated until the preset execution area is within the updated reachable region. The preset execution area being within the updated reachable region indicates that any point within the preset execution area is reachable by the robotic arm's end effector, which facilitates the movement of the robotic arm's end effector within the preset execution area.

[0083] S208 controls the support arm to stop moving and controls the end effector of the robotic arm to move within a preset execution area.

[0084] When the preset execution area is within the reachable area of ​​the robotic arm, the computer device controls the support arm to stop moving, ensuring that the preset execution area is always within the reachable area of ​​the robotic arm.

[0085] In some embodiments, if the preset execution area is updated, it is determined whether the updated execution area is within the reachable region of the robotic arm's end effector. If the updated execution area is not within the reachable region, the support arm is controlled to move based on the real-time pose of the robotic arm's end effector and the updated execution area, updating the reachable region until the updated execution area is within the updated reachable region. At this point, the support arm stops moving, and the robotic arm is controlled to move within the updated execution area. Therefore, even when the preset execution area is updated, controlling the movement of the support arm ensures that the preset execution area is always within the reachable region of the robotic arm's end effector, improving the control efficiency of robotic arm positioning in the robot system.

[0086] In the aforementioned robot system control method, the reachable area of ​​the robotic arm's end effector is determined by acquiring its real-time pose. If the preset execution area is not within the reachable area, the support arm is controlled to move based on the real-time pose of the end effector and the preset execution area, updating the reachable area until the preset execution area falls within the updated reachable area. This method, which ensures the reachable area of ​​the robotic arm's end effector covers the preset execution area even when the preset execution area is not within it, improves the efficiency of robotic arm positioning in the robot system.

[0087] In one embodiment, such as Figure 5 As shown, the movement of the support arm is controlled based on the real-time pose of the robotic arm's end effector and the preset execution area, including:

[0088] S502, acquires the first pose of the fixed target in the tracker coordinate system, the second pose of the fixed target in the image coordinate system, and the real-time pose of the robotic arm end effector in the base coordinate system.

[0089] The robot system includes a fixed target, a tracker, and an image scanning device. The tracker can be an optical tracker, and the fixed target has optical markers that the tracker can recognize. The computer equipment establishes a tracker coordinate system based on the tracker. The tracker is used to acquire the first pose of the fixed target in the tracker coordinate system. The computer equipment obtains the first pose of the fixed target in the tracker coordinate system through the tracker.

[0090] An image scanning device is a device capable of scanning images of target objects and stationary targets. For example, an image scanning device can be a CT (Computed Tomography) scanner. The image scanning device is used to scan target objects and stationary targets. The computer equipment establishes an image coordinate system based on the image scanning device and obtains the second pose of the stationary target in the image coordinate system.

[0091] The steps for obtaining the real-time pose of the robotic arm end effector in the base coordinate system are as follows: The computer device obtains the real-time pose of the robotic arm end effector in the base coordinate system based on the real-time pose of the robotic arm end effector and through the kinematics of the robotic arm.

[0092] S504, based on the first pose of the fixed target in the tracker coordinate system and the second pose of the fixed target in the image coordinate system, register the image coordinate system and the tracker coordinate system.

[0093] The computer equipment registers the image coordinate system and the tracker coordinate system based on the first pose of the fixed target in the tracker coordinate system and the second pose of the fixed target in the image coordinate system, thus obtaining the registration relationship between the image coordinate system and the tracker coordinate system.

[0094] S506. Based on the registration relationship between the image coordinate system and the tracker coordinate system, the real-time pose of the robotic arm end effector, the pose of the support arm in the base coordinate system, and the real-time pose of the robotic arm end effector in the base coordinate system, the pose of the support arm in the image coordinate system is obtained.

[0095] The computer device obtains the pose of the support arm in the tracker coordinate system from the real-time pose of the end effector of the robotic arm, the pose of the support arm in the base coordinate system, and the real-time pose of the end effector of the robotic arm in the base coordinate system. Then, based on the registration relationship between the image coordinate system and the tracker coordinate system, the pose of the support arm in the image coordinate system is obtained.

[0096] S508 controls the movement of the support arm based on the pose of the support arm in the image coordinate system and the preset execution area, so that the preset execution area is located in the reachable area.

[0097] The computer takes the pose of the support arm in the image coordinate system as the starting point and controls the support arm to move towards the preset execution area, so that the preset execution area is located in the reachable area of ​​the end of the robotic arm.

[0098] In this embodiment, by acquiring the first pose of the fixed target in the tracker coordinate system, the second pose of the fixed target in the image coordinate system, and the real-time pose of the end effector of the robotic arm in the base coordinate system, the pose of the support arm in the image coordinate system is determined. The support arm is then controlled to move towards the preset execution area in the image coordinate system, thereby ensuring that the preset execution area is always within the reachable area of ​​the end effector of the robotic arm, which improves the control efficiency of the robotic arm's positioning in the robot system.

[0099] In one embodiment, such as Figure 6 As shown, the robot control method also includes:

[0100] S602, obtain the safe zone corresponding to the target object; determine whether the reachable area is within the safe zone.

[0101] The safe zone is the area where no danger will occur during the movement of the robotic arm's end effector. The preset execution area is within the safe zone. The computer device obtains the safe zone corresponding to the target object and determines whether the reachable area is within the safe zone.

[0102] S604, when the reachable area is not within the safe area, control the movement of the support arm based on the non-overlapping area between the reachable area and the safe area, update the reachable area until the updated reachable area is within the safe area; determine whether the preset execution area is within the updated reachable area.

[0103] The statement that the reachable area is not within the safe zone indicates that the reachable area of ​​the robotic arm may contain portions outside the safe zone, or that the reachable area is not within the safe zone at all. The computer device acquires the non-overlapping area between the reachable area and the safe zone. Based on this non-overlapping area, the computer device controls the support arm to move towards the safe zone. As the support arm moves, the real-time pose of the robotic arm's end effector is continuously updated, and the reachable area of ​​the end effector is continuously updated until the updated reachable area falls within the safe zone. Once the updated reachable area is within the safe zone, the computer device further determines whether the preset execution area is also within the updated reachable area.

[0104] S606, if the reachable area is within the safe area, determine whether the preset execution area is within the reachable area.

[0105] In cases where the reachable area at the end of the robotic arm is within a safe zone, the computer device directly determines whether the preset execution area is within the reachable zone.

[0106] In this embodiment, before controlling the movement of the support arm, the safe area corresponding to the target object is obtained, and it is determined whether the reachable area is within the safe area. If the reachable area is not within the safe area, the support arm movement is controlled based on the non-overlapping area between the reachable area and the safe area, thereby ensuring that the updated reachable area is within the safe area. This, in turn, determines whether the preset execution area is within the updated reachable area. This facilitates the control of the support arm movement, ensuring that the reachable area of ​​the robotic arm's end effector is within the safe area and preventing the end effector from entering an unsafe area. When the reachable area is within the safe area, directly determining whether the preset execution area is within the reachable area improves the control efficiency of the robotic arm's positioning in the robot system.

[0107] In one embodiment, the step of determining the safe area includes: acquiring a three-dimensional model of the target object, the three-dimensional model being obtained by three-dimensional reconstruction based on a scanned image of the target object; extracting the outline of the target object from the three-dimensional model; and determining the safe area based on the outline.

[0108] The process involves using computer equipment to acquire scanned images of the target object obtained by an image scanning device. Three-dimensional reconstruction is then performed on the scanned images to obtain a three-dimensional model of the target object. The contour lines of the target object are then extracted from the three-dimensional model. For example... Figure 7 The diagram illustrates the extraction of the contour line. A safe area is determined based on the contour line. In some embodiments, the computer device defines the area above the contour line as the safe area. In other embodiments, the computer device extends outwards by a predetermined distance along the contour line in the 3D model to form a safe contour line. The area above the safe contour line is then defined as the safe area.

[0109] In this embodiment, by extracting the outline of the target object from its 3D model, determining the safe area based on the outline, and controlling the movement of the robotic arm end effector within the safe area, the robotic arm end effector can be prevented from moving to an unsafe area.

[0110] In one embodiment, the method further includes: controlling the end effector of the robotic arm to move within the preset execution area when the preset execution area is within the reachable area.

[0111] The fact that the preset execution area is within the reachable area indicates that the end effector of the robotic arm can reach the preset execution area, and the computer device controls the end effector of the robotic arm to move within the preset execution area.

[0112] In this embodiment, the robotic arm end effector is controlled to move within the preset execution area when the preset execution area is within the reachable area.

[0113] In one embodiment, controlling the movement of the support arm based on the real-time pose of the robotic arm end effector and a preset execution area includes: acquiring the drag force of the support arm; and controlling the movement of the support arm through gravity compensation based on the drag force, the real-time pose of the robotic arm end effector, and the preset execution area.

[0114] The drag force of the support arm is generated when the operator drags it. The support arm can also be controlled by dragging. The computer equipment acquires the drag force of the support arm. When the support arm is dragged to control its movement, gravity will cause a control deviation. The computer equipment, based on the drag force, the real-time pose of the robotic arm's end effector, and the preset execution area, controls the support arm's movement through gravity compensation, thereby eliminating the influence of gravity during dragging and improving the accuracy of support arm control.

[0115] In this embodiment, by using the dragging force of the support arm, the real-time pose of the robotic arm end effector, and the preset execution area, the movement of the support arm is controlled through gravity compensation, which can eliminate the influence of gravity during the dragging process and improve the accuracy of support arm control.

[0116] To illustrate the robot system control method and its effects in this solution in detail, a specific embodiment is described below:

[0117] The robot system control method is applied to a robot control system, which includes a robotic arm, a support arm, a fixed target, a tracker, an image scanning device, and a computer. The robotic arm is fixedly connected to the support arm. The tracker is used to acquire the real-time pose of the robotic arm's end effector and the first pose of the fixed target in the tracker's coordinate system. The image scanning device is used to scan the target object and the fixed target to obtain scanned images of them. The computer includes a memory and a processor. The memory stores the computer program, and the processor executes the computer program to implement the robot system control method. Figure 8 The diagram shown is a structural schematic of a support arm in one embodiment. The support arm includes an X-axis, a Z-axis, and a Z1-axis. Figure 9 The diagram shows the structure of the support arm along the X-axis. The support arm, mainly composed of a lead screw and guide rails, can move horizontally along the X-axis. Figure 10 The diagram shows the structure of the support arm's Z-axis. The Z-axis consists of a lead screw and a guide rail, allowing for vertical movement. Figure 11 The diagram shows the structure of the support arm along the Z1 axis. The Z1 axis is directly driven by a motor and can rotate around it. The support arm moves in multiple axial directions. The computer determines the support arm's motion area based on its real-time pose and the available space in these multiple axial directions. Similarly, the computer determines the robotic arm's motion area based on its real-time pose and the available space in these multiple axial directions. The robotic arm's own motion area, plus the support arm's motion area, gives the reachable area of ​​the robotic arm's end effector. Figure 12 The diagram shows the structure of the robotic arm. The robotic arm includes three rotation axes and one translation axis.

[0118] like Figure 13 The diagram shows the overall flow of the robot system control method. The computer acquires the real-time pose of the robotic arm's end effector. Based on the real-time pose and the configuration of the robotic arm, the reachable area of ​​the end effector is determined. The safe area corresponding to the target object is obtained. If the preset execution area is within the safe area, it is determined whether the reachable area is also within the safe area. The steps for determining the safe area include: the computer acquires a 3D model of the target object, which is obtained through 3D reconstruction based on a scanned image of the target object; the contour lines of the target object are extracted from the 3D model; and the safe area is determined based on the contour lines. If the reachable area is not within the safe area, the support arm is controlled to move based on the non-overlapping area between the reachable area and the safe area, updating the reachable area until the updated reachable area is within the safe area; it is determined whether the preset execution area is within the updated reachable area; and if the reachable area is within the safe area, it is determined whether the preset execution area is also within the reachable area. Figure 14The diagram illustrates the control method of the robot system. The computer performs image segmentation on the target image, distinguishing the lesion region within the target object and reconstructing a 3D model to provide image and trajectory planning information for subsequent methods. Coordinate system registration fuses the coordinates of the lesion, fixed target, robotic arm, and support arm into the image coordinate system, determining their relative positions. Path planning selects needle entry points on the surface of the target object and on the lesion region on the reconstructed 3D model; the lesion region is often located inside the target object. The path is automatically calculated in the coordinate system based on the two selected needle entry points. The needle path is simulated in the 3D model, allowing observation of important blood vessels, organs, trachea, etc., to avoid needle entry into critical areas.

[0119] like Figure 15 The diagram illustrates the movement of a robotic arm's end effector from an unreachable region to a reachable region. When the preset execution area is not within the reachable region, the computer controls the movement of the support arm based on the real-time pose of the robotic arm's end effector and the preset execution area, updating the reachable region until the preset execution area falls within the updated reachable region. Once the preset execution area is within the updated reachable region, the support arm stops moving, and the computer controls the robotic arm's end effector to move within the preset execution area based on its real-time pose in the image coordinate system and the preset planned trajectory. The computer controls the movement of the support arm based on the real-time pose of the robotic arm's end effector and the preset execution area.

[0120] like Figure 16 The diagram illustrates multiple coordinate systems. The main coordinate systems involved include the image coordinate system, the fixed target coordinate system, the base coordinate system, the support arm coordinate system, the tracker coordinate system, and the robotic arm end effector coordinate system. Figure 17 The diagram illustrates the establishment of the robotic arm's end-effector coordinate system. Reflective spheres are used as navigation markers at the end of the robotic arm, and three reflective spheres are used to construct the end-effector coordinate system. The computer acquires the first pose of the fixed target in the tracker coordinate system, the second pose of the fixed target in the image coordinate system, and the real-time pose of the robotic arm's end-effector in the base coordinate system. Based on the first pose of the fixed target in the tracker coordinate system and the second pose of the fixed target in the image coordinate system, the image coordinate system and the tracker coordinate system are registered. Based on the registration relationship between the image coordinate system and the tracker coordinate system, the real-time pose of the robotic arm's end-effector, the pose of the support arm in the base coordinate system, and the real-time pose of the robotic arm's end-effector in the base coordinate system, the pose of the support arm in the image coordinate system is obtained. (The diagram is incomplete in the original text.) Figure 18The diagram illustrates the coordinate system transformations. The image scanned by the image scanning device is segmented to obtain a segmented image. The positional relationship between the lesion and the fixed target is then obtained from the segmented image. The pose of the fixed target in the tracker's coordinate system is measured using a tracker. The robotic arm has a reflective ball at its end. The pose of the robotic arm's end in the tracker's coordinate system is measured using a tracker, and then the position of the robotic arm base in the tracker's coordinate system is obtained through forward kinematics calculations. The relative position of the robotic arm base and the support arm is determined by design parameters, thus determining the relative position of the entire robot system in the image coordinate system. The formula for determining the pose of the support arm in the image coordinate system is:

[0121]

[0122] This indicates the pose of the support arm in the image coordinate system. This represents the second pose of the fixed target in the image coordinate system. This represents the first pose of the fixed target in the tracker's coordinate system. This indicates the real-time pose of the robotic arm's end effector. This represents the real-time pose of the robotic arm's end effector in the base coordinate system. This indicates the pose of the support arm in the base coordinate system.

[0123] Based on the pose of the support arm in the image coordinate system and the preset execution area, the movement of the support arm is controlled so that the preset execution area is located in the reachable area.

[0124] When the preset execution area is within the reachable area, the computer equipment controls the end effector of the robotic arm to move within the preset execution area according to a preset planned trajectory. For example... Figure 19 This is a schematic diagram for automatically planning motion trajectories. In the diagram, × indicates an obstacle, ○ represents the starting point, ★ represents the ending point, √ represents the Closed list, and □ represents the Open list. The preset execution area is divided into a two-dimensional array, with each element corresponding to a small square. The robotic arm's end effector moves from the starting point to the ending point within the preset execution area. Squares to be detected during path planning are stored in the Open list, while detected squares are stored in the Closed list. For example... Figure 20 The diagram illustrates the process of automatically planning a motion trajectory. The starting node is added to the Open list. The node with the smallest F value in the Open list is selected as the current node and added to the Close list. A path is generated based on the parent node's direction. Here, F = G + H determines which cell to move to, G represents the cost from the initial position along the generated path to the cell to be detected, and H represents the estimated movement cost from the cell to be detected to the target point.

[0125] In some embodiments, the movement of the support arm can be controlled based on the real-time pose of the robotic arm's end effector and a preset execution area, and can also be controlled by dragging. Figure 21 The diagram illustrates the gravity compensation method. Here, kp represents the position feedback error proportionality coefficient, and kd represents the velocity feedback error proportionality coefficient. The computer acquires the drag force of the support arm. Based on the drag force, the real-time pose of the robotic arm's end effector, and the preset execution area, gravity compensation is used to control the support arm's movement. During the drag control of the support arm, gravity compensation helps eliminate the influence of gravity and improves the control accuracy of the support arm.

[0126] The aforementioned robot system control method determines the reachable area of ​​the robotic arm's end effector by acquiring its real-time pose and combining it with the robotic arm's configuration. If the preset execution area is not within the reachable area, the method controls the movement of the support arm based on the real-time pose of the robotic arm's end effector and the preset execution area, updating the reachable area until the preset execution area falls within the updated reachable area. This method, which controls the movement of the support arm even when the preset execution area is not within the reachable area, ensures that the reachable area of ​​the robotic arm's end effector covers the preset execution area, thus improving the control efficiency of the robotic arm's positioning in the robot system.

[0127] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0128] Based on the same inventive concept, this application also provides a robot system control device for implementing the robot system control method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more robot system control device embodiments provided below can be found in the limitations of the robot system control method described above, and will not be repeated here.

[0129] In one embodiment, such as Figure 22 As shown, a robot system control device 100 is provided. The device includes: an acquisition module 120, a determination module 140, a first control module 160, and a second control module 180, wherein:

[0130] The acquisition module 120 is used to acquire the real-time pose of the robotic arm's end effector.

[0131] The determination module 140 is used to determine the reachable area of ​​the robotic arm end effector based on the real-time pose of the robotic arm end effector and the configuration of the robotic arm.

[0132] The first control module 160 is used to control the movement of the support arm and update the reachable area based on the real-time pose of the end effector and the preset execution area when the preset execution area is not within the reachable area, until the preset execution area is within the updated reachable area.

[0133] The second control module 180 is used to control the support arm to stop moving and to control the end effector of the robotic arm to move within a preset execution area.

[0134] The aforementioned robot system control device, by acquiring the real-time pose of the robotic arm's end effector and combining it with the robotic arm's configuration, determines the reachable area of ​​the end effector. If the preset execution area is not within the reachable area, the device controls the movement of the support arm based on the real-time pose of the end effector and the preset execution area, updating the reachable area until the preset execution area falls within the updated reachable area. Controlling the support arm's movement ensures that the reachable area of ​​the robotic arm's end effector covers the preset execution area, thus improving the control efficiency of the robotic arm's positioning in the robot system.

[0135] Each module in the aforementioned robot system control device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the operations corresponding to each module.

[0136] In one embodiment, a robot system control system is provided. The system includes: a robotic arm, a support arm, a fixed target, a tracker, an image scanning device, and a computer device. The robotic arm is fixedly connected to the support arm. The tracker is used to acquire the real-time pose of the robotic arm's end effector and the first pose of the fixed target in the tracker's coordinate system. The image scanning device is used to scan the target object and the fixed target to obtain scanned images of the target object and the fixed target. The computer device can be a terminal. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface.

[0137] Those skilled in the art will understand that the above-described computer device composition structure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0138] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: acquiring the real-time pose of a robotic arm end effector; determining the reachable region of the robotic arm end effector based on the real-time pose of the robotic arm end effector and the configuration of the robotic arm; if a preset execution region is not within the reachable region, controlling the movement of a support arm based on the real-time pose of the robotic arm end effector and the preset execution region to update the reachable region until the preset execution region is within the updated reachable region; controlling the support arm to stop moving, and controlling the robotic arm end effector to move within the preset execution region.

[0139] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Furthermore, any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory.

[0140] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0141] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A robot system control device, characterized in that, The device includes: The acquisition module is used to acquire the real-time pose of the robotic arm's end effector. The determination module is used to determine the reachable area of ​​the robotic arm end effector based on the real-time pose of the robotic arm end effector and the configuration of the robotic arm; the reachable area refers to the region composed of all position points that the robotic arm end effector can reach. The control module is used to control the movement of the support arm and update the reachable area based on the real-time pose of the end effector of the robotic arm and the preset execution area when the preset execution area is not within the reachable area, until the preset execution area is within the updated reachable area. When the preset execution area is within the updated reachable area, the control module is also used to control the support arm to stop moving and to control the end effector of the robotic arm to move within the preset execution area; The control module is also used for: The first pose of the fixed target in the tracker coordinate system, the second pose of the fixed target in the image coordinate system, and the real-time pose of the end effector of the robotic arm in the base coordinate system are obtained. Based on the first pose of the fixed target in the tracker coordinate system and the second pose of the fixed target in the image coordinate system, the image coordinate system and the tracker coordinate system are registered. Based on the real-time pose of the robotic arm's end effector, the pose of the support arm in the base coordinate system, and the real-time pose of the robotic arm's end effector in the base coordinate system, the pose of the support arm in the tracker coordinate system is obtained; based on the registration relationship between the image coordinate system and the tracker coordinate system, and the pose of the support arm in the tracker coordinate system, the pose of the support arm in the image coordinate system is obtained; the real-time pose of the robotic arm's end effector is acquired by the tracker. Starting from the pose of the support arm in the image coordinate system, the support arm is controlled to move towards the preset execution area, so that the preset execution area is located in the reachable area.

2. The apparatus according to claim 1, characterized in that, The control module is also used for: Based on the real-time pose of the robotic arm end effector in the image coordinate system and the preset planned trajectory, the robotic arm end effector is controlled to move within the preset execution area.

3. The apparatus according to claim 1, characterized in that, The control module is also used for: Obtain the safe zone corresponding to the target object; Determine whether the reachable area is within the safe area; If the reachable area is not within the safe area, the support arm is controlled to move according to the non-overlapping area between the reachable area and the safe area, and the reachable area is updated until the updated reachable area is within the safe area; Determine whether the preset execution region is within the updated reachable region; If the reachable area is within the safe area, determine whether the preset execution area is within the reachable area.

4. The apparatus according to claim 3, characterized in that, The control module is also used for: A three-dimensional model of the target object is obtained, which is obtained by three-dimensional reconstruction based on a scanned image of the target object; Extract the outline of the target object from the 3D model, and determine the safe area based on the outline.

5. The apparatus according to claim 1, characterized in that, The control module is also used for: Obtain the drag force of the support arm; Based on the drag force, the real-time pose of the robotic arm end effector, and the preset execution area, the movement of the support arm is controlled through gravity compensation.

6. The apparatus according to claim 1, characterized in that, The control module is also used for: The preset execution area and the reachable area are displayed on a single display interface.

7. The apparatus according to claim 4, characterized in that, The control module is also used for: Extend a predetermined distance outward along the contour line in the three-dimensional model to form a safety contour line; The area outside the safety contour line is defined as the safety zone.

8. The apparatus according to claim 1, characterized in that, The support arm includes an X-axis, a Z-axis, and a Z1-axis; the X-axis direction of the support arm consists of a lead screw and a guide rail, used for horizontal movement; the Z-axis direction of the support arm consists of a lead screw and a guide rail, used for vertical movement; the Z1-axis direction of the support arm is directly driven by a motor, used for rotational movement around the Z-axis.

9. A robot system, characterized in that, The robot system includes: a robotic arm, a support arm, a fixed target, a tracker, an image scanning device, and a computer device; The robotic arm is fixedly connected to the support arm; The tracker is used to collect the real-time pose of the end effector of the robotic arm and the first fixed pose of the fixed target in the tracker's coordinate system. The image scanning device is used to scan the target object and the fixed target to obtain a scanned image including the target object and the fixed target; The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the robot system control device according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the robot system control device according to any one of claims 1 to 8.

Images