Methods, devices, systems, and computer equipment for switching the motion speed of robotic arms

Abstract

This application relates to a method, apparatus, system, computer device, storage medium, and computer program product for switching the movement speed of a robotic arm. The method includes: acquiring the real-time positions of the end effector of the surgical robot's robotic arm and the target part of a target object; determining the distance between the end effector and the target part based on the real-time positions; and controlling and switching the movement speed of the surgical robot's robotic arm based on the relationship between the distance and a preset distance threshold. This method can accurately determine the timing for switching the movement speed of the robotic arm.

Description

Technical Field

[0001] This application relates to the technical field, and in particular to a method, apparatus, system, computer equipment, storage medium, and computer program product for switching the movement speed of a robotic arm. Background Technology

[0002] With the advancement of technology, robotics has matured and is being widely applied. Robots typically include robotic arms, and ensuring the safety of the robotic arm itself and objects in the environment is one of the key research issues when facing complex task requirements and unknown working environments.

[0003] In neurosurgical robotic bone screw contact registration, spinal surgery robotics, and hip replacement robotics, during the positioning and imaging process of the C-arm, it is usually necessary to switch the robotic arm's movement speed to prevent collisions caused by excessively rapid movement. However, in related technologies, it is often impossible to accurately determine the timing of switching the robotic arm's movement speed, leading to inappropriate movement speeds and potentially causing unexpected collisions.

[0004] Therefore, there is a problem with the inaccurate judgment of the timing of the switching of the robotic arm's movement speed in related technologies. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, apparatus, computer device, computer-readable storage medium, and computer program product for switching the movement speed of a robotic arm, which can more accurately determine the timing of switching the movement speed of the robotic arm, in order to address the above-mentioned technical problems.

[0006] Firstly, this application provides a method for switching the movement speed of a robotic arm. The method includes:

[0007] To obtain the real-time position of the surgical robot's end effector and the target part of the target object;

[0008] Based on the real-time position, determine the distance between the robotic arm end effector and the target area;

[0009] The movement speed of the surgical robot's robotic arm is controlled and switched based on the relationship between the distance and a preset distance threshold.

[0010] In one embodiment, controlling the switching of the robotic arm's movement speed based on the relationship between the distance and a preset distance threshold includes:

[0011] If the distance is greater than the preset distance threshold, the movement speed of the robotic arm is controlled to a first movement speed;

[0012] When the distance is less than the preset distance threshold, the movement speed of the robotic arm is controlled to a second movement speed; the second movement speed is less than the first movement speed.

[0013] In one embodiment, prior to the step of obtaining the real-time position of the surgical robot's end effector and the target part of the target object, the method further includes:

[0014] Obtain the robot type corresponding to the surgical robot;

[0015] Based on the robot type, identify the end effector of the robotic arm and the target area.

[0016] In one embodiment, identifying the robotic arm end effector and the target part based on the robot type includes:

[0017] Obtain a target recognition model; the target recognition model is trained based on sample depth image data;

[0018] Based on the target recognition model and the robot type, the end effector of the robotic arm and the target part are identified.

[0019] In one embodiment, obtaining the target recognition model includes:

[0020] Acquire the sample depth image data;

[0021] The target sample area and the robotic arm end effector in the sample depth image data are labeled to obtain labeled sample depth image data;

[0022] The target recognition model is obtained by training the model based on the sample depth image data and the labeled sample depth image data.

[0023] In one embodiment, the robot type includes a first type matched with neurosurgical robots and a second type matched with spinal surgery robots or hip replacement surgery robots; acquiring the sample depth image data includes:

[0024] For the first type, in the bone nail contact registration scenario, the depth image data of the first sample is obtained;

[0025] For the second type, in the C-arm imaging positioning scenario, acquire the second sample depth image data;

[0026] The sample depth image data is obtained based on the first sample depth image data and the second sample depth image data.

[0027] In one embodiment, acquiring the real-time position of the surgical robot's end effector and the target part of the target object includes:

[0028] Real-time acquisition of depth images including the robotic arm end effector and the target area;

[0029] The depth image is matched with the target recognition model to identify the tool image data corresponding to the end effector of the robotic arm and the part image data corresponding to the target part in real time.

[0030] Based on the tool image data and the part image data, the real-time positions of the robotic arm end effector and the target part are determined.

[0031] In one embodiment, determining the distance between the robotic arm end effector and the target location based on the real-time position includes:

[0032] Determine the center point or bounding box of the tool image data as the first center point or first bounding box;

[0033] Determine the center point or bounding box of the image data of the aforementioned region as the second center point or second bounding box;

[0034] Determine the distance between the first center point and the second center point, or determine the distance between the first bounding box and the second bounding box, to obtain the distance between the end effector of the robotic arm and the target part.

[0035] Secondly, this application also provides a device for switching the movement speed of a robotic arm. The device includes:

[0036] The acquisition module is used to acquire the real-time position of the surgical robot's end effector and the target part of the target object;

[0037] The determination module is used to determine the distance between the end effector of the robotic arm and the target part based on the real-time position;

[0038] The control module is used to control and switch the movement speed of the robotic arm of the surgical robot according to the relationship between the distance and a preset distance threshold.

[0039] Thirdly, this application also provides a system for switching the movement speed of a robotic arm. The system includes: a depth camera, a surgical robot, and a controller;

[0040] The depth camera is used to acquire depth images including the end effector of the surgical robot and the target part of the target object; the depth images are used to obtain the real-time position of the end effector and the target part.

[0041] The controller is used to determine the distance between the end effector of the robotic arm and the target part based on the real-time position;

[0042] The controller is also used to control and switch the movement speed of the robotic arm of the surgical robot according to the relationship between the distance and a preset distance threshold.

[0043] Fourthly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0044] To obtain the real-time position of the surgical robot's end effector and the target part of the target object;

[0045] Based on the real-time position, determine the distance between the end effector of the robotic arm and the target area;

[0046] The movement speed of the surgical robot's robotic arm is controlled and switched based on the relationship between the distance and a preset distance threshold.

[0047] Fifthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0048] To obtain the real-time position of the surgical robot's end effector and the target part of the target object;

[0049] Based on the real-time position, determine the distance between the robotic arm end effector and the target area;

[0050] The movement speed of the surgical robot's robotic arm is controlled and switched based on the relationship between the distance and a preset distance threshold.

[0051] Sixthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0052] To obtain the real-time position of the surgical robot's end effector and the target part of the target object;

[0053] Based on the real-time position, determine the distance between the robotic arm end effector and the target area;

[0054] The movement speed of the surgical robot's robotic arm is controlled and switched based on the relationship between the distance and a preset distance threshold.

[0055] The aforementioned method, apparatus, system, computer equipment, storage medium, and computer program product for switching the movement speed of the robotic arm acquires the real-time position of the end effector of the surgical robot and the target part of the target object; determines the distance between the end effector and the target part based on the real-time position; and controls the switching of the movement speed of the surgical robot's robotic arm based on the relationship between the distance and a preset distance threshold.

[0056] Thus, based on the real-time position between the robotic arm's end effector and the target area, the real-time distance between them can be determined. By comparing this real-time distance with a preset distance threshold set to prevent the robotic arm from colliding with the target area due to excessive proximity, the timing for switching the robotic arm's movement speed from fast to slow can be more accurately determined as the end effector approaches the target area. This prevents collisions with the target area due to excessive speed, reducing the occurrence of unexpected collisions. Furthermore, the timing for switching the robotic arm's movement speed from slow to fast can be more accurately determined as the end effector moves away from the target area, allowing the surgical robot to move at a faster speed and improving its operational efficiency. Therefore, by using this solution to determine the timing for controlling and switching the robotic arm's movement speed based on the relationship between the distance between the end effector and the target area and a preset distance threshold, the switching timing for controlling and switching the robotic arm's movement speed can be accurately determined. Attached Figure Description

[0057] Figure 1 This is a flowchart illustrating a method for switching the movement speed of a robotic arm in one embodiment;

[0058] Figure 2 This is a flowchart illustrating the steps for obtaining a target recognition model in one embodiment;

[0059] Figure 3 This is a schematic diagram illustrating the construction of a training network in one embodiment;

[0060] Figure 4 This is a flowchart illustrating a method for switching the movement speed of a robotic arm in another embodiment;

[0061] Figure 5 This is a flowchart illustrating another method for switching the movement speed of a robotic arm in one embodiment;

[0062] Figure 6 A structural block diagram of a switching device for the movement speed of a robotic arm in one embodiment;

[0063] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0065] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0066] In one embodiment, such as Figure 1 As shown, a method for switching the movement speed of a robotic arm is provided. This embodiment illustrates the application of this method to a computer device. It is understood that the computer device can be a terminal or a system including a terminal and a server, with the method implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0067] Step S110: Obtain the real-time position of the surgical robot's end-effector and the target part of the target object.

[0068] The target object is not limited to patients, but can also be a model prosthesis, etc., which can be used by users to train or verify the surgery. This application does not limit the operation scenario.

[0069] The end effector is the tool that connects to the end of the surgical robot's robotic arm. The end effector is the end that is close to the target part of the object.

[0070] In specific implementation, under the target application scenario, when the end effector of the surgical robot's robotic arm approaches the target part of the target object, or when the end effector of the surgical robot's robotic arm moves away from the target part of the target object, the computer device can obtain the real-time position of the end effector and the target part.

[0071] Step S120: Determine the distance between the end effector of the robotic arm and the target part based on the real-time position.

[0072] In practice, the computer equipment can determine the real-time distance between the end effector of the robotic arm and the target part based on their real-time positions.

[0073] Step S130: Control the movement speed of the robotic arm of the surgical robot according to the relationship between the distance and the preset distance threshold.

[0074] The preset distance threshold is a distance threshold set in advance to prevent the end effector of the robotic arm from getting too close to the target part and colliding with it.

[0075] In practice, the computer device can control the current movement speed of the surgical robot based on the real-time distance between the end effector and the target area and the preset distance threshold.

[0076] In the above-mentioned method for switching the movement speed of the robotic arm, the real-time positions of the end effector of the surgical robot and the target part of the target object are obtained; the distance between the end effector and the target part is determined based on the real-time position; and the movement speed of the surgical robot is controlled and switched based on the relationship between the distance and a preset distance threshold.

[0077] Thus, based on the real-time position between the robotic arm's end effector and the target area, the real-time distance between them can be determined. By comparing this real-time distance with a preset distance threshold set to prevent the robotic arm from colliding with the target area due to excessive proximity, the timing for switching the robotic arm's movement speed from fast to slow can be more accurately determined as the end effector approaches the target area. This prevents collisions with the target area due to excessive speed, reducing the occurrence of unexpected collisions. Furthermore, the timing for switching the robotic arm's movement speed from slow to fast can be more accurately determined as the end effector moves away from the target area, allowing the surgical robot to move at a faster speed and improving its operational efficiency. Therefore, by using this solution to determine the timing for controlling and switching the robotic arm's movement speed based on the relationship between the distance between the end effector and the target area and a preset distance threshold, the switching timing for controlling and switching the robotic arm's movement speed can be accurately determined.

[0078] In one embodiment, controlling the movement speed of the robotic arm of the surgical robot according to the relationship between the distance and a preset distance threshold includes: controlling the movement speed of the robotic arm to a first movement speed when the distance is greater than the preset distance threshold; and controlling the movement speed of the robotic arm to a second movement speed when the distance is less than the preset distance threshold.

[0079] The second speed is less than the first speed.

[0080] The first speed of motion can be named the speed of motion in the fast motion phase; the second speed of motion can be named the speed of motion in the slow motion phase.

[0081] In practice, as the surgical robot's end effector moves closer to or further away from the target area, the user manually drags the robot arm to move it freely. The computer controls the robot arm's movement speed based on the relationship between the distance and a preset distance threshold. Specifically, if the distance between the end effector and the target area is greater than the preset distance threshold, the computer controls the robot arm's movement speed to a first speed (fast); if the distance is less than or equal to the preset distance threshold, the computer controls the robot arm's movement speed to a second speed (slow).

[0082] In some embodiments, when the end-effector of the robotic arm is around the target area (i.e., the distance between it and the target area is less than a preset distance threshold), and during the process of the end-effector moving away from the target area at a second speed, if the real-time distance between the end-effector and the target area is greater than the preset distance threshold, the computer device can determine that this is the moment for the robotic arm's movement speed to switch from slow to fast. The computer device can then control the robotic arm's movement speed to switch from the second speed to the first speed, allowing the surgical robot's robotic arm to move at a faster speed, thereby improving the robotic arm's operational efficiency. When the robot arm moves away from the target area (i.e., the distance between the robot arm and the target area is greater than a preset distance threshold), if the real-time distance between the robot arm and the target area is less than the preset distance threshold during the process of the robotic arm end-effector approaching the target area at a first speed, the computer device can determine that this is the moment when the robot arm's movement speed should switch from fast to slow. The computer device can then control the robot arm's movement speed to switch from the first speed to the second speed, so that the surgical robot's arm moves at a slower speed, preventing the robot arm from colliding with the target area due to excessive movement speed when it is near the target area, thus effectively improving safety.

[0083] The technical solution of this embodiment controls the robotic arm to move at a first speed when the distance is greater than a preset distance threshold, and at a second speed when the distance is less than the preset distance threshold. Thus, by controlling the robotic arm's movement speed to switch between different speeds based on the relationship between the distance between the robotic arm's end effector and the target area and the preset distance threshold, the user can avoid manually switching the speed, simplifying the human-computer interaction process and improving operational efficiency.

[0084] In one embodiment, before obtaining the real-time position of the surgical robot's end effector and the target part of the target object, the method further includes: obtaining the robot type corresponding to the surgical robot; and identifying the end effector and the target part based on the robot type.

[0085] In practice, before the computer device obtains the real-time position of the surgical robot's end effector and the target part of the target object, since different types of surgical robots are used in different target application scenarios, the computer device can obtain the robot type corresponding to the surgical robot and identify the end effector and the target part based on the robot type corresponding to the surgical robot.

[0086] Users can input the type of the surgical robot being operated into the computer device's software for switching the robotic arm's movement speed, allowing the computer device to directly retrieve the corresponding robot type. Furthermore, the computer device can also determine the robot type based on the system configuration of the switching software.

[0087] The technical solution of this embodiment obtains the robot type corresponding to the surgical robot; based on the robot type, it identifies the end effector and target area of ​​the robotic arm. Thus, since different types of surgical robots are used in different target application scenarios, identifying the end effector and target area based on the robot type can effectively improve the accuracy of identification.

[0088] In one embodiment, identifying the end effector and target part of the robotic arm according to the robot type includes: acquiring a target recognition model; the target recognition model is trained based on sample depth image data; and identifying the end effector and target part of the robotic arm according to the target recognition model and the robot type.

[0089] In practice, when the computer device identifies the end effector and target part of the robotic arm according to the robot type, the computer device can acquire the target recognition model trained based on the sample depth image data. Based on the target recognition model and the robot type, the computer device can identify the end effector and target part of the robotic arm in the depth image acquired in real time, including the end effector and target part.

[0090] The technical solution of this embodiment involves acquiring a target recognition model, which is trained based on sample depth image data. Based on the target recognition model and the robot type, the end effector and target parts of the robotic arm are identified. Thus, by using the target recognition model trained on sample depth image data to identify the end effector and target parts based on the robot type, the accuracy of the identification can be further improved.

[0091] In one embodiment, such as Figure 2 As shown, obtaining the target recognition model includes the following steps:

[0092] Step S210: Obtain sample depth image data.

[0093] In practice, the computer device can acquire sample depth image data. This sample depth image data can be pre-stored in the computer device's database or downloaded from the internet.

[0094] Step S220: Label the target sample parts and the robotic arm end effector in the sample depth image data to obtain labeled sample depth image data.

[0095] The target sample region is the region in the sample depth image data that matches the target region.

[0096] In practice, users can annotate the target sample parts and the robotic arm end effector in the sample depth image data. Specifically, users can segment the target sample parts and the robotic arm end effector in the sample depth image data and determine the corresponding labels so that the computer device can obtain the annotated sample depth image data.

[0097] Step S230: Based on the sample depth image data and the labeled sample depth image data, perform model training to obtain the target recognition model.

[0098] In practice, computer equipment can train a model based on sample depth image data and labeled sample depth image data to obtain a target recognition model, and then store the target recognition model offline.

[0099] It should be noted that the method for establishing the target recognition model described in steps S210 to S230 above can also be executed by other computer devices so that the computer device controlling the movement speed of the robotic arm of the surgical robot can directly obtain the target recognition model.

[0100] The technical solution of this embodiment involves acquiring sample depth image data; annotating the target sample parts and the robotic arm end effector in the sample depth image data to obtain annotated sample depth image data; and training a model based on the sample depth image data and the annotated sample depth image data to obtain a target recognition model. Thus, the target recognition model trained using the aforementioned sample depth image data and annotated sample depth image data can more accurately identify the robotic arm end effector and target parts in depth images, including the robotic arm end effector and the target parts.

[0101] In one embodiment, the robot type includes a first type matched with a neurosurgical robot and a second type matched with a spinal surgery robot or a hip replacement surgery robot; acquiring sample depth image data includes: for the first type, acquiring first sample depth image data in a bone screw contact registration scenario; for the second type, acquiring second sample depth image data in a C-arm imaging positioning scenario; and obtaining sample depth image data based on the first and second sample depth image data.

[0102] The target application scenarios may include bone screw contact registration as the first application scenario and C-arm imaging positioning as the second application scenario.

[0103] Among them, neurosurgical robots are used in bone screw contact registration scenarios, while spinal surgery robots or hip replacement surgery robots are used in C-arm imaging positioning scenarios.

[0104] Among them, C-arms can include, but are not limited to, small C-arms, medium C-arms, etc.

[0105] In specific implementation, the acquisition of sample depth image data can be achieved by: acquiring depth image data from a depth camera in a bone screw contact registration scenario for a neurosurgical robot of the first type, as the first sample depth image data; and acquiring depth image data from a depth camera in a C-arm imaging positioning scenario for a spinal surgery robot or hip replacement surgery robot of the second type, as the second sample depth image data. Thus, based on the first and second sample depth image data, the sample depth image data can be obtained.

[0106] In the neurosurgical robot bone screw contact registration scenario, the target sample site and target site can be the skull, and the end effector of the neurosurgical robot's robotic arm can be the bone screw registration probe; in the spinal surgery robot or hip replacement surgery robot C-arm imaging positioning scenario, the target sample site and target site can be a site with a tracking array, and the end effector of the spinal surgery robot or hip replacement surgery robot can be a calibration target.

[0107] During the process of acquiring sample depth image data, the depth camera can acquire depth image data offline from multiple angles. The depth image data includes 3D point cloud data, which contains tool 3D point cloud data corresponding to the robotic arm end effector and 3D point cloud data corresponding to the target sample part.

[0108] In the process of offline acquisition of depth image data from multiple angles, an offline handheld depth camera can be used to acquire 3D point cloud data from various angles (for example, no less than 6 angles, or other numbers, no specific limit is made here). The data acquired from each angle can be greater than or equal to 100 sets (or other numbers, no specific limit is made here), and the angles should be spread out as much as possible.

[0109] In some embodiments, during the annotation of target sample areas and robotic arm end effectors in sample depth image data, the computer device can obtain annotated 3D point cloud data based on the 3D point cloud data and corresponding labels of the robotic arm end effector segmented from the 3D point cloud data, as well as the 3D point cloud data and corresponding labels of the target sample areas segmented from the 3D point cloud data. Specifically, for the bone screw contact registration scenario of a neurosurgical robot, the 3D point cloud data corresponding to the bone screw registration probe and the 3D point cloud data corresponding to the head can be segmented from the acquired 3D point cloud data and labeled accordingly to obtain annotated 3D point cloud data for the bone screw contact registration scenario. For example, the label of the 3D point cloud data corresponding to the bone screw registration probe is 0, and the label of the 3D point cloud data corresponding to the head is 1.

[0110] For C-arm imaging positioning scenarios in spinal surgery robots or hip replacement surgery robots, the computer equipment can obtain labeled 3D point cloud data for the C-arm imaging positioning scenario based on the 3D point cloud data and labels corresponding to the calibration target segmented from the acquired 3D point cloud data, as well as the 3D point cloud data and labels corresponding to the parts with tracking arrays. For example, the label for the 3D point cloud data corresponding to the calibration target is 0, and the label for the 3D point cloud data corresponding to the parts with tracking arrays is 1.

[0111] In this way, labeled sample depth image data can be obtained from labeled 3D point cloud data in different application scenarios.

[0112] Among these methods, third-party 3D point cloud data annotation tools can be used, such as point-cloud-annotation-tool.

[0113] Therefore, in the process of training a model based on sample depth image data and labeled sample depth image data to obtain a target recognition model, the computer device can also train a model using a point cloud segmentation network (e.g., PointNet network) based on 3D point cloud data acquired by a depth camera and labeled 3D point cloud data to obtain a target recognition model that can be used for point cloud segmentation. A schematic diagram of the PointNet network is shown below. Figure 3As shown, 3D point cloud data collected by a depth camera is input into the PointNet training network, which can output segmentation results for the end effector of the robotic arm and the target sample. The model is trained by comparing the segmentation results with the labeled 3D point cloud data.

[0114] The technical solution of this embodiment includes a first type of robot matched with a neurosurgical robot and a second type matched with a spinal surgery robot or a hip replacement surgery robot. For the first type, first sample depth image data is acquired in a bone screw contact registration scenario; for the second type, second sample depth image data is acquired in a C-arm imaging positioning scenario. Sample depth image data is obtained based on the first and second sample depth image data. Thus, in different application scenarios corresponding to different robot types, corresponding sample depth image data is collected to train a target recognition model. Therefore, based on the target recognition model and the robot type, the end effector of the robotic arm and the target part can be more accurately identified.

[0115] In one embodiment, obtaining the real-time position of the end effector of the surgical robot and the target part of the target object includes: acquiring depth images including the end effector and the target part in real time; matching the depth images with a target recognition model to identify the tool image data corresponding to the end effector and the part image data corresponding to the target part in real time; and determining the real-time position of the end effector and the target part based on the tool image data and the part image data.

[0116] The target recognition model may include a first target recognition model that matches the bone screw contact registration scenario and a second target recognition model that matches the C-arm X-ray positioning scenario.

[0117] The first target recognition model was trained using 3D point cloud data collected in a bone screw contact registration scenario, as well as labeled 3D point cloud data. The second target recognition model was trained using 3D point cloud data collected in a C-arm X-ray positioning scenario, as well as labeled 3D point cloud data.

[0118] Among them, the first target recognition model is matched with the neurosurgical robot of the first type, and is used to identify the end tool and target part of the robotic arm in the neurosurgical robot bone nail contact registration scenario.

[0119] The second target recognition model is matched with the spinal surgery robot or hip replacement surgery robot of the second type, and is used to identify the end effector and target part of the robotic arm in the C-arm imaging positioning scenario of the spinal surgery robot or hip replacement surgery robot.

[0120] In practice, when the surgical robot's end-effector approaches or moves away from the target area of ​​the target object in the target application scenario, the computer device can acquire depth images in real time, including the end-effector and the target area, captured by a depth camera. By matching the depth images with the target recognition model, the robot type corresponding to the surgical robot can be identified. Based on the offline-trained target recognition model corresponding to the robot type, the tool image data corresponding to the end-effector and the area image data corresponding to the target area can be identified in real time using the depth images.

[0121] For example, if the robot type is identified as type 1, a first target recognition model is used to identify the tool image data corresponding to the end effector of the robotic arm and the part image data corresponding to the target part in real time using depth images; if the robot type is identified as type 2, a second target recognition model is used to identify the tool image data corresponding to the end effector of the robotic arm and the part image data corresponding to the target part in real time using depth images.

[0122] Then, the computer equipment can determine the real-time position of the end effector tool and target part based on the identified tool image data and part image data.

[0123] The technical solution of this embodiment acquires depth images, including the end effector of the robotic arm and the target area, in real time; matches the depth images with a target recognition model to identify the tool image data corresponding to the end effector and the part image data corresponding to the target area in real time; and determines the real-time position of the end effector and the target area based on the tool image data and the part image data. Thus, the real-time position of the end effector and the target area can be determined more accurately based on the three-dimensional tool image data and part image data.

[0124] In one embodiment, determining the distance between the end effector of the robotic arm and the target part based on the real-time position includes: determining the center point or bounding box of the tool image data as a first center point or first bounding box; determining the center point or bounding box of the part image data as a second center point or second bounding box; determining the distance between the first center point and the second center point, or determining the distance between the first bounding box and the second bounding box, to obtain the distance between the end effector of the robotic arm and the target part.

[0125] The computer equipment can obtain real-time 3D point cloud data from depth images, including the robotic arm's end effector and the target area. This real-time 3D point cloud data is then used to match the depth images with a target recognition model to identify the tool image data corresponding to the robotic arm's end effector and the target area image data in real time. Specifically, the computer equipment can match the real-time 3D point cloud data with the target recognition model, identifying the tool 3D point cloud data and the target area 3D point cloud data within the real-time 3D point cloud data. The tool 3D point cloud data is then used as the tool image data, and the target area 3D point cloud data is used as the target area image data.

[0126] In practice, when determining the distance between the end effector of the robotic arm and the target part based on the real-time position, the computer device can determine the center point or bounding box of the tool's 3D point cloud data as the first center point or bounding box; determine the center point or bounding box of the part's 3D point cloud data as the second center point or bounding box; and determine the distance between the first center point and the second center point, or the distance between the first bounding box and the second bounding box, as the distance between the end effector of the robotic arm and the target part.

[0127] The technical solution of this embodiment determines the distance between the end effector of the robotic arm and the target part by determining the center point or bounding box of the tool image data as the first center point or bounding box; determining the center point or bounding box of the part image data as the second center point or bounding box; and determining the distance between the first center point and the second center point, or the distance between the first bounding box and the second bounding box. Thus, by determining the center points or bounding boxes of the three-dimensional tool image data and part image data, the distance between the end effector of the robotic arm and the target part can be accurately determined.

[0128] In some embodiments, the preset distance threshold may include a first distance threshold matching a bone screw contact registration scenario as a first application scenario, and a second distance threshold matching a C-arm radiography positioning scenario as a second application scenario. The distance threshold can be configured in the form of a configuration file.

[0129] During model training, after segmenting the 3D point cloud data corresponding to the robotic arm end effector and the target sample part from the collected 3D point cloud data, the computer device can calculate the distance between the robotic arm end effector and the target sample part after the robotic arm is dragged into place (i.e., the robotic arm's movement speed has switched from fast to slow), thereby setting a preset distance threshold.

[0130] For example, in the case of bone screw contact registration, a first distance threshold is set based on the statistically calculated distance between the bone screw registration probe and the skull (e.g., 5 cm, or other values, without specific limitations). In the case of C-arm imaging positioning, a distance range is determined as the corresponding second distance threshold based on the statistically calculated distance between the calibration target and the part with the tracking array.

[0131] In the C-arm imaging positioning scenario, the robotic arm with a calibration target mounted at its end effector needs to enter the area with a tracking array and the probe plate of the C-arm. During the acquisition of labeled 3D point cloud data for the C-arm imaging positioning scenario, the 3D point cloud data corresponding to the probe plate can be segmented. By labeling the 3D point cloud data corresponding to the calibration target as 0 and the 3D point cloud data corresponding to the area with the tracking array as 1, the label of the 3D point cloud data corresponding to the probe plate is set to 2, resulting in labeled 3D point cloud data for the C-arm imaging positioning scenario. Furthermore, in addition to calculating the distance between the calibration target and the area with the tracking array, the distance between the calibration target and the probe plate can also be calculated to set a distance threshold for the C-arm imaging positioning scenario. Thus, the computer equipment can use the second target recognition model corresponding to the C-arm imaging positioning scenario to identify the distance between the robotic arm end effector and the target area, as well as the distance between the robotic arm end effector and the probe plate. The smaller distance is compared with the second distance threshold corresponding to the C-arm imaging positioning scenario to control the switching of the robotic arm's movement speed.

[0132] In this process, the depth camera acquires depth image data offline from multiple angles. This depth image data can be data captured by the depth camera when the robotic arm is rapidly dragged towards the target sample area (as the robotic arm's movement speed is about to switch to a slower speed). Specifically, as the robotic arm's end effector approaches the target sample area, depth image data is acquired within a preset range before and after the surgical robot switches its movement speed. For example, data is acquired within a preset time range before the surgical robot's robotic arm transitions from a fast movement phase to a slow movement phase, and data is acquired within a preset time range after the surgical robot's robotic arm enters the slow movement phase. Another example is acquiring data as the surgical robot's robotic arm moves from a preset position at the speed corresponding to the fast movement phase until it reaches the target position or stops at the speed corresponding to the slow movement phase. Alternatively, data is acquired within a preset time range as the surgical robot's robotic arm moves from a preset position at the speed corresponding to the fast movement phase until it moves at the speed corresponding to the slow movement phase.

[0133] In another embodiment, such as Figure 4As shown, a method for switching the movement speed of a robotic arm is provided. Taking the application of this method to a computer device as an example, the method includes the following steps:

[0134] Step S402: Real-time acquisition of depth images including the robotic arm end effector and the target area.

[0135] Step S404: Obtain the target recognition model, match the depth image with the target recognition model, and identify the tool image data corresponding to the end effector of the robotic arm and the part image data corresponding to the target part in real time.

[0136] Step S406: Determine the center point or bounding box of the tool image data as the first center point or first bounding box.

[0137] Step S408: Determine the center point or bounding box of the part image data as the second center point or second bounding box.

[0138] Step S410: Determine the distance between the first center point and the second center point, or determine the distance between the first bounding box and the second bounding box, to obtain the distance between the end effector of the robotic arm and the target part.

[0139] Step S412: When the distance is greater than a preset distance threshold, control the movement speed of the robotic arm to the first movement speed.

[0140] Step S414: When the distance is less than a preset distance threshold, control the movement speed of the robotic arm to the second movement speed.

[0141] It should be noted that the specific limitations of the above steps can be found in the specific limitations of a method for switching the movement speed of a robotic arm described above.

[0142] In one embodiment, a system for switching the movement speed of a robotic arm is provided, characterized in that the system includes: a depth camera, a surgical robot, and a controller; the depth camera is used to acquire depth images including the end effector of the surgical robot's robotic arm and the target part of the target object; the depth images are used to obtain the real-time positions of the end effector and the target part; the controller is used to determine the distance between the end effector and the target part based on the real-time positions; the controller is also used to control and switch the movement speed of the surgical robot's robotic arm based on the relationship between the distance and a preset distance threshold.

[0143] The controller is also used to execute the steps that implement the above-described methods.

[0144] In one embodiment, for ease of understanding by those skilled in the art, Figure 5 Another method for switching the movement speed of a robotic arm is provided, applicable to a robotic arm movement speed switching system. For example... Figure 5 As shown:

[0145] Neurosurgical robots:

[0146] 1.1 Training Phase: Depth cameras offline collect data from various angles during the bone screw contact registration process, capturing the first sample depth image data as the robotic arm rapidly drags near the head (before the robotic arm's movement speed switches to slow). The collected data is organized and categorized (primarily categorizing and labeling the robotic arm's end effector and the head in each dataset), and then supervised learning model training is performed. This yields the first target recognition model for identifying the robotic arm's end effector and the head of the neurosurgical robot.

[0147] 1.2. Application Phase: Upon initiation of the bone screw contact registration, the system background is activated, and a depth camera captures real-time images, collecting data (depth images) of the user manually dragging the robotic arm's end effector towards the target object's head. Using the first target recognition model pre-trained offline in step 1.1, the collected data is segmented and recognized between the robotic arm's end effector and the head. The distance between the robotic arm's end effector and the head is then calculated. This calculated distance is then compared with a first distance threshold set by the system (e.g., ...). Figure 5 The flowchart compares the 5cm threshold. When the speed is greater than the threshold, the robot arm's movement speed is automatically set to fast, and when it is less than or equal to the threshold, the robot arm's movement speed is automatically set to slow.

[0148] Spinal surgery robot / hip replacement surgery robot:

[0149] 2.1. The depth camera offline collects data (second sample depth image data) of the target part of the target object, the calibration target held by the end of the robotic arm, and the small C detection plate under various angles in the case of frontal and lateral positioning of the small C-shaped film. The collected data is sorted and classified (mainly the calibration target, the part with the tracking array and the detection plate in each data is marked), and supervised training is performed to obtain the corresponding second target recognition model.

[0150] 2.2. During the positioning process of the small C X-ray, after the robotic arm is dragged into place, the distance range between the target and the small C detection plate and the part is calculated to determine the second distance threshold.

[0151] 2.3. During the initial positioning process, the system background is activated, and the depth camera takes real-time photos, collecting data (depth images) of the user manually dragging the robotic arm's end-effector, the calibration target, towards the target area with the tracking array. Using the second target recognition model trained offline in step 2.1, the collected data is used to identify the calibration target and the target area, and then the distance between the calibration target and the target area is calculated. It is then determined whether the calculated distance is within the range specified in step 2.2. If it is within the range, the robotic arm's movement speed is automatically set to slow, and the user is prompted to release the robotic arm movement control pedal to stop the robotic arm's movement.

[0152] 2.4 The user moves the small C device over and places the small C detection tablet at its location.

[0153] 2.5 In step 2.3, the movement speed of the robotic arm has been adjusted to a slow speed. The user can step on the pedal to drag the end of the robotic arm to fine-tune the pose of the calibration target, so that the parallelism between the calibration target and the small C detection plate meets the preset conditions, so that the calibration target and the small C detection plate are roughly parallel.

[0154] This system utilizes existing hardware cameras, avoiding the introduction of new hardware or custom-designed robotic arms (such as custom-designed robotic arms with speed switching buttons added to the end of the robotic arm), which can effectively reduce hardware costs.

[0155] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0156] Based on the same inventive concept, this application also provides a robotic arm speed switching device for implementing the above-described robotic arm speed switching method. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of the one or more robotic arm speed switching device embodiments provided below can be found in the above-described limitations of the robotic arm speed switching method, and will not be repeated here.

[0157] In one embodiment, such as Figure 6As shown, a device for switching the movement speed of a robotic arm is provided, comprising: an acquisition module 610, a determination module 620, and a control module 630, wherein:

[0158] The acquisition module 610 is used to acquire the real-time position of the end effector of the surgical robot and the target part of the target object.

[0159] The determining module 620 is used to determine the distance between the end effector of the robotic arm and the target part based on the real-time position.

[0160] The control module 630 is used to control and switch the movement speed of the robotic arm of the surgical robot according to the relationship between the distance and the preset distance threshold.

[0161] In one embodiment, the control module 630 is specifically configured to control the movement speed of the robotic arm to a first movement speed when the distance is greater than the preset distance threshold; and to control the movement speed of the robotic arm to a second movement speed when the distance is less than the preset distance threshold; wherein the second movement speed is less than the first movement speed.

[0162] In one embodiment, the acquisition module 610 is further configured to acquire the robot type corresponding to the surgical robot; and identify the end effector of the robotic arm and the target part based on the robot type.

[0163] In one embodiment, the acquisition module 610 is further specifically used to acquire a target recognition model; the target recognition model is trained based on sample depth image data; and the robot type is used to identify the end effector of the robotic arm and the target part.

[0164] In one embodiment, the acquisition module 610 is further specifically used to acquire the sample depth image data; to annotate the target sample parts and the robotic arm end tool in the sample depth image data to obtain annotated sample depth image data; and to perform model training based on the sample depth image data and the annotated sample depth image data to obtain the target recognition model.

[0165] In one embodiment, the robot type includes a first type matched with a neurosurgical robot and a second type matched with a spinal surgery robot or a hip replacement surgery robot; the acquisition module 610 is further specifically configured to acquire first sample depth image data in a bone screw contact registration scenario for the first type; acquire second sample depth image data in a C-arm imaging positioning scenario for the second type; and obtain the sample depth image data based on the first sample depth image data and the second sample depth image data.

[0166] In one embodiment, the acquisition module 610 is specifically used to acquire depth images including the robotic arm end-effector and the target part in real time; match the depth images with the target recognition model to identify the tool image data corresponding to the robotic arm end-effector and the part image data corresponding to the target part in real time; and determine the real-time position of the robotic arm end-effector and the target part based on the tool image data and the part image data.

[0167] In one embodiment, the determining module 620 is specifically used to determine the center point or bounding box of the tool image data as a first center point or first bounding box; determine the center point or bounding box of the part image data as a second center point or second bounding box; determine the distance between the first center point and the second center point, or determine the distance between the first bounding box and the second bounding box, to obtain the distance between the robotic arm end tool and the target part.

[0168] Each module in the aforementioned robotic arm speed switching device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0169] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a method for switching the movement speed of a robotic arm. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0170] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0171] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0172] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0173] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0174] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0175] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0176] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0177] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for switching the movement speed of a robotic arm, characterized in that, The method includes: Obtain the robot type corresponding to the surgical robot; Based on the robot type, identify the end effector of the surgical robot and the target part of the target object; including: acquiring a target recognition model; the target recognition model is trained based on sample depth image data; and identifying the end effector of the surgical robot and the target part based on the target recognition model and the robot type. To obtain the real-time position of the surgical robot's end effector and the target part of the target object; Based on the real-time position, determine the distance between the robotic arm end effector and the target area; Based on the relationship between the distance and a preset distance threshold, the movement speed of the surgical robot's robotic arm is controlled to prevent the robotic arm from colliding with the target area.

2. The method according to claim 1, characterized in that, The step of controlling the switching of the robotic arm's movement speed based on the relationship between the distance and a preset distance threshold includes: If the distance is greater than the preset distance threshold, the movement speed of the robotic arm is controlled to a first movement speed; When the distance is less than the preset distance threshold, the movement speed of the robotic arm is controlled to a second movement speed; the second movement speed is less than the first movement speed.

3. The method according to claim 1, characterized in that, The acquisition of the target recognition model includes: Acquire the sample depth image data; The target sample area and the robotic arm end effector in the sample depth image data are labeled to obtain labeled sample depth image data; The target recognition model is obtained by training the model based on the sample depth image data and the labeled sample depth image data.

4. The method according to claim 3, characterized in that, The robot type includes a first type matched with neurosurgical robots and a second type matched with spinal surgery robots or hip replacement surgery robots; acquiring the sample depth image data includes: For the first type, in the bone nail contact registration scenario, the depth image data of the first sample is obtained; For the second type, in the C-arm imaging positioning scenario, acquire the second sample depth image data; The sample depth image data is obtained based on the first sample depth image data and the second sample depth image data.

5. The method according to claim 1, characterized in that, The process of acquiring the real-time position of the surgical robot's end effector and the target part of the target object includes: Real-time acquisition of depth images including the robotic arm end effector and the target area; The depth image is matched with the target recognition model to identify the tool image data corresponding to the end effector of the robotic arm and the part image data corresponding to the target part in real time. Based on the tool image data and the part image data, the real-time positions of the robotic arm end effector and the target part are determined.

6. The method according to claim 5, characterized in that, Determining the distance between the robotic arm end effector and the target location based on the real-time position includes: Determine the center point or bounding box of the tool image data as the first center point or first bounding box; Determine the center point or bounding box of the image data of the aforementioned region as the second center point or second bounding box; Determine the distance between the first center point and the second center point, or determine the distance between the first bounding box and the second bounding box, to obtain the distance between the end effector of the robotic arm and the target part.

7. A device for switching the movement speed of a robotic arm, characterized in that, The device includes: The acquisition module is used to obtain the robot type corresponding to the surgical robot; The acquisition module is also used to identify the end effector of the surgical robot and the target part of the target object according to the robot type; The acquisition module is further specifically used to acquire a target recognition model; the target recognition model is trained based on sample depth image data; and based on the target recognition model and the robot type, the end effector of the robotic arm and the target part are identified. The acquisition module is also used to acquire the real-time position of the surgical robot's end effector and the target part of the target object; The determination module is used to determine the distance between the end effector of the robotic arm and the target part based on the real-time position; The control module is used to control and switch the movement speed of the surgical robot's robotic arm according to the relationship between the distance and a preset distance threshold, so as to prevent the robotic arm of the surgical robot from colliding with the target area.

8. A system for switching the movement speed of a robotic arm, characterized in that, The system includes: a depth camera, a surgical robot, and a controller; The depth camera is used to acquire depth images including the end effector of the surgical robot and the target part of the target object; the depth images are used by the controller to identify the end effector and the target part according to the target recognition model and the robot type corresponding to the surgical robot, and to obtain the real-time position of the end effector and the target part; the target recognition model is trained based on sample depth image data; The controller is used to determine the distance between the end effector of the robotic arm and the target part based on the real-time position; The controller is also used to control the switching speed of the surgical robot's robotic arm based on the relationship between the distance and a preset distance threshold, so as to prevent the robotic arm of the surgical robot from colliding with the target area.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

Images