Mechanical arm movement speed switching method, device and system and computer equipment

Abstract

The invention relates to a mechanical arm movement speed switching method, device and system, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring real-time positions of a mechanical arm tail end tool of the surgical robot and a target part of a target object; according to the real-time position, the distance between the mechanical arm tail end tool and the target part is determined; according to the size relation between the distance and a preset distance threshold value, the mechanical arm movement speed of the surgical robot is controlled and switched. By adopting the method, the switching opportunity for controlling and switching the movement speed of the mechanical arm can be accurately determined.

Description

Technical Field

[0001] The present application relates to the technical field, and in particular to a method, device, system, computer equipment, storage medium and computer program product for switching the motion speed of a robot arm. Background Art

[0002] With the advancement of science and technology, robotics technology has gradually matured and been widely used. Robots usually include robotic arms. Faced with complex task requirements and unknown working environments, how to ensure the safety of the robotic arm and objects in the environment is one of the key research issues.

[0003] During the process of neurosurgery robot bone screw contact registration, spinal surgery robot and hip replacement surgery robot Xiao C filming and positioning, it is usually necessary to switch the robot arm movement speed to prevent the robot arm from moving too fast and causing collision. However, in related technologies, it is often impossible to accurately determine the timing of switching the robot arm movement speed, resulting in an inappropriate robot arm movement speed, which may lead to unexpected collisions.

[0004] Therefore, there is a problem in the related art of inaccurate judgment of the switching timing of the robot arm movement speed. Summary of the invention

[0005] Based on this, it is necessary to provide a method, device, computer equipment, computer-readable storage medium and computer program product for switching the movement speed of the robot arm, which can more accurately determine the switching timing of the movement speed of the robot arm, in order to address the above technical problems.

[0006] In a first aspect, the present application provides a method for switching the movement speed of a robot arm. The method comprises:

[0007] Acquire the real-time position of the end tool of the surgical robot's robotic arm and the target part of the target object;

[0008] Determining the distance between the end tool of the robot arm and the target part according to the real-time position;

[0009] According to the relationship between the distance and a preset distance threshold, the movement speed of the robotic arm of the surgical robot is controlled and switched.

[0010] In one embodiment, controlling and switching the movement speed of the robot arm of the surgical robot according to the size relationship between the distance and a preset distance threshold includes:

[0011] When the distance is greater than the preset distance threshold, controlling the movement speed of the robotic arm to be a first movement speed;

[0012] When the distance is less than the preset distance threshold, the movement speed of the robotic arm is controlled to be a second movement speed; and the second movement speed is less than the first movement speed.

[0013] In one embodiment, before the step of obtaining the real-time position of the end tool of the surgical robot's mechanical arm and the target part of the target object, the method further includes:

[0014] Obtaining a robot type corresponding to the surgical robot;

[0015] According to the robot type, the end-of-arm tool and the target part are identified.

[0016] In one embodiment, the identifying the robot end tool and the target part according to the robot type includes:

[0017] Obtaining a target recognition model; the target recognition model is trained based on sample depth image data;

[0018] The robot arm end tool and the target part are identified according to the target recognition model and the robot type.

[0019] In one embodiment, obtaining the target recognition model includes:

[0020] Acquire the sample depth image data;

[0021] Annotating the target sample part and the end tool of the robot arm in the sample depth image data to obtain annotated sample depth image data;

[0022] Model training is performed based on the sample depth image data and the labeled sample depth image data to obtain the target recognition model.

[0023] In one embodiment, the robot type includes a first type matching a neurosurgery robot and a second type matching a spinal surgery robot or a hip replacement surgery robot; and acquiring the sample depth image data includes:

[0024] For the first type, in a bone screw contact registration scenario, obtaining first sample depth image data;

[0025] For the second type, in a C-arm radiography positioning scenario, second sample depth image data is acquired;

[0026] The sample depth image data is obtained according to the first sample depth image data and the second sample depth image data.

[0027] In one embodiment, the step of obtaining the real-time position of the end tool of the surgical robot's mechanical arm and the target part of the target object includes:

[0028] Acquire a depth image including the tool at the end of the robot arm and the target part in real time;

[0029] Matching the depth image with the target recognition model to identify in real time the tool image data corresponding to the tool at the end of the robot arm and the part image data corresponding to the target part;

[0030] The real-time positions of the tool at the end of the robot arm and the target part are determined according to the tool image data and the part image data.

[0031] In one embodiment, determining the distance between the end tool of the robot arm and the target part according to the real-time position includes:

[0032] Determine a center point or a bounding box of the tool image data as a first center point or a first bounding box;

[0033] Determine a center point or a bounding box of the part image data as a second center point or a second bounding box;

[0034] The distance between the first center point and the second center point is determined, or the distance between the first bounding box and the second bounding box is determined, to obtain the distance between the end tool of the robot arm and the target part.

[0035] In a second aspect, the present application also provides a device for switching the movement speed of a robot arm. The device comprises:

[0036] An acquisition module, used to acquire the real-time position of the end tool of the surgical robot's mechanical arm and the target part of the target object;

[0037] A determination module, used to determine the distance between the end tool of the robot arm and the target part according to the real-time position;

[0038] The control module is used to control the switching of the movement speed of the mechanical arm of the surgical robot according to the size relationship between the distance and a preset distance threshold.

[0039] In a third aspect, the present application also provides a system for switching the movement speed of a robotic arm. The system includes: a depth camera, a surgical robot, and a controller;

[0040] The depth camera is used to collect a depth image including the end tool of the robotic arm of the surgical robot and the target part of the target object; the depth image is used to obtain the real-time position of the end tool of the robotic arm and the target part;

[0041] The controller is used to determine the distance between the end tool of the robot arm and the target part according to the real-time position;

[0042] The controller is also used to control switching of the movement speed of the robotic arm of the surgical robot according to the size relationship between the distance and a preset distance threshold.

[0043] In a fourth aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0044] Acquire the real-time position of the end tool of the surgical robot's robotic arm and the target part of the target object;

[0045] Determining the distance between the end tool of the robot arm and the target part according to the real-time position;

[0046] According to the relationship between the distance and a preset distance threshold, the movement speed of the robotic arm of the surgical robot is controlled and switched.

[0047] In a fifth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0048] Acquire the real-time position of the end tool of the surgical robot's robotic arm and the target part of the target object;

[0049] Determining the distance between the end tool of the robot arm and the target part according to the real-time position;

[0050] According to the relationship between the distance and a preset distance threshold, the movement speed of the robotic arm of the surgical robot is controlled and switched.

[0051] In a sixth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0052] Acquire the real-time position of the end tool of the surgical robot's robotic arm and the target part of the target object;

[0053] Determining the distance between the end tool of the robot arm and the target part according to the real-time position;

[0054] According to the relationship between the distance and a preset distance threshold, the movement speed of the robotic arm of the surgical robot is controlled and switched.

[0055] The above-mentioned method, device, system, computer equipment, storage medium and computer program product for switching the movement speed of the robotic arm obtain the real-time position of the robotic arm end tool of the surgical robot and the target part of the target object; determine the distance between the robotic arm end tool and the target part based on the real-time position; and control the switching of the movement speed of the robotic arm of the surgical robot based on the relationship between the distance and a preset distance threshold.

[0056] In this way, according to the real-time position between the end tool of the robot arm and the target part, the real-time distance between the end tool of the robot arm and the target part can be determined. By comparing the real-time distance with the preset distance threshold set to prevent the robot arm from being too close to the target part and causing a collision with the target part, based on the size relationship between the distance and the preset distance threshold, the switching time of the robot arm movement speed from fast to slow can be more accurately determined in the process of the end tool of the robot arm approaching the target part of the target object, preventing the robot arm from colliding with the target part due to excessive movement speed when it is near the target part, reducing the occurrence of unexpected collision problems; it can also be more accurately determined when the end tool of the robot arm is away from the target part of the target object. The switching time of the robot arm movement speed is switched from slow to fast, so that the surgical robot moves at a faster speed and improves the operating efficiency of the robot arm. Therefore, through the relationship between the distance between the end tool of the robot arm and the target part and the preset distance threshold, the switching time of controlling the switching of the robot arm movement speed can be accurately determined. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic flow chart of a method for switching the motion speed of a robot arm in one embodiment;

[0058] Figure 2 A schematic diagram of a flow chart of steps for obtaining a target recognition model in one embodiment;

[0059] Figure 3 A schematic diagram of the construction of a training network in one embodiment;

[0060] Figure 4 is a schematic flow chart of a method for switching the movement speed of a robot arm in another embodiment;

[0061] Figure 5 A schematic flow chart of another method for switching the motion speed of a robot arm in one embodiment;

[0062] Figure 6 is a structural block diagram of a switching device for a robot arm movement speed in one embodiment;

[0063] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0065] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0066] In one embodiment, Figure 1 As shown, a method for switching the movement speed of a robot arm is provided. This embodiment takes the method applied to a computer device as an example. It can be understood that the computer device can be a terminal or a system including a terminal and a server. The method is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0067] Step S110, obtaining the real-time positions of the tool at the end of the robotic arm of the surgical robot and the target part of the target object.

[0068] Among them, the target object is not limited to the patient, but can also be a model prosthesis, etc., which can be used by users for training or verification of surgery, etc. This application does not limit the operation scenario.

[0069] The end tool of the robotic arm is a tool connected to the end of the robotic arm of the surgical robot. The end of the robotic arm is the end close to the target part of the target object.

[0070] In a specific implementation, in a target application scenario, when the end tool of the surgical robot's robotic arm approaches the target part of the target object, or when the end tool of the surgical robot's robotic arm moves away from the target part of the target object, the computer device can obtain the real-time position of the end tool of the robotic arm and the target part.

[0071] Step S120, determining the distance between the tool at the end of the robot arm and the target part according to the real-time position.

[0072] In a specific implementation, the computer device can determine the real-time distance between the tool at the end of the robot arm and the target part according to the real-time positions of the tool at the end of the robot arm and the target part.

[0073] Step S130, controlling the movement speed of the robotic arm of the surgical robot according to the relationship between the distance and a preset distance threshold.

[0074] The preset distance threshold is a distance threshold set in advance to prevent the end tool of the robot arm from being too close to the target part, thereby causing a collision with the target part.

[0075] In a specific implementation, the computer device can control the switching of the current robotic arm movement speed of the surgical robot based on the relationship between the real-time distance between the tool at the end of the robotic arm and the target part and a preset distance threshold.

[0076] In the above-mentioned method for switching the movement speed of the robotic arm, the real-time positions of the robotic arm end tool of the surgical robot and the target part of the target object are obtained; based on the real-time positions, the distance between the robotic arm end tool and the target part is determined; and based on the relationship between the distance and a preset distance threshold, the movement speed of the robotic arm of the surgical robot is controlled and switched.

[0077] In this way, according to the real-time position between the end tool of the robot arm and the target part, the real-time distance between the end tool of the robot arm and the target part can be determined. By comparing the real-time distance with the preset distance threshold set to prevent the robot arm from being too close to the target part and causing a collision with the target part, based on the size relationship between the distance and the preset distance threshold, the switching time of the robot arm movement speed from fast to slow can be more accurately determined in the process of the end tool of the robot arm approaching the target part of the target object, preventing the robot arm from colliding with the target part due to excessive movement speed when it is near the target part, reducing the occurrence of unexpected collision problems; it can also be more accurately determined when the end tool of the robot arm is away from the target part of the target object. The switching time of the robot arm movement speed is switched from slow to fast, so that the surgical robot moves at a faster speed and improves the operating efficiency of the robot arm. Therefore, through the relationship between the distance between the end tool of the robot arm and the target part and the preset distance threshold, the switching time of controlling the switching of the robot arm movement speed can be accurately determined.

[0078] In one embodiment, the movement speed of the robotic arm of the surgical robot is controlled and switched according to the relationship between the distance and a preset distance threshold, including: when the distance is greater than the preset distance threshold, the movement speed of the robotic arm is controlled to be a first movement speed; when the distance is less than the preset distance threshold, the movement speed of the robotic arm is controlled to be a second movement speed.

[0079] The second movement speed is lower than the first movement speed.

[0080] Among them, the first movement speed can be named as the movement speed of the fast movement stage; the second movement speed can be named as the movement speed of the slow movement stage.

[0081] In a specific implementation, when the end tool of the surgical robot's robotic arm approaches or moves away from the target part of the target object, the user manually drags the robotic arm to move freely. In this way, when the computer device controls the switching of the surgical robot's robotic arm movement speed according to the relationship between the distance and a preset distance threshold, the computer device can control the robotic arm movement speed to be a first movement speed (fast) when the distance between the end tool of the robotic arm and the target part is greater than the preset distance threshold; and control the robotic arm movement speed to be a second movement speed (slow) when the distance between the end tool of the robotic arm and the target part is less than or equal to the preset distance threshold.

[0082] In some embodiments, when the end tool of the robotic arm is around the target part (i.e., when the distance between the end tool of the robotic arm and the target part is less than a preset distance threshold), while the end tool of the robotic arm is moving away from the target part of the target object at a second movement speed, when the real-time distance between the end tool of the robotic arm and the target part is greater than the preset distance threshold, the computer device can determine that this is the time for the robotic arm movement speed to switch from a slow speed to a fast speed, and the computer device can control the robotic arm movement speed to switch from the second movement speed to the first movement speed, so that the robotic arm of the surgical robot moves at a faster movement speed, thereby improving the operating efficiency of the robotic arm; when the end tool of the robotic arm is moving away from the target part of the target object at a second movement speed, the computer device can control the robotic arm movement speed to switch from the second movement speed to the first movement speed, so that the robotic arm of the surgical robot moves at a faster movement speed, thereby improving the operating efficiency of the robotic arm; When far away from the target part (i.e., when the distance between the target part and the target part is greater than a preset distance threshold), in the process of the end tool of the robotic arm approaching the target part of the target object at a first movement speed, when the real-time distance between the end tool of the robotic arm and the target part is less than the preset distance threshold, the computer device can determine that this is the time to switch the movement speed of the robotic arm from fast to slow. The computer device can control the movement speed of the robotic arm to switch from the first movement speed to the second movement speed, so that the robotic arm of the surgical robot moves at a slower movement speed, thereby preventing the robotic arm from colliding with the target part due to excessive movement speed when near the target part, thereby effectively improving safety.

[0083] The technical solution of this embodiment controls the movement speed of the robot arm to be the first movement speed when the distance is greater than the preset distance threshold; and controls the movement speed of the robot arm to be the second movement speed when the distance is less than the preset distance threshold. In this way, the movement speed of the robot arm is controlled to switch to different movement speeds based on the relationship between the distance between the tool at the end of the robot arm and the target part and the preset distance threshold, thereby avoiding the need for the user to manually switch the movement speed, simplifying the human-computer interaction steps, and improving the operation efficiency.

[0084] In one embodiment, before the step of obtaining the real-time position of the robotic arm end tool of the surgical robot and the target part of the target object, the method further includes: obtaining the robot type corresponding to the surgical robot; and identifying the robotic arm end tool and the target part according to the robot type.

[0085] In a specific implementation, before a computer device obtains the real-time position of the surgical robot's end-arm tool and the target part of the target object, since surgical robots of different robot types are used in different target application scenarios, the computer device can obtain the robot type corresponding to the surgical robot, and identify the end-arm tool and the target part according to the robot type corresponding to the surgical robot.

[0086] The user can input the robot type corresponding to the currently operated surgical robot in the switching software of the robot arm movement speed of the computer device, so that the computer device can directly obtain the robot type corresponding to the surgical robot. In addition, the computer device can also determine the robot type according to the system configuration of the switching software.

[0087] The technical solution of this embodiment obtains the robot type corresponding to the surgical robot; and identifies the end tool and target part of the robot arm according to the robot type. In this way, since different types of surgical robots are used in different target application scenarios, the end tool and target part of the robot arm can be identified according to the robot type, which can effectively improve the recognition accuracy.

[0088] In one embodiment, according to the robot type, the end tool of the robot arm and the target part are identified, including: obtaining a target recognition model; the target recognition model is trained according to sample depth image data; according to the target recognition model and the robot type, the end tool of the robot arm and the target part are identified.

[0089] In a specific implementation, when a computer device identifies a tool at the end of a robot arm and a target part according to the type of robot, the computer device can obtain a target recognition model trained based on sample depth image data, and based on the target recognition model and the type of robot, identify the tool at the end of a robot arm and the target part in the depth image including the tool at the end of a robot arm and the target part obtained in real time.

[0090] The technical solution of this embodiment is to obtain a target recognition model; the target recognition model is trained according to the sample depth image data; and the end tool of the robot arm and the target part are identified according to the target recognition model and the robot type. In this way, on the basis of identifying the end tool of the robot arm and the target part according to the robot type, the target recognition model trained according to the sample depth image data is used for recognition, which can further improve the recognition accuracy.

[0091] In one embodiment, Figure 2 As shown, obtaining a target recognition model includes the following steps:

[0092] Step S210, obtaining sample depth image data.

[0093] In a specific implementation, the computer device may obtain sample depth image data. The sample depth image data may be pre-stored in a database of the computer device or may be obtained by downloading from the Internet.

[0094] Step S220 , annotating the target sample part and the end tool of the robot arm in the sample depth image data to obtain the annotated sample depth image data.

[0095] The target sample part is a part in the sample depth image data that matches the target part.

[0096] In a specific implementation, the user can annotate the target sample part and the end tool of the robotic arm in the sample depth image data. Specifically, the user can segment the target sample part and the end tool of the robotic arm in the sample depth image data and determine the corresponding labels so that the computer device can obtain the annotated sample depth image data.

[0097] Step S230, performing model training based on the sample depth image data and the labeled sample depth image data to obtain a target recognition model.

[0098] In a specific implementation, the computer device may perform model training based on the sample depth image data and the labeled sample depth image data to obtain a target recognition model, and store the target recognition model offline.

[0099] It should be noted that the method for establishing the target recognition model described in the above steps S210 to S230 can also be executed by other computer devices, so that the computer device that controls the movement speed of the robotic arm of the surgical robot can directly obtain the target recognition model.

[0100] The technical solution of this embodiment obtains sample depth image data; annotates the target sample parts and the end tool of the robot arm in the sample depth image data to obtain the annotated sample depth image data; and performs model training based on the sample depth image data and the annotated sample depth image data to obtain a target recognition model. In this way, the target recognition model obtained by training with the sample depth image data and the annotated sample depth image data can more accurately identify the end tool of the robot arm and the target part in the depth image including the end tool of the robot arm and the target part.

[0101] In one embodiment, the robot type includes a first type that matches a neurosurgery robot, and a second type that matches a spinal surgery robot or a hip replacement surgery robot; obtaining sample depth image data includes: for the first type, in a bone screw contact registration scenario, obtaining first sample depth image data; for the second type, in a C-arm film positioning scenario, obtaining second sample depth image data; obtaining sample depth image data based on the first sample depth image data and the second sample depth image data.

[0102] Among them, the target application scenarios may include a bone screw contact registration scenario as a first application scenario, and a C-arm radiography positioning scenario as a second application scenario.

[0103] Among them, neurosurgery robots are used in bone screw contact registration scenarios, and spinal surgery robots or hip replacement surgery robots are used in C-arm filming and positioning scenarios.

[0104] The C-arm may include but is not limited to a small C-arm, a medium C-arm, and the like.

[0105] In a specific implementation, for the collection of sample depth image data, the depth image data collected by the depth camera can be obtained as the first sample depth image data in the neurosurgery robot bone screw contact registration scene for the neurosurgery robot corresponding to the first type; and the depth image data collected by the depth camera can be obtained as the second sample depth image data in the spinal surgery robot or hip replacement surgery robot C-arm filming positioning scene for the spinal surgery robot or hip replacement surgery robot corresponding to the second type. In this way, the sample depth image data can be obtained according to the first sample depth image data and the second sample depth image data.

[0106] Among them, in the neurosurgery robot bone screw contact registration scenario, the target sample part and the target part can be the head, and the neurosurgery robot's robotic arm end tool can be a bone screw registration probe; in the spinal surgery robot or hip replacement surgery robot C-arm filming and positioning scenario, the target sample part and the target part can be a part with a tracking array, and the spinal surgery robot or hip replacement surgery robot's robotic arm end tool can be a calibration target.

[0107] In the process of collecting sample depth image data, the depth camera can collect depth image data offline at multiple angles. The depth image data includes 3D (three-dimensional) point cloud data, which includes tool 3D point cloud data corresponding to the end tool of the robotic arm and 3D point cloud data corresponding to the target sample part.

[0108] In the process of the depth camera collecting depth image data at multiple angles offline, an offline handheld depth camera can be used to collect 3D point cloud data from various angles (for example, no less than 6 angles, or other numbers, not specifically limited here), and the data collected at each angle can be greater than or equal to 100 groups (or other numbers, not specifically limited here), and the angles should be dispersed as much as possible.

[0109] Among them, in some embodiments, in the process of annotating the target sample parts and the end tool of the robot arm in the sample depth image data, the computer device can obtain the annotated 3D point cloud data based on the 3D point cloud data corresponding to the end tool of the robot arm segmented in the 3D point cloud data and the corresponding label, and the 3D point cloud data corresponding to the segmented target sample parts and the corresponding label. Specifically, for the contact registration scenario of bone screws in neurosurgery robots, the 3D point cloud data corresponding to the bone screw registration probe and the 3D point cloud data corresponding to the skull can be segmented from the collected 3D point cloud data, and the corresponding labels can be marked to obtain the annotated 3D point cloud data in the bone screw contact registration scenario. For example, the label of the 3D point cloud data corresponding to the bone screw registration probe is 0, and the 3D point cloud data corresponding to the skull is 1.

[0110] For the C-arm radiography positioning scenario of a spinal surgery robot or a hip replacement surgery robot, the computer device can obtain the labeled 3D point cloud data in the C-arm radiography positioning scenario based on the 3D point cloud data and corresponding labels corresponding to the calibration target segmented from the collected 3D point cloud data, and the 3D point cloud data and corresponding labels corresponding to the part with the tracking array. For example, the label of the 3D point cloud data corresponding to the calibration target is 0, and the label of the 3D point cloud data corresponding to the part with the tracking array is 1.

[0111] In this way, the labeled sample depth image data can be obtained through the labeled 3D point cloud data in different application scenarios.

[0112] Among them, third-party 3D point cloud data annotation tools can be used, such as: point-cloud-annotation-tool.

[0113] Thus, when the computer device performs model training based on the sample depth image data and the labeled sample depth image data to obtain the target recognition model, the computer device can use the point cloud segmentation network (e.g., PointNet network) to perform model training based on the 3D point cloud data collected by the depth camera and the labeled 3D point cloud data to obtain the target recognition model that can be used for point cloud segmentation. The schematic diagram of the construction of the PointNet network is shown in FIG. Figure 3As shown, the 3D point cloud data collected by the depth camera is input into the PointNet training network, which can output the segmentation results of the tool at the end of the robot arm and the target sample part. The model training is performed based on the difference between the segmentation results and the labeled 3D point cloud data.

[0114] In the technical solution of this embodiment, the robot types include a first type that matches a neurosurgery robot, and a second type that matches a spinal surgery robot or a hip replacement surgery robot; for the first type, in a bone screw contact registration scenario, the first sample depth image data is obtained; for the second type, in a C-arm filming positioning scenario, the second sample depth image data is obtained; and based on the first sample depth image data and the second sample depth image data, the sample depth image data is obtained. In this way, in different application scenarios corresponding to different robot types, the corresponding sample depth image data is collected to train a target recognition model, so that the end tool and target part of the robot arm can be more accurately identified based on the target recognition model and the robot type.

[0115] In one embodiment, the real-time position of a tool at the end of a surgical robot's robotic arm and a target part of a target object is obtained, including: obtaining a depth image including the tool at the end of the robotic arm and the target part in real time; matching the depth image with a target recognition model to identify in real time tool image data corresponding to the tool at the end of the robotic arm, and part image data corresponding to the target part; and determining the real-time position of the tool at the end of the robotic arm and the target part based on the tool image data and the part image data.

[0116] Among them, the target recognition model may include a first target recognition model that matches the bone screw contact registration scene, and a second target recognition model that matches the C-arm film positioning scene.

[0117] The first target recognition model is trained by using the 3D point cloud data collected in the bone screw contact registration scenario and the annotated 3D point cloud data. The second target recognition model is trained by using the 3D point cloud data collected in the C-arm radiography positioning scenario and the annotated 3D point cloud data.

[0118] Among them, the first target recognition model is matched with the neurosurgery robot corresponding to the first type, and is used to identify the end tool of the robotic arm and the target part in the neurosurgery robot bone screw contact registration scenario.

[0119] Among them, the second target recognition model is matched with the corresponding second type of spinal surgery robot or hip replacement surgery robot, and is used to identify the end tool and target part of the robotic arm in the C-arm filming and positioning scenario of the spinal surgery robot or hip replacement surgery robot.

[0120] In a specific implementation, in a target application scenario, when the end tool of the surgical robot's robotic arm approaches or moves away from the target part of the target object, the computer device can obtain in real time a depth image including the end tool of the robotic arm and the target part captured by the depth camera; by matching the depth image with the target recognition model, the robot type corresponding to the surgical robot can be identified, and based on the offline trained target recognition model corresponding to the robot type, the depth image can be used to identify in real time the tool image data corresponding to the end tool of the robotic arm and the part image data corresponding to the target part.

[0121] For example, if the robot type is identified as the first type, the first target recognition model is used to use the depth image to identify in real time the tool image data corresponding to the tool at the end of the robot arm and the part image data corresponding to the target part; if the robot type is identified as the second type, the second target recognition model is used to use the depth image to identify in real time the tool image data corresponding to the tool at the end of the robot arm and the part image data corresponding to the target part.

[0122] Then, the computer device can determine the real-time position of the tool and the target part at the end of the robot arm based on the identified tool image data and part image data.

[0123] The technical solution of this embodiment is to obtain a depth image including the tool at the end of the robot arm and the target part in real time; match the depth image with the target recognition model, and recognize the tool image data corresponding to the tool at the end of the robot arm and the part image data corresponding to the target part in real time; and determine the real-time position of the tool at the end of the robot arm and the target part according to the tool image data and the part image data. In this way, the real-time position of the tool at the end of the robot arm and the target part can be determined more accurately according to the three-dimensional tool image data and the part image data.

[0124] In one embodiment, the distance between the tool at the end of the robot arm and the target part is determined based on the real-time position, including: determining the center point or bounding box of the tool image data as the first center point or the first bounding box; determining the center point or bounding box of the part image data as the second center point or the second bounding box; determining the distance between the first center point and the second center point, or determining the distance between the first bounding box and the second bounding box, to obtain the distance between the tool at the end of the robot arm and the target part.

[0125] Among them, the computer device can obtain the real-time acquired 3D point cloud data as real-time 3D point cloud data based on the real-time acquired depth image including the end tool of the robot arm and the target part. In this way, when the computer device matches the depth image with the target recognition model and recognizes the tool image data corresponding to the end tool of the robot arm and the part image data corresponding to the target part in real time, the computer device can match the real-time 3D point cloud data with the target recognition model, and recognize the tool 3D point cloud data corresponding to the end tool of the robot arm and the part 3D point cloud data corresponding to the target part in the real-time 3D point cloud data through the matched target recognition model; then, the tool 3D point cloud data is used as the tool image data, and the part 3D point cloud data is used as the part image data.

[0126] In a specific implementation, when a computer device determines the distance between a tool at the end of the robot arm and a target part based on a real-time position, the computer device can determine a center point or a bounding box of the tool 3D point cloud data as a first center point or a first bounding box; determine a center point or a bounding box of the part 3D point cloud data as a second center point or a second bounding box; determine the distance between the first center point and the second center point, or determine the distance between the first bounding box and the second bounding box as the distance between the tool at the end of the robot arm and the target part.

[0127] The technical solution of this embodiment determines the center point or bounding box of the tool image data as the first center point or the first bounding box; determines the center point or bounding box of the part image data as the second center point or the second bounding box; determines the distance between the first center point and the second center point, or determines the distance between the first bounding box and the second bounding box, to obtain the distance between the tool at the end of the robot arm and the target part. In this way, by determining the center point or bounding box of the three-dimensional tool image data and the part image data, the distance between the tool at the end of the robot arm and the target part can be accurately determined.

[0128] In some embodiments, the preset distance threshold may include a first distance threshold matching a bone screw contact registration scenario as a first application scenario, and a second distance threshold matching a C-arm radiography positioning scenario as a second application scenario. The distance threshold may be configured in the form of a configuration file.

[0129] Among them, during the model training process, after segmenting the 3D point cloud data corresponding to the end tool of the robotic arm and the 3D point cloud data corresponding to the target sample part from the collected 3D point cloud data, the computer device can count the distance between the end tool of the robotic arm and the target sample part after the robotic arm is dragged into place (that is, the movement speed of the robotic arm has switched from fast to slow), thereby setting a preset distance threshold.

[0130] For example, for the bone screw contact registration scenario, the corresponding first distance threshold is set according to the statistical distance between the bone screw registration probe and the skull (for example, it can be 5 cm, or other values, which are not specifically limited here). For the C-arm radiography positioning scenario, the distance range is determined as the corresponding second distance threshold according to the statistical distance between the calibration target and the part with the tracking array.

[0131] Among them, in the C-arm filming positioning scene, the robot arm with a calibration target installed at the end of the machine needs to enter between the part with the tracking array and the detection plate of the C-arm. In the process of obtaining the annotated 3D point cloud data for the C-arm filming positioning scene, the 3D point cloud data corresponding to the detection plate can also be segmented. On the basis of setting the label of the 3D point cloud data corresponding to the calibration target to 0 and the 3D point cloud data corresponding to the part with the tracking array to 1, the label of the 3D point cloud data corresponding to the detection plate is set to 2, and the annotated 3D point cloud data in the C-arm filming positioning scene is obtained. In addition, on the basis of counting the distance between the calibration target and the part with the tracking array, the distance between the calibration target and the detection plate can also be counted to set the distance threshold corresponding to the C-arm filming positioning scene. In this way, the computer device can identify the distance between the tool at the end of the robot arm and the target part, as well as the distance between the tool at the end of the robot arm and the detection plate through the second target recognition model corresponding to the C-arm filming positioning scene, and compare the smaller distance with the second distance threshold corresponding to the C-arm filming positioning scene to control the switching of the robot arm movement speed.

[0132] Among them, in the process of the depth camera collecting depth image data at multiple angles offline, the depth image data can be the data collected by the depth camera when the robot arm is dragged quickly around the target sample part (the movement speed of the robot arm is about to switch to slow speed). Specifically, in the process of the end tool of the robot arm approaching the target sample part, the depth image data collected within the preset range before and after the surgical robot switches the movement speed of the robot arm is collected. For example, the data within the preset time range before the robot arm of the surgical robot enters the slow movement stage from the fast movement stage is collected, and the data within the preset time range after the robot arm of the surgical robot enters the slow movement stage is collected. For another example, the robot arm of the surgical robot can be collected at a preset position at a movement speed corresponding to the fast movement stage until it moves to the target position or stops at a movement speed corresponding to the slow movement stage. Alternatively, the robot arm of the surgical robot is collected at a preset position at a movement speed corresponding to the fast movement stage until it moves at a preset time range at a movement speed corresponding to the slow movement stage.

[0133] In another embodiment, Figure 4As shown, a method for switching the movement speed of a robot arm is provided, and the method is applied to a computer device as an example for explanation, and includes the following steps:

[0134] Step S402, acquiring a depth image including the tool at the end of the robot arm and the target part in real time.

[0135] Step S404, obtaining a target recognition model, matching the depth image with the target recognition model, and identifying in real time the tool image data corresponding to the tool at the end of the robot arm and the part image data corresponding to the target part.

[0136] Step S406: determine a center point or a bounding box of the tool image data as a first center point or a first bounding box.

[0137] Step S408: determine the center point or bounding box of the part image data as the second center point or second bounding box.

[0138] Step S410, determining the distance between the first center point and the second center point, or determining the distance between the first bounding box and the second bounding box, to obtain the distance between the end tool of the robot arm and the target part.

[0139] Step S412: when the distance is greater than a preset distance threshold, control the movement speed of the robot arm to be a first movement speed.

[0140] Step S414, when the distance is less than a preset distance threshold, controlling the movement speed of the robot arm to be a second movement speed.

[0141] It should be noted that the specific limitations of the above steps can refer to the specific limitations of a method for switching the movement speed of a robot arm mentioned above.

[0142] In one embodiment, a system for switching the movement speed of a robotic arm is provided, characterized in that the system includes: a depth camera, a surgical robot and a controller; the depth camera is used to collect depth images including a tool at the end of the robotic arm of the surgical robot and a target part of a target object; the depth image is used to obtain the real-time position of the tool at the end of the robotic arm and the target part; the controller is used to determine the distance between the tool at the end of the robotic arm and the target part based on the real-time position; the controller is also used to control the switching movement speed of the robotic arm of the surgical robot based on the relationship between the distance and a preset distance threshold.

[0143] The controller is also used to execute the steps of implementing the above method.

[0144] In one embodiment, to facilitate understanding by those skilled in the art, Figure 5 Another method for switching the motion speed of a robot arm is provided, which is applied to a switching system for the motion speed of a robot arm. Figure 5 As shown:

[0145] Neurosurgery Robot:

[0146] 1.1. Training phase: The depth camera collects data (first sample depth image data) from all angles offline when the robot arm is dragged quickly near the head during the contact registration of the bone screw (the robot arm movement speed is about to switch to slow speed), and organizes and classifies the collected data (mainly classifying and annotating the end tool of the robot arm and the head in each data), and conducts supervised learning model training. The first target recognition model for identifying the end tool of the robot arm and the head of the neurosurgery robot is obtained.

[0147] 1.2. Application stage: When the bone screw contact registration starts, the system background is turned on, and the depth camera takes real-time photos to collect data (depth image) of the user manually dragging the end of the robot arm to approach the target object's head. The first target recognition model trained offline in advance in step 1.1 is used to segment and recognize the end tool of the robot arm and the head on the collected data, and then the distance between the end tool of the robot arm and the head is calculated. The calculated distance is then compared with the first distance threshold set by the system (for example, Figure 5 When the speed is greater than the threshold, the robot arm movement speed is automatically set to fast; when the speed is less than or equal to the threshold, the robot arm movement speed is automatically set to slow.

[0148] Spine surgery robot / hip replacement surgery robot:

[0149] 2.1. The depth camera collects data of the target part of the target object, the calibration target clamped by the end of the robot arm, and the small C detection plate in the case of small C radiography in the front and side positions from various angles offline (second sample depth image data), and organizes and classifies the collected data (mainly marking the calibration target, the part with the tracking array and the detection plate in each data), and performs supervised training and learning to obtain the corresponding second target recognition model.

[0150] 2.2. During the positioning process of small C for filming, the distance ranges between the mark target and the small C detection plate and the part after the robotic arm is dragged into place are counted to determine the second distance threshold.

[0151] 2.3. When the positioning process begins, the system background is turned on, and the depth camera takes photos in real time. The data (depth image) of the user manually dragging the calibration target at the end of the robotic arm close to the target part with the tracking array is collected. The second target recognition model trained offline in advance in step 2.1 is used to identify the calibration target and the target part of the collected data, and then the distance between the calibrated target and the target part is calculated. Then it is determined whether the calculated distance is within the range in step 2.2. If it is within the range, the robotic arm movement speed is automatically set to slow, and the user is prompted to release the robotic arm movement control pedal to stop the robotic arm from moving.

[0152] 2.4. The user moves the small C device over and places the small C to detect the tablet position.

[0153] 2.5. In step 2.3, the movement speed of the robotic arm has been adjusted to slow. The user can step on the pedal and drag the end of the robotic arm to fine-tune the calibration target posture so that the parallelism between the calibration target and the small C detection plate meets the preset conditions, so that the calibration target and the small C detection plate are roughly parallel.

[0154] This system utilizes existing hardware devices and cameras, avoiding the introduction of new hardware devices or customized design of robotic arms (such as custom-designing a robotic arm and adding a speed switching hardware button at the end of the robotic arm), which can effectively reduce hardware costs.

[0155] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0156] Based on the same inventive concept, the embodiment of the present application also provides a switching device for the robot arm movement speed for implementing the above-mentioned switching method for the robot arm movement speed. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the embodiments of one or more switching devices for the robot arm movement speed provided below can refer to the limitations of a switching method for the robot arm movement speed above, and will not be repeated here.

[0157] In one embodiment, Figure 6As shown, a switching device for the movement speed of a robot arm is provided, comprising: an acquisition module 610, a determination module 620 and a control module 630, wherein:

[0158] The acquisition module 610 is used to acquire the real-time position of the tool at the end of the robotic arm of the surgical robot and the target part of the target object.

[0159] The determination module 620 is used to determine the distance between the robot arm end tool and the target part according to the real-time position.

[0160] The control module 630 is used to control the switching of the movement speed of the robotic arm of the surgical robot according to the relationship between the distance and a preset distance threshold.

[0161] In one embodiment, the control module 630 is specifically used to control the movement speed of the robotic arm to a first movement speed when the distance is greater than the preset distance threshold; and to control the movement speed of the robotic arm to a second movement speed when the distance is less than the preset distance threshold; and the second movement speed is less than the first movement speed.

[0162] In one embodiment, the acquisition module 610 is further used to acquire the robot type corresponding to the surgical robot; and identify the robot arm end tool and the target part according to the robot type.

[0163] In one embodiment, the acquisition module 610 is further specifically used to acquire a target recognition model; the target recognition model is trained based on sample depth image data; and the end tool of the robot arm and the target part are identified based on the target recognition model and the robot type.

[0164] In one embodiment, the acquisition module 610 is further specifically used to acquire the sample depth image data; annotate the target sample part and the robot arm end tool in the sample depth image data to obtain the annotated sample depth image data; perform model training based on the sample depth image data and the annotated sample depth image data to obtain the target recognition model.

[0165] In one embodiment, the robot type includes a first type that matches a neurosurgery robot and a second type that matches a spinal surgery robot or a hip replacement surgery robot; the acquisition module 610 is also specifically used to obtain first sample depth image data for the first type in a bone screw contact registration scenario; and to obtain second sample depth image data for the second type in a C-arm film positioning scenario; and the sample depth image data is obtained based on the first sample depth image data and the second sample depth image data.

[0166] In one embodiment, the acquisition module 610 is specifically used to acquire in real time a depth image including the robot arm end tool and the target part; match the depth image with the target recognition model to identify in real time the tool image data corresponding to the robot arm end tool and the part image data corresponding to the target part; and determine the real-time positions of the robot arm end tool and the target part based on the tool image data and the part image data.

[0167] In one embodiment, the determination module 620 is specifically used to determine the center point or bounding box of the tool image data as the first center point or the first bounding box; determine the center point or the bounding box of the part image data as the second center point or the second bounding box; determine the distance between the first center point and the second center point, or determine the distance between the first bounding box and the second bounding box, to obtain the distance between the tool at the end of the robot arm and the target part.

[0168] Each module in the above-mentioned switching device for the movement speed of the robot arm can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0169] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for switching the movement speed of a robotic arm is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0170] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0171] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0172] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0173] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0174] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0175] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0176] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0177] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for switching the motion speed of a robot arm, characterized in that: The method comprises: Acquire the real-time position of the end tool of the surgical robot's robotic arm and the target part of the target object; Determining the distance between the end tool of the robot arm and the target part according to the real-time position; According to the relationship between the distance and a preset distance threshold, the movement speed of the robotic arm of the surgical robot is controlled and switched.

2. The method according to claim 1, characterized in that The controlling and switching of the movement speed of the mechanical arm of the surgical robot according to the magnitude relationship between the distance and a preset distance threshold comprises: When the distance is greater than the preset distance threshold, controlling the movement speed of the robotic arm to be a first movement speed; When the distance is less than the preset distance threshold, the movement speed of the robotic arm is controlled to be a second movement speed; and the second movement speed is less than the first movement speed.

3. The method according to claim 1, characterized in that Before the step of obtaining the real-time position of the tool at the end of the robotic arm of the surgical robot and the target part of the target object, the method further comprises: Obtaining the robot type corresponding to the surgical robot; According to the robot type, the end-of-arm tool and the target part are identified.

4. The method according to claim 3, characterized in that The step of identifying the robot arm end tool and the target part according to the robot type includes: Obtaining a target recognition model; the target recognition model is trained based on sample depth image data; The robot arm end tool and the target part are identified according to the target recognition model and the robot type.

5. The method according to claim 4, characterized in that The acquiring of the target recognition model comprises: Acquire the sample depth image data; Annotating the target sample part and the end tool of the robot arm in the sample depth image data to obtain annotated sample depth image data; Model training is performed based on the sample depth image data and the labeled sample depth image data to obtain the target recognition model.

6. The method according to claim 5, characterized in that The robot type includes a first type matching a neurosurgery robot and a second type matching a spinal surgery robot or a hip replacement surgery robot; the acquiring of the sample depth image data includes: For the first type, in a bone screw contact registration scenario, obtaining first sample depth image data; For the second type, in a C-arm radiography positioning scenario, second sample depth image data is acquired; The sample depth image data is obtained according to the first sample depth image data and the second sample depth image data.

7. The method according to claim 4, characterized in that The method of obtaining the real-time position of the tool at the end of the robotic arm of the surgical robot and the target part of the target object comprises: Acquire a depth image including the tool at the end of the robot arm and the target part in real time; Matching the depth image with the target recognition model to identify in real time the tool image data corresponding to the tool at the end of the robot arm and the part image data corresponding to the target part; The real-time positions of the tool at the end of the robot arm and the target part are determined according to the tool image data and the part image data.

8. The method according to claim 7, characterized in that Determining the distance between the end tool of the robot arm and the target part according to the real-time position includes: Determine a center point or a bounding box of the tool image data as a first center point or a first bounding box; Determine a center point or a bounding box of the part image data as a second center point or a second bounding box; The distance between the first center point and the second center point is determined, or the distance between the first bounding box and the second bounding box is determined to obtain the distance between the end tool of the robot arm and the target part.

9. A switching device for the movement speed of a robot arm, characterized in that: The device comprises: An acquisition module, used to acquire the real-time position of the end tool of the surgical robot's mechanical arm and the target part of the target object; A determination module, used to determine the distance between the end tool of the robot arm and the target part according to the real-time position; The control module is used to control the switching of the movement speed of the mechanical arm of the surgical robot according to the size relationship between the distance and a preset distance threshold.

10. A switching system for the movement speed of a robot arm, characterized in that: The system includes: a depth camera, a surgical robot and a controller; The depth camera is used to collect a depth image including the end tool of the robotic arm of the surgical robot and the target part of the target object; the depth image is used to obtain the real-time position of the end tool of the robotic arm and the target part; The controller is used to determine the distance between the end tool of the robot arm and the target part according to the real-time position; The controller is also used to control switching of the movement speed of the robotic arm of the surgical robot according to the size relationship between the distance and a preset distance threshold.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images