Control method and device based on double mechanical arms and force feedback, surgical robot and electronic device

Abstract

The application discloses a control method and device based on double mechanical arms and force feedback, a surgical robot and electronic equipment. The method comprises the following steps: after a first mechanical arm drives a perception sensor to enter a target phantom, first perception information is acquired. A first control instruction is acquired, and the first control instruction indicates that a first ablation device is controlled to enter the target phantom. After the second mechanical arm drives the first ablation device to enter the target phantom based on the first control instruction and the first perception information, information of a first force is acquired, the first force being a force borne by the first ablation device after entering the target phantom. A second control instruction is generated based on the size of the first force. The second control instruction is sent to a damping device. The method can enable a user to know the force borne by the first ablation device after entering the target phantom in a timely manner, and thus the control effect can be improved.

Description

Technical Field

[0001] This application relates to the field of medical technology, and in particular to a control method and device based on dual robotic arms and force feedback, a surgical robot, and electronic equipment. Background Technology

[0002] Ablation devices are applied to human simulation models to simulate ablation surgery, which can mimic real clinical ablation procedures, observe the distribution of ablated necrotic areas and tissue thermal response patterns, and enable pre-operative planning and testing of ablation effects. However, current technology does not allow for effective control of the ablation device. Summary of the Invention

[0003] This application provides a control method and device, surgical robot, and electronic device based on dual robotic arms and force feedback. By controlling the damping device in the control device to generate a second force to impede the movement of the control device, the user can be informed in a timely manner of the force situation of the first ablation device after entering the target phantom, thereby improving the ablation effect of controlling the first ablation device.

[0004] In a first aspect, a control method based on dual robotic arms and force feedback is provided, the method comprising: After controlling the first robotic arm to drive the sensing sensor into the target model, the first sensing information is obtained. The first sensing information is the sensing information collected by the sensing sensor inside the target model. A first control command is obtained, which instructs the first ablation device to enter the target phantom, the target phantom including a region to be ablated, the first ablation device being used to ablate the region to be ablated, and the first control command being obtained through a movement control device; After controlling the second robotic arm to drive the first ablation device into the target phantom based on the first control command and the first sensing information, the information of the first force is obtained. The information of the first force includes the magnitude of the first force, which is the force that the first ablation device experiences after entering the target phantom. Based on the magnitude of the first force, the magnitude of the second force is determined. The second force is used to impede the movement of the control device. The magnitude of the second force is positively correlated with the magnitude of the first force. Based on the magnitude of the second force, a second control command is generated, which instructs the damping device in the control device to generate the second force, the damping device being used to generate the force that hinders the movement of the control device; The second control command is sent to the damping device to cause the damping device to generate the second force to impede the movement of the control device.

[0005] In the first aspect, after acquiring the first control command and the first sensing information, the control device controls the first ablation device to enter the target phantom based on the first control command and the first sensing information, which can improve control accuracy. Then, it acquires information about the first force experienced by the first ablation device after entering the target phantom, and based on the first force information, determines the magnitude of the second force used to impede the movement of the control device. The first control command is obtained by moving the control device. Then, based on the magnitude of the second force, a second control command is generated and sent to the damping device in the control device, causing the damping device to generate the second force used to impede the movement of the control device. This allows the user to promptly know the force situation experienced by the first ablation device after entering the target phantom. Therefore, based on dual robotic arms and force feedback, the control effect of the first ablation device can be improved.

[0006] In any embodiment of this application, the first control command instructs the first ablation device to move from outside the target phantom to inside the target phantom based on a first reference path; Before determining the magnitude of the second force based on the magnitude of the first force, the method further includes: Obtain the first reference path; After controlling the second robotic arm to drive the first ablation device into the target phantom based on the first control command and the first sensing information, the position of the first ablation device is obtained; Based on the first reference path and the position of the first ablation device, the depth to which the first ablation device enters the target phantom is obtained; Determining the magnitude of the second force based on the magnitude of the first force includes: The magnitude of the second force is determined based on the magnitude of the first force and the depth, wherein the magnitude of the second force is positively correlated with the depth.

[0007] In conjunction with any embodiment of this application, determining the magnitude of the second force based on the magnitude of the first force and the depth includes: When the depth is less than or equal to the length of the first reference path, the magnitude of the second force is determined based on the magnitude of the first force, the depth, and a first value; wherein the magnitude of the second force is positively correlated with a first product, the first product being the product of the depth and the first value, and the first value being a positive number; When the depth is greater than the length of the first reference path, the magnitude of the second force is determined based on the magnitude of the first force, the depth, and the second value; wherein the magnitude of the second force is positively correlated with a second product, the second product being the product of the depth and the second value, and the second value being greater than the first value.

[0008] In conjunction with any embodiment of this application, before obtaining the first reference path, the method further includes: Information about a third force is obtained, which is the force experienced by the second ablation device after entering the target phantom. The second ablation device is used to ablate the area to be ablated. The second ablation device enters the target phantom before the first ablation device enters the target phantom. Based on the information of the third force and the target model, the target deformation is estimated, wherein the target model is used to estimate the deformation of the target model based on the force acting on the target model; Obtain a second reference path, which is the path taken by the first ablation device from outside the target phantom into the target phantom. The second reference path is planned before the second ablation device enters the target phantom. The process of obtaining the first reference path includes: Based on the target deformation, the second reference path is modified to obtain the first reference path.

[0009] In any embodiment of this application, the first sensing information is collected before the second ablation device enters the target phantom; before estimating the target deformation of the target phantom based on the third force and the target model, the method further includes: Acquire second sensing information, which is sensing information collected by the sensing sensor inside the target phantom, and the second sensing information is collected after the second ablation device enters the target phantom; The estimation of the target deformation variables of the target model based on the information of the third force and the target model includes: If the difference between the first perceived information and the second perceived information is greater than a first threshold, the target deformation of the target model is estimated based on the information of the third force and the target model.

[0010] In conjunction with any embodiment of this application, the step of modifying the second reference path based on the target deformation to obtain the first reference path includes: Based on the second perceived information, a three-dimensional reconstruction is performed on the target phantom to obtain the target's three-dimensional information; Based on the target deformation, the starting position of the second reference path is corrected to obtain the corrected starting position; Based on the target's three-dimensional information, the termination position of the second reference path is corrected to obtain the corrected termination position. The first reference path is obtained based on the corrected start position and the corrected end position.

[0011] Secondly, a control device based on dual robotic arms and force feedback is provided, the control device based on dual robotic arms and force feedback comprising: The acquisition unit is used to acquire first sensing information after controlling the first robotic arm to drive the sensing sensor into the target model. The first sensing information is the sensing information collected by the sensing sensor inside the target model. The acquisition unit is used to acquire a first control command, which instructs the first ablation device to enter the target phantom, the target phantom including a region to be ablated, the first ablation device being used to ablate the region to be ablated, and the first control command being obtained through a movement control device. The acquisition unit is further configured to acquire information about a first force after controlling the second robotic arm to drive the first ablation device into the target phantom based on the first control command and the first sensing information. The information about the first force includes the magnitude of the first force, which is the force experienced by the first ablation device after entering the target phantom. The processing unit is configured to determine the magnitude of a second force based on the magnitude of the first force, the second force being used to impede the movement of the control device, and the magnitude of the second force being positively correlated with the magnitude of the first force. The processing unit is further configured to generate a second control command based on the magnitude of the second force, the second control command instructing the damping device in the control device to generate the second force, the damping device being configured to generate the force that hinders the movement of the control device; The processing unit is further configured to send the second control command to the damping device to cause the damping device to generate the second force for resisting the movement of the control device.

[0012] In any embodiment of this application, the first control command instructs the first ablation device to move from outside the target phantom to inside the target phantom based on a first reference path; The acquisition unit is further configured to acquire the first reference path; The acquisition unit is further configured to acquire the position of the first ablation device after controlling the second robotic arm to drive the first ablation device into the target phantom based on the first control command and the first sensing information. The processing unit is further configured to obtain the depth of the first ablation device entering the target phantom based on the first reference path and the position of the first ablation device; The processing unit is further configured to determine the magnitude of the second force based on the magnitude of the first force and the depth, wherein the magnitude of the second force is positively correlated with the depth.

[0013] In conjunction with any embodiment of this application, the processing unit is used for: When the depth is less than or equal to the length of the first reference path, the magnitude of the second force is determined based on the magnitude of the first force, the depth, and a first value; wherein the magnitude of the second force is positively correlated with a first product, the first product being the product of the depth and the first value, and the first value being a positive number; When the depth is greater than the length of the first reference path, the magnitude of the second force is determined based on the magnitude of the first force, the depth, and the second value; wherein the magnitude of the second force is positively correlated with a second product, the second product being the product of the depth and the second value, and the second value being greater than the first value.

[0014] In any embodiment of this application, the acquisition unit is further configured to acquire information about a third force, which is the force experienced by the second ablation device after entering the target phantom. The second ablation device is configured to ablate the area to be ablated, and the second ablation device enters the target phantom before the first ablation device enters the target phantom. The processing unit is further configured to estimate the target deformation of the target model based on the information of the third force and the target model, wherein the target model is configured to estimate the deformation of the target model based on the force acting on the target model; The acquisition unit is further configured to acquire a second reference path, which is the path by which the first ablation device enters the target phantom from outside the target phantom. The second reference path is planned before the second ablation device enters the target phantom. The acquisition unit is further configured to modify the second reference path based on the target deformation to obtain the first reference path.

[0015] In any embodiment of this application, the first sensing information is collected before the second ablation device enters the target phantom; the acquisition unit is further configured to acquire second sensing information, which is sensing information collected by the sensing sensor inside the target phantom, and the second sensing information is collected after the second ablation device enters the target phantom; The processing unit is further configured to estimate the target deformation based on the information of the third force and the target model when the difference between the first perceived information and the second perceived information is greater than a first threshold.

[0016] In conjunction with any embodiment of this application, the processing unit is used for: Based on the second perceived information, a three-dimensional reconstruction is performed on the target phantom to obtain the target's three-dimensional information; Based on the target deformation, the starting position of the second reference path is corrected to obtain the corrected starting position; Based on the target's three-dimensional information, the termination position of the second reference path is corrected to obtain the corrected termination position. The first reference path is obtained based on the corrected start position and the corrected end position.

[0017] Thirdly, a surgical robot is provided, including a control device based on dual robotic arms and force feedback as described in the second aspect. In this third aspect, the surgical robot can execute a control method based on dual robotic arms and force feedback via the control device, and can improve the control effect by outputting feedback information.

[0018] Fourthly, an electronic device is provided, comprising: a processor and a memory, the memory for storing computer program code, the computer program code including computer instructions, wherein, when the processor executes the computer instructions, the electronic device performs a method as described in the first aspect above and any possible implementation thereof.

[0019] Fifthly, another electronic device is provided, comprising: a processor, a transmitting device, an input device, an output device, and a memory, the memory being used to store computer program code, the computer program code including computer instructions, wherein, when the processor executes the computer instructions, the electronic device performs a method as described in the first aspect above and any possible implementation thereof.

[0020] In a sixth aspect, a computer-readable storage medium is provided, wherein a computer program is stored therein, the computer program including program instructions that, when executed by a processor, cause the processor to perform a method as described in the first aspect above and any possible implementation thereof.

[0021] In a seventh aspect, a computer program product is provided, the computer program product comprising a computer program or instructions, wherein, when the computer program or instructions are executed on a computer, the computer performs the method described in the first aspect and any possible implementation thereof.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments of this application will be described below.

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0025] Figure 1 A flowchart illustrating a control method based on dual robotic arms and force feedback, provided for an embodiment of this application; Figure 2 A schematic diagram of a control device provided in an embodiment of this application; Figure 3 A schematic diagram of a control scenario based on dual robotic arms and force feedback provided for an embodiment of this application; Figure 4 A schematic diagram of the architecture of a control system based on dual robotic arms and force feedback provided for an embodiment of this application; Figure 5 This application provides a schematic diagram of a scene outside an operating room. Figure 6 A schematic diagram of a control device based on dual robotic arms and force feedback provided in this application embodiment; Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0027] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0028] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments. It should be understood that in this application, "at least one" means one or more, "more" means two or more, and "at least two" means two or three or more.

[0029] The execution subject of this application embodiment is a control device based on dual robotic arms and force feedback (hereinafter referred to as the control device). The control device can be any electronic device capable of executing the technical solutions disclosed in the method embodiments of this application. Optionally, the control device can be one of the following: a computer or a server.

[0030] It should be understood that the method embodiments of this application can also be implemented by a processor executing computer program code. The embodiments of this application are described below with reference to the accompanying drawings. Please refer to... Figure 1 , Figure 1 This is a flowchart illustrating a control method based on dual robotic arms and force feedback, provided as an embodiment of this application.

[0031] 101. After the control device controls the first robotic arm to drive the sensing sensor into the target model, it acquires the first sensing information, which is the sensing information collected by the sensing sensor inside the target model.

[0032] In this embodiment, the target phantom is a human body phantom, which is a physical model. Optionally, the material used for the target phantom includes at least one of the following: silicone, resin, gel, and polyurethane. Exemplarily, the target phantom is a gelatin phantom or an agar phantom.

[0033] The sensing sensor is connected to the first robotic arm. Moving the first robotic arm moves the sensing sensor. In step 101, the control device controls the first robotic arm to move the sensing sensor into the target phantom. After entering the target phantom, the sensing sensor collects sensing information within the target phantom, i.e., the first sensing information. Optionally, the target phantom includes an area to be ablated. The control device controls the first robotic arm to move the sensing sensor to a preset area within the target phantom, wherein the distance from the preset area to the area to be ablated is a preset range. For example, the preset range is 2 cm to 5 cm.

[0034] In some solutions, the sensing sensor includes an endoscope, such as a fluoroscope. After the endoscope is inserted into the target phantom, images of the target phantom can be acquired through the endoscope; that is, the first sensing information includes the first image acquired by the endoscope.

[0035] 102. The control device acquires a first control command, which instructs the first ablation device to enter the target phantom. The target phantom includes a region to be ablated. The first ablation device is used to ablate the region to be ablated. The first control command is obtained through a movement control device.

[0036] In this embodiment, the ablation device (including the first ablation device mentioned above and the second ablation device mentioned below) can output energy to the ablation medium to complete the ablation treatment of the ablation medium. For example, the target phantom described above is one type of ablation medium. By using the ablation device to output energy to the target phantom, the ablation treatment of the target phantom can be achieved. In some embodiments, the ablation device consists of an energy generation unit and an energy release unit. The energy generation unit is responsible for generating the energy required for ablation, and the energy release unit is responsible for transmitting and outputting the energy. For example, after placing the ablation device in the ablation medium, the energy generation unit first generates energy, and then the energy is introduced into the ablation medium through the energy release unit implanted in the medium, thereby achieving the ablation treatment of the medium. The ablation device can be of various types, optionally including radiofrequency ablation devices, microwave ablation devices, and cryoablation devices. Radiofrequency ablation devices output radiofrequency electrical energy, microwave ablation devices output microwave electromagnetic energy, and cryoablation devices output cooling energy. For example, the cryoablation device includes a cryoprobe, and the cooling energy output by the cryoprobe includes a high-pressure refrigerant.

[0037] The first control command instructs the control device to control the first ablation device to enter the target phantom, so as to ablate the area to be ablated within the target phantom. In some implementations, the first ablation device is connected to a second robotic arm. The control device can drive the first ablation device into the target phantom by controlling the movement of the second robotic arm. Optionally, the area to be ablated is used to simulate a lesion area in the human body, such as a tumor area.

[0038] The first control command is obtained through a mobile control device. In one possible scenario, the control device is deployed outside the operating room, while the target phantom and the first ablation device are deployed inside the operating room. The user can move the control device outside the operating room to control the first ablation device inside the operating room to enter the target phantom, thereby achieving remote control.

[0039] In one alternative implementation, Figure 2 This is a schematic diagram of a control device provided in an embodiment of this application. Figure 2As shown, the control device includes a joystick, a control slider, and a display device. The joystick is used to control the position and orientation of the first ablation device, the control slider is used to control the displacement of the first ablation device along the axis of the placement hole, and the display device is used to display information related to controlling the first ablation device and to interact with the user.

[0040] The first ablation device is a cryoprobe, which is placed within the placement hole of a guide connected to a second robotic arm. With the cryoprobe placed within the placement hole, its movement path coincides with the axis of the placement hole. The guide also includes a drive unit for moving the cryoprobe within the placement hole along its axis. Optionally, the control device sends drive commands to the drive unit via a transmission control protocol (TCP) to move the first ablation device.

[0041] The path taken by the first ablation device to move from outside the target phantom to inside the target phantom is the first reference path, which is planned before the user moves the control device. After acquiring the first reference path, the control device can control the movement of the second robotic arm based on the first reference path, moving the guide to a reference pose so that the axis of the placement hole coincides with the reference path. Optionally, if the user notices a deviation between the axis of the placement hole and the reference path after the guide has moved to the reference pose, the user can fine-tune the pose of the guide by moving the control lever to control the movement of the second robotic arm, thereby reducing the overlap between the axis of the placement hole and the reference path. Optionally, the rotation angle of the control lever is the same as the rotation angle of the first ablation device. Optionally, the sensing sensor includes an endoscope. After acquiring images collected in real time by the endoscope inside the target phantom, the control device can control the display device to display the real-time images, allowing the user to adjust the pose of the first ablation device based on the real-time images to improve the ablation effect.

[0042] After the axis of the placement hole coincides with the first reference path, the user can control the drive device to move the cryoprobe placed in the placement hole along the axis by moving the control slider, thereby allowing the cryoprobe to enter the target phantom. For example, by moving the control slider to the right, the first ablation device is controlled to continue moving deeper into the target phantom, resulting in a greater insertion depth; or, by moving the control slider to the left, the first ablation device is controlled to continue moving away from the target phantom, resulting in a smaller insertion depth. In this case, the first control command is obtained by moving the control slider.

[0043] Optionally, the control device includes a confirmation button, which confirms the movement of the first ablation device based on a first reference path. After detecting a user clicking the confirmation button, the control device controls the movement of the second robotic arm based on the first reference path. In some implementations, the confirmation button is a physical button, such as a physical button located on a joystick. In other implementations, the confirmation button is a virtual button. The display device can display the pre-planned path and the confirmation button.

[0044] In one optional implementation, the region to be ablated within the target phantom requires ablation using both a first ablation device and a second ablation device. The second ablation device enters the target phantom first, followed by the first ablation device. Before the second ablation device enters the target phantom, at least one computed tomography (CT) image is acquired, where each CT image is obtained by scanning the target phantom using a CT scanner. Three-dimensional reconstruction is performed based on the at least one CT image to obtain a three-dimensional CT image. A second reference path is planned based on the three-dimensional CT image, where the second reference path is the path taken by the first ablation device from outside the target phantom into the target phantom. In other words, the planned path for the first ablation device to enter the target phantom before the second ablation device enters is the second reference path. Optionally, the target phantom includes important tissue models, which include models that are expected to avoid collisions. Optionally, the important tissue model includes at least one of the following: a skeletal model, a vascular model, a tracheal model, and an organ model other than an ablation organ model, wherein the ablation organ model is an organ model that includes the area to be ablated. Optionally, the ablation organ model is a liver model or a kidney model.

[0045] Considering that the second ablation device will deform upon entering the target phantom, thus altering the position of the area to be ablated, if the first ablation device is then controlled to enter the target phantom based on the second reference path, the distance between the first ablation device and the area to be ablated may become too great, resulting in poor ablation efficacy. Therefore, after the second ablation device enters the target phantom, the second reference path can be modified based on the deformation of the target phantom to obtain a first reference path. Then, controlling the first ablation device to enter the target phantom based on the first reference path can improve the ablation efficacy of the first ablation device in the area to be ablated.

[0046] Based on this, before acquiring the first reference path, the control device performs the following steps: acquiring information about a third force, wherein the third force is the force experienced by the second ablation device after entering the target phantom, and the second ablation device enters the target phantom before the first ablation device enters the target phantom. Based on the information about the third force and the target model, the target deformation of the target phantom is estimated, and the target model is used to estimate the deformation of the target phantom based on the force experienced by the target phantom. The second reference path is acquired. After acquiring the target deformation and the control device, the control device corrects the second reference path based on the target deformation to obtain the first reference path.

[0047] The target deformation variables of the target phantom include the deformation variables of all regions within the target phantom. For example, if the target phantom includes a skin region, an organ model, and a blood vessel model, then the target deformation variables include the deformation variables of the skin region, the deformation variables of the organ model, and the deformation variables of the blood vessel model.

[0048] In some schemes, the target model is a neural network trained as follows: Training force information is input into the neural network to obtain estimated deformation variables, where the training force is the force acting on the training phantom, and the estimated deformation variables are the deformation variables of the training phantom. The training phantom can be any phantom; optionally, the training phantom is the same as the target phantom. Optionally, the training force is the interaction force between the second ablation device and the training phantom after the second ablation device enters the training phantom. The actual deformation variables of the training phantom are obtained. Based on the difference between the estimated deformation variables and the actual deformation variables, the loss of the neural network is obtained. Based on the loss of the neural network, the parameters of the neural network are adjusted until the loss of the neural network converges. In other schemes, the target model is a finite element model.

[0049] The following describes several implementation methods for modifying the second reference path based on the target deformation: Implementation Method 1 involves determining the deformation variables at the starting and ending positions of the second reference path based on the target deformation variables. The starting position of the second reference path is located within the skin region of the target phantom, and the ending position is located within the region to be ablated. Based on the deformation variables at the starting and ending positions of the second reference path, the deformed starting position is obtained. Similarly, based on the deformation variables at the ending positions of the second reference path, the deformed ending position is obtained. Based on the deformed starting and ending positions, a first reference path is obtained, where the starting position of the first reference path is the deformed starting position, and the ending position of the first reference path is the deformed ending position. The control device, based on Implementation Method 1, corrects the second reference path, which improves correction efficiency.

[0050] Implementation Method 2: A dangerous location is determined from the second reference path, where the dangerous location is the closest position to the important organization model within the second reference path. For example, the second reference path includes location 1 and location 2, where the minimum distance between location 1 and the important organization model is d1, and the minimum distance between location 2 and the important organization model is d2. If d1 is less than d2, then location 1 is a dangerous location. Based on the target deformation, the deformation of the dangerous location at the end of the second reference path is determined. Based on the deformation of the end of the second reference path and the end of the second reference path, the deformed end position is obtained. The control device obtains the deformed dangerous location based on the deformation of the dangerous location and the dangerous location itself. If the deformed dangerous location is outside the area corresponding to the important organization model, a first reference path is obtained based on the deformed dangerous location and the deformed end position, where the end position of the first reference path is the deformed end position, and the first reference path passes through the deformed dangerous location. If the deformed danger location is within the area corresponding to the important organizational model, the deformed danger location is adjusted based on the minimum distance from the danger location to the important organizational model to obtain an adjusted danger location, ensuring that the adjusted danger location is outside the area corresponding to the important organizational model. Based on the adjusted danger location and the termination location after deformation, a first reference path is obtained, wherein the termination location of the first reference path is the termination location after deformation, and the first reference path passes through the adjusted danger location.

[0051] In implementation method 2, since the dangerous location is the closest to the important organizational model, if the deformed dangerous location is outside the area corresponding to the important organizational model, then the probability that the deformed locations of other locations are also outside the area corresponding to the important organizational model is relatively high. Therefore, when the deformed dangerous location is outside the area corresponding to the important organizational model, the control device obtains a first reference path based on the deformed dangerous location and the deformed termination location, which can reduce the probability that the first reference path passes through the important organizational model. When the deformed dangerous location is within the area corresponding to the important organizational model, the deformed dangerous location is adjusted to obtain an adjusted dangerous location. Then, based on the adjusted dangerous location and the deformed termination location, the first reference path is obtained, which can also reduce the probability that the first reference path passes through the important organizational model.

[0052] The information about the third force includes its magnitude and direction. In one optional embodiment, the second ablation device includes a first force sensor for detecting the force acting on the second ablation device. In some embodiments, the second ablation device obtains the information about the third force based on the data detected by the first force sensor and sends this information to a control device, which in turn receives the information about the third force from the second ablation device. In other embodiments, the second ablation device sends the data detected by the first force sensor to the control device after it has already received the data. The control device then obtains the information about the third force based on the data detected by the first force sensor.

[0053] In one optional implementation, the first sensing information is acquired before the second ablation device enters the target phantom. Before executing the step "estimate the target deformation of the target phantom based on the third force and the target model," the control device further performs the following steps: acquiring second sensing information, wherein the second sensing information is sensing information acquired by the sensing sensors within the target phantom, and the second sensing information is acquired after the second ablation device enters the target phantom. After obtaining the first and second sensing information, the control device performs the following steps during the execution of the step "estimate the target deformation of the target phantom based on the third force and the target model": if the difference between the first and second sensing information is greater than a first threshold, estimate the target deformation of the target phantom based on the third force and the target model.

[0054] Because the first sensing information is collected before the second ablation device enters the target phantom, and the second sensing information is collected after the second ablation device enters the target, the difference between the first and second sensing information is caused by the second ablation device entering the target phantom. Furthermore, the difference between the first and second sensing information can characterize the magnitude of the deformation caused by the second ablation device entering the target phantom. Therefore, before estimating the target deformation of the target phantom, the control device can first estimate whether the deformation of the target phantom is large or small based on the difference between the first and second sensing information. If the deformation of the target phantom is small, then there is no need to correct the second reference path, and the second reference path can be directly used as the first reference path. If the deformation of the target phantom is large, then it is necessary to estimate the target deformation of the target phantom, and then the second reference path can be corrected based on the target deformation. Therefore, when the difference between the first and second sensing information is greater than the first threshold, the control device can estimate the target deformation of the target phantom based on the third force and the target model, which can reduce the amount of data processing caused by correction and improve correction efficiency.

[0055] In some solutions, the sensing sensor includes an endoscope, such as a fluoroscope. The second sensing information includes a second image acquired by the endoscope.

[0056] In one optional implementation, the control device performs the following steps during the execution of step "correcting the second reference path based on the target deformation to obtain the first reference path": Based on the second perceived information, perform three-dimensional reconstruction of the target phantom to obtain the target's three-dimensional information. Based on the target deformation, correct the starting position of the second reference path to obtain a corrected starting position. Based on the target's three-dimensional information, correct the ending position of the second reference path to obtain a corrected ending position. Based on the corrected starting position and the corrected ending position, obtain the first reference path.

[0057] In this embodiment, the second sensing information can be used for three-dimensional reconstruction of the target phantom. In some embodiments, the sensing sensor includes a binocular vision sensor, and the second sensing information includes binocular images acquired by the binocular vision sensor. In other embodiments, the sensing sensor includes a structured light sensor, and the second sensing information includes a three-dimensional point cloud acquired by the structured light sensor.

[0058] On the one hand, because the target's 3D information within the target phantom is richer than its deformation information, correcting the termination position of the second reference path based on the target's 3D information is more accurate than correcting it based on the target's deformation. For example, if the second ablation device enters the target phantom, causing anomalies in the organ model, such as large protrusions, it's difficult to estimate the anomaly based on the third force information. However, the target's 3D information can accurately determine the anomaly. On the other hand, the target's 3D information does not include information outside the target phantom, such as the skin region. The starting position of the second reference path is located within the skin region. Therefore, the starting position of the second reference path can be corrected based on the target's deformation.

[0059] Based on this, the control device corrects the start and end positions of the second reference path according to this embodiment, and obtains the first reference path based on the corrected start and end positions, which can improve the accuracy of the first reference path.

[0060] Optionally, after obtaining the first reference path by correcting the second reference path, the control device sends an instruction message to the operating device. This instruction message instructs the operating device to output a prompt message indicating that the second reference path has been corrected to the first reference path. Thus, after the operating device outputs the prompt message, the user is aware that the second reference path has been corrected to the first reference path.

[0061] 103. After the control device controls the second robotic arm to drive the first ablation device into the target phantom based on the first control command and the first sensing information, it acquires the information of the first force, wherein the first force is the force that the first ablation device receives after entering the target phantom.

[0062] After acquiring the first control command, the control device can control the movement of the second robotic arm based on the first control command and the first sensing information, thereby enabling the second robotic arm to carry the first ablation device into the target phantom. Controlling the movement of the second robotic arm based on the first control command responds to user input to the control device. Controlling the movement of the second robotic arm based on the first sensing information allows for the movement of the first ablation device based on sensing information collected by sensors within the target phantom, improving the control effect on the first ablation device and thus enhancing its ablation effect. For example, the first sensing information includes an image within the target phantom; fine-tuning the pose of the robotic arm based on this image allows for fine-tuning of the pose of the first ablation device.

[0063] After entering the target phantom, the first ablation device will encounter resistance from the tissue model within the target phantom. For example, the target phantom includes a skeletal model. If the first ablation device collides with the skeletal model after entering the target phantom, an interaction force will be generated between the first ablation device and the skeletal model; in this case, the first force includes this interaction force.

[0064] In some schemes, the information of the first force includes the magnitude and direction of the first force, wherein the direction of the first force can be represented based on a pre-planned path. Optionally, the direction from the starting position to the ending position of the pre-planned path is called the target direction, and the direction of the first force can be represented based on the angle between the first direction and the target direction. For example, the direction of the first force is 30 degrees, that is, the angle between the direction of the first force and the target direction is 30 degrees.

[0065] In one alternative implementation, the first ablation device includes a second force sensor for detecting the force applied to the first ablation device. In some embodiments, the first ablation device obtains information about a first force based on the data detected by the second force sensor and sends this information to a control device, which in turn receives the first force information from the first ablation device. In other embodiments, the first ablation device sends the data detected by the second force sensor to the control device after it has already received the data. The control device then obtains the first force information based on the data detected by the second force sensor.

[0066] 104. The control device determines the magnitude of the second force based on the magnitude of the first force, wherein the second force is used to impede the movement of the control device, and the magnitude of the second force is positively correlated with the magnitude of the first force.

[0067] 105. The control device generates a second control command based on the magnitude of the second force, wherein the second control command instructs the damping device in the control device to generate a second force, and the damping device is used to generate a force that hinders the movement of the control device.

[0068] 106. The control device sends a second control command to the damping device to cause the damping device to generate a second force to impede the movement of the control device.

[0069] In this embodiment, the second force is a feedback information, which indicates the force experienced by the first ablation device after entering the target phantom. The control device executes steps 104 to 106, controlling the damping device in the manipulation device to output a second force that hinders the movement of the manipulation device, thereby outputting the feedback information. This allows the user to receive tactile feedback when moving the manipulation device, making the user's perception of the force experienced by the first ablation device more intuitive. After learning about the force experienced by the first ablation device after entering the target phantom, the user can adjust the entry depth and / or pose of the first ablation device in a timely manner based on this force, reducing the probability of collision between the first ablation device and important tissue models within the target phantom.

[0070] In steps 104 to 106, in order to output feedback information via tactile feedback, the control device first determines the magnitude of a second force based on the magnitude of a first force. Then, based on the magnitude of the second force, it controls the damping device to output a second force acting on the control device to impede the movement of the control device. For example, the damping device includes a damping motor, which outputs a force to impede the movement of the control device.

[0071] In some schemes, the first control command is generated by the user by moving the control slider of the control device, and the second force is used to resist the movement of the control slider.

[0072] In some embodiments, the control device outputs feedback information by controlling a display device to display the feedback information; for example, the control device sends a display command to the display device of the operating device to cause the display device to display the feedback information. In other embodiments, the control device outputs feedback information by controlling a speaker to output voice containing the feedback information. In still other embodiments, outputting feedback information includes controlling the operating device to vibrate, so as to provide feedback on the magnitude of the force applied to the first ablation device through the vibration amplitude.

[0073] exist Figure 1In the method shown, after acquiring a first control command and first sensing information, the control device controls the first ablation device to enter the target phantom based on the first control command and first sensing information, which can improve control accuracy. Then, it acquires information about the first force experienced by the first ablation device after entering the target phantom, and determines the magnitude of a second force to hinder the movement of the control device based on this information. The first control command is obtained by moving the control device. Based on the magnitude of the second force, a second control command is generated and sent to a damping device in the control device, causing the damping device to generate a second force to hinder the movement of the control device. This allows the user to promptly know the force situation experienced by the first ablation device after entering the target phantom. Furthermore, it improves the control effect when remotely controlling the first ablation device to ablate the target phantom.

[0074] In one possible application scenario, based on Figure 1 The control method shown enables remote control of the first ablation device to enter the target phantom, allowing the device to ablate the area to be ablated within the phantom. Therefore, the effectiveness of the remotely controlled ablation can be determined based on the ablation effect on the area to be ablated. Figure 1 The control method shown can be used to simulate ablation surgery based on remote control.

[0075] As an optional implementation, the first control command instructs the first ablation device to move from outside the target phantom to inside the target phantom based on the first reference path. Optionally, after the control device controls the second robotic arm to move based on the first reference path, aligning the axis of the guide's placement hole with the first reference path, the user controls the drive device to propel the first ablation device into the target phantom along the first reference path by moving the control slider of the control device. Upon detecting a sliding operation on the control slider, the control device generates the first control command. The control device then sends the first control command to the control device, at which point the first control command controls the first ablation device to move from outside the target phantom to inside the target phantom based on the first reference path.

[0076] In this embodiment, before executing the step "determine the magnitude of the second force based on the magnitude of the first force," the control device further performs the following steps: obtaining a first reference path. After controlling the second robotic arm to drive the first ablation device into the target phantom based on the first control command and the first sensing information, the position of the first ablation device is obtained. Based on the first reference path and the position of the first ablation device, the depth of the first ablation device entering the target phantom is obtained. After obtaining this depth, the control device performs the following steps during the execution of the step "determine the magnitude of the second force based on the magnitude of the first force": determining the magnitude of the second force based on the magnitude of the first force and the depth, wherein the magnitude of the second force is positively correlated with the depth. In this way, the second force can not only provide feedback on the force received by the first ablation device after entering the target phantom, but also on the depth of the first ablation device entering the target phantom, thereby improving the feedback effect.

[0077] As an optional implementation, the control device performs the following steps during the execution of step "determining the magnitude of the second force based on the magnitude and depth of the first force": When the depth is less than or equal to the length of the first reference path, the magnitude of the second force is determined based on the magnitude of the first force, the depth, and a first value, wherein the magnitude of the second force is positively correlated with a first product, which is the product of depth and the first value, and the first value is a positive number. When the depth is greater than the length of the first reference path, the magnitude of the second force is determined based on the magnitude of the first force, the depth, and a second value, wherein the magnitude of the second force is positively correlated with a second product, which is the product of depth and the second value, and the second value is greater than the first value.

[0078] In this embodiment, the starting position of the first reference path is located within the skin region of the target phantom, and the ending position of the first reference path is located within the region to be ablated. If the depth of the first ablation device entering the target phantom is less than or equal to the length of the first reference path, it indicates that the first ablation device has not yet reached the ending position of the first reference path, and in this case, the first ablation device can continue to penetrate deeper into the target phantom. If the depth of the first ablation device entering the target phantom is greater than the length of the first reference path, it indicates that the first ablation device may have already passed the ending position of the first reference path. Continuing to penetrate deeper into the target phantom in this case not only easily leads to collisions with other tissue models but also easily causes the energy release unit of the first ablation device to move away from the region to be ablated. Therefore, in this embodiment, feedback is provided in different ways for these two situations. Specifically, when the depth of the first ablation device entering the target phantom is less than or equal to the length of the first reference path, the feedback force is relatively smaller; when the depth of the first ablation device entering the target phantom is greater than the length of the first reference path, the feedback force is relatively larger. This allows for an improvement in the feedback effect through a second force.

[0079] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating a control scenario based on dual robotic arms and force feedback, provided as an embodiment of this application. Figure 3 As shown, the scenario includes a target phantom, an optical tracking device, a scanning bed, a first robotic arm, a first display device, a control device for the first robotic arm, a second display device, a control device for the second robotic arm, and a second robotic arm. The target phantom is placed on the scanning bed, which is a CT bed. Optionally, the control device mentioned above includes a control device for the first robotic arm and a control device for the second robotic arm.

[0080] The first robotic arm is connected to a sensing sensor, for example, to an endoscope. By controlling the movement of the first robotic arm, the sensing sensor can be moved into the target phantom. The control device of the first robotic arm can control a first display device to display the sensing information collected by the sensing sensor.

[0081] The control device of the second robotic arm controls the scanning bed to perform a CT scan on the target phantom, obtaining at least one CT image. Based on this CT image, the control device performs 3D reconstruction, obtaining a 3D CT image. Based on the 3D CT image, the control device plans the path for the first ablation device to enter the target phantom, obtaining a planned path. At least six optical markers are attached to the skin region of the target phantom; the positions of these six optical markers in the world coordinate system can be obtained using an optical tracking device. Based on the positions of these six optical markers in both the world and CT image coordinate systems, the control device derives a target transformation relationship, which is a transformation between the image coordinate system and the world coordinate system. Using this target transformation relationship, the control device transforms the planned path, obtaining a second reference path.

[0082] The second robotic arm is connected to a guide, which includes a placement hole in which the first ablation device is placed. By controlling the movement of the second robotic arm, the first ablation device can be driven into the target phantom. The control device of the second robotic arm can control a second display device to display at least one of the following information: a three-dimensional CT image, the depth of the first ablation device entering the target phantom, the position information of the first ablation device, and information on the first force.

[0083] To better understand how control is achieved in this scenario using a dual-manipulator and force feedback control method, please refer to [link / reference needed]. Figure 4 , Figure 4 This is a schematic diagram of the architecture of a control system based on dual robotic arms and force feedback, provided for an embodiment of this application. Figure 4As shown, the operating room includes a control device, which performs the following functions: outputting position information, displaying images, remote control based on a control slider, and remote control based on a joystick. Outputting position information refers to outputting the position information of the first ablation device. Displaying images refers to displaying images inside the operating room. Remote control based on a control slider refers to sending a first control command to the control device of the second robotic arm based on a sliding operation of the control slider. Optionally, the control device sends the first control command to the control device based on Recommended Standard 485 (RS485). Remote control based on a joystick refers to sending a third control command to the second robotic arm based on a movement operation of the joystick, so that the second robotic arm controls the movement of the second robotic arm.

[0084] Optionally, Figure 5 This is a schematic diagram of a scene outside an operating room, provided as an embodiment of this application. Figure 5 As shown, outside the operating room, the control device and the force feedback device have a communication connection. The force feedback device has a communication connection with the first force sensor of the first ablation device. Through this communication connection, it can acquire data detected by the first sensor, and then obtain information about the first force based on this data. The force feedback device then obtains the magnitude of the second force based on the information about the first force. Based on the magnitude of the second force, it generates a second control command and sends the second control command to the damping device of the control device. The force feedback device can also acquire the first pose of the first force sensor at a first moment and the second pose of the first force sensor at a second moment through this communication connection, and send the first pose and the second pose to the control device. After acquiring the first pose and the second pose, the control device can determine the difference between the first pose and the second pose (hereinafter referred to as the pose difference). For example, the pose of the first force sensor is... The pose of the sensing sensor is The pose difference is .

[0085] The control device converts the pose difference into a pose in the sensing coordinate system, obtaining the transformed pose, where the sensing sensor is the coordinate system constructed based on the sensing sensor. The control device sends the transformed pose to the control device of the first robotic arm. The control device then sends the pose difference to the control device of the first robotic arm. Based on the pose of the sensing sensor at time t1 and the transformed pose, the control device of the first robotic arm obtains the target pose of the sensing sensor at time t2, and adjusts the pose of the sensing sensor at time t2 to the target pose. For example, the pose of the sensing sensor at time t1 is... This pose is in the sensing coordinate system. This allows the pose of the sensing sensor to be close to that of the first force sensor. Since the first force sensor is deployed on the first ablation device, the close proximity of the sensing sensor's pose to the first force sensor allows the sensing sensor to better perceive the environment around the first sensor. For example, when the first ablation device reaches the termination position of the first reference path, the distance from the sensing sensor to the area to be ablated is 2 cm to 5 cm.

[0086] The operating room includes a control device for a second robotic arm, an indoor camera, and a control device for a first robotic arm. The control device for the second robotic arm performs image processing, specifically 3D reconstruction based on at least one CT image to obtain a 3D CT image. It also plans a path, obtaining a second reference path based on the 3D CT image. Furthermore, it controls navigation and positioning, moving the second robotic arm based on the first reference path, with navigation commands instructing the control device to move the second robotic arm along the first reference path. The control device also drives a first ablation device into the target phantom, controlling a drive mechanism based on a first control command to move the first ablation device along the axis of the placement hole. The indoor camera captures images from inside the operating room and transmits them to a control device deployed outside the operating room, allowing a user outside to view the situation inside. The control device for the first robotic arm drives a sensing sensor, controlling its movement to bring the sensor into the target phantom and adjusting the sensor's pose. It also acquires sensing information from the sensor and transmits this information to the control device.

[0087] The control device sends the following information to the control equipment of the second robotic arm: a first control command and a target depth, wherein the target depth is the desired depth to which the first ablation device enters the target phantom. The control equipment of the second robotic arm can then control the movement of the second robotic arm based on the first control command and the target depth. The control equipment of the second robotic arm sends the following information to the control device: the depth of the first ablation device entering the target phantom, information on the first force, and the position information of the first ablation device, so that the user outside the operating room can receive this information in a timely manner.

[0088] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0089] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, while using clear signs / information to inform users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, personal information processing may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0090] The methods of the embodiments of this application have been described in detail above, and the apparatus of the embodiments of this application is provided below.

[0091] Please see Figure 6 , Figure 6 This application provides a schematic diagram of a control device based on dual robotic arms and force feedback. The control device 1 includes an acquisition unit 11 and a processing unit 12, wherein: The acquisition unit 11 is used to acquire first sensing information after controlling the first robotic arm to drive the sensing sensor into the target model. The first sensing information is the sensing information collected by the sensing sensor in the target model. The acquisition unit 11 is used to acquire a first control command, which instructs the first ablation device to enter the target phantom, the target phantom including a region to be ablated, the first ablation device being used to ablate the region to be ablated, and the first control command being obtained through a movement control device. The acquisition unit 11 is further configured to acquire information about a first force after controlling the second robotic arm to drive the first ablation device into the target model based on the first control command and the first sensing information. The information about the first force includes the magnitude of the first force, which is the force experienced by the first ablation device after entering the target model. Processing unit 12 is configured to determine the magnitude of a second force based on the magnitude of the first force, the second force being used to impede the movement of the control device, and the magnitude of the second force being positively correlated with the magnitude of the first force; The processing unit 12 is further configured to generate a second control command based on the magnitude of the second force, the second control command instructing the damping device in the control device to generate the second force, the damping device being configured to generate the force that hinders the movement of the control device; The processing unit 12 is further configured to send the second control command to the damping device to cause the damping device to generate the second force for resisting the movement of the control device.

[0092] In any embodiment of this application, the feedback information includes a second force, which is used to impede the movement of the control device, and the information of the first force includes the magnitude of the first force; The processing unit 12 is used for: The magnitude of the second force is determined based on the magnitude of the first force, and the magnitude of the second force is positively correlated with the magnitude of the first force. Based on the magnitude of the second force, a second control command is generated, which instructs the damping device in the control device to generate the second force, the damping device being used to generate the force that hinders the movement of the control device; The second control command is sent to the damping device to cause the damping device to generate the second force to impede the movement of the control device.

[0093] In any embodiment of this application, the first control command instructs the first ablation device to move from outside the target phantom to inside the target phantom based on a first reference path; The acquisition unit 11 is further configured to acquire the first reference path; The acquisition unit 11 is further configured to acquire the position of the first ablation device after controlling the second robotic arm to drive the first ablation device into the target phantom based on the first control command and the first sensing information. The processing unit 12 is further configured to obtain the depth of the first ablation device entering the target phantom based on the first reference path and the position of the first ablation device; The processing unit 12 is further configured to determine the magnitude of the second force based on the magnitude of the first force and the depth, wherein the magnitude of the second force is positively correlated with the depth.

[0094] In any embodiment of this application, the processing unit 12 is used for: When the depth is less than or equal to the length of the first reference path, the magnitude of the second force is determined based on the magnitude of the first force, the depth, and a first value; wherein the magnitude of the second force is positively correlated with a first product, the first product being the product of the depth and the first value, and the first value being a positive number; When the depth is greater than the length of the first reference path, the magnitude of the second force is determined based on the magnitude of the first force, the depth, and the second value; wherein the magnitude of the second force is positively correlated with a second product, the second product being the product of the depth and the second value, and the second value being greater than the first value.

[0095] In any embodiment of this application, the acquisition unit 11 is further configured to acquire information about a third force, which is the force experienced by the second ablation device after entering the target phantom. The second ablation device is configured to ablate the area to be ablated, and the second ablation device enters the target phantom before the first ablation device enters the target phantom. The processing unit 12 is further configured to estimate the target deformation of the target model based on the information of the third force and the target model, wherein the target model is configured to estimate the deformation of the target model based on the force acting on the target model. The acquisition unit 11 is further configured to acquire a second reference path, which is the path by which the first ablation device enters the target phantom from outside the target phantom. The second reference path is planned before the second ablation device enters the target phantom. The acquisition unit 11 is further configured to modify the second reference path based on the target deformation to obtain the first reference path.

[0096] In any embodiment of this application, the first sensing information is collected before the second ablation device enters the target phantom; the acquisition unit 11 is further configured to acquire second sensing information, which is sensing information collected by the sensing sensor inside the target phantom, and the second sensing information is collected after the second ablation device enters the target phantom. The processing unit 12 is further configured to estimate the target deformation of the target model based on the third force and the target model when the difference between the first perceived information and the second perceived information is greater than a first threshold.

[0097] In any embodiment of this application, the processing unit 12 is used for: Based on the second perceived information, a three-dimensional reconstruction is performed on the target phantom to obtain the target's three-dimensional information; Based on the target deformation, the starting position of the second reference path is corrected to obtain the corrected starting position; Based on the target's three-dimensional information, the termination position of the second reference path is corrected to obtain the corrected termination position. The first reference path is obtained based on the corrected start position and the corrected end position.

[0098] In this embodiment, after acquiring a first control command and first sensing information, the control device controls the first ablation device to enter the target phantom based on the first control command and first sensing information, thereby improving control accuracy. Then, it acquires information about the first force experienced by the first ablation device after entering the target phantom, and determines the magnitude of a second force to hinder the movement of the control device based on this information. The first control command is obtained by moving the control device. Based on the magnitude of the second force, a second control command is generated and sent to a damping device in the control device, causing the damping device to generate a second force to hinder the movement of the control device. This allows the user to promptly know the force situation experienced by the first ablation device after entering the target phantom. Furthermore, this improves the control effect when remotely controlling the first ablation device to ablate the target phantom.

[0099] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0100] Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device 2 includes a processor 21 and a memory 22. Optionally, the electronic device 2 also includes an input device 23 and an output device 24. The processor 21, memory 22, input device 23, and output device 24 are coupled together via connectors, which include various interfaces, transmission lines, or buses, etc., and are not limited in this embodiment. It should be understood that in the various embodiments of this application, coupling refers to mutual connection in a specific way, including direct connection or indirect connection through other devices, such as through various interfaces, transmission lines, buses, etc.

[0101] Processor 21 can be one or more graphics processing units (GPUs). If processor 21 is a GPU, the GPU can be a single-core GPU or a multi-core GPU. Optionally, processor 21 can be a processor group composed of multiple GPUs, with the multiple processors coupled to each other via one or more buses. Optionally, the processor can also be other types of processors, etc., which are not limited in this embodiment.

[0102] The memory 22 can be used to store computer program instructions, as well as various types of computer program code, including program code for executing the scheme of this application. Optionally, the memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), which is used for related instructions and data.

[0103] Input device 23 is used to input data and / or signals, and output device 24 is used to output data and / or signals. Input device 23 and output device 24 can be independent devices or an integrated device.

[0104] It is understood that in this embodiment of the application, the memory 22 can be used not only to store related instructions, but also to store related data. This embodiment of the application does not limit the specific data stored in the memory.

[0105] Understandable, Figure 7 This is merely a simplified design of an electronic device. In practical applications, the electronic device may also include other necessary components, including, but not limited to, any number of input / output devices, processors, memories, etc., and all electronic devices that can implement the embodiments of this application are within the protection scope of this application.

[0106] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0107] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will also readily understand that the various embodiments of this application have different focuses, and for the sake of convenience and brevity, the same or similar parts may not be repeated in different embodiments. Therefore, parts not described or not described in detail in one embodiment can be referred to the descriptions in other embodiments.

[0108] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0109] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0110] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0111] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0112] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or RAM, magnetic disks, or optical disks.

Claims

1. A control method based on dual robotic arms and force feedback, characterized in that, The method includes: After controlling the first robotic arm to drive the sensing sensor into the target model, the first sensing information is obtained. The first sensing information is the sensing information collected by the sensing sensor inside the target model. A first control command is obtained, which instructs a first ablation device to move from outside the target phantom to inside the target phantom based on a first reference path. The target phantom includes a region to be ablated. The first ablation device is used to ablate the region to be ablated. The first control command is obtained through a movement control device. After controlling the second robotic arm to drive the first ablation device into the target phantom based on the first control command and the first sensing information, the information of the first force is obtained. The information of the first force includes the magnitude of the first force, which is the force that the first ablation device experiences after entering the target phantom. Information about a third force is obtained, which is the force experienced by the second ablation device after entering the target phantom. The second ablation device is used to ablate the area to be ablated. The second ablation device enters the target phantom before the first ablation device enters the target phantom. Based on the information of the third force and the target model, the target deformation of the target model is estimated, and the target model is used to estimate the deformation of the target model based on the force acting on the target model. Obtain a second reference path, which is the path taken by the first ablation device from outside the target phantom into the target phantom. The second reference path is planned before the second ablation device enters the target phantom. Obtain the first reference path; obtaining the first reference path includes: modifying the second reference path based on the target deformation to obtain the first reference path; After controlling the second robotic arm to drive the first ablation device into the target phantom based on the first control command and the first sensing information, the position of the first ablation device is obtained; Based on the first reference path and the position of the first ablation device, the depth to which the first ablation device enters the target phantom is obtained; Based on the magnitude of the first force, the magnitude of the second force is determined, the second force being used to impede the movement of the control device, and the magnitude of the second force being positively correlated with the magnitude of the first force; determining the magnitude of the second force based on the magnitude of the first force includes: determining the magnitude of the second force based on the magnitude of the first force and the depth, and the magnitude of the second force being positively correlated with the depth; Based on the magnitude of the second force, a second control command is generated, which instructs the damping device in the control device to generate the second force, the damping device being used to generate the force that hinders the movement of the control device; The second control command is sent to the damping device to cause the damping device to generate the second force to impede the movement of the control device.

2. The method according to claim 1, characterized in that, Determining the magnitude of the second force based on the magnitude of the first force and the depth includes: When the depth is less than or equal to the length of the first reference path, the magnitude of the second force is determined based on the magnitude of the first force, the depth, and a first value; wherein the magnitude of the second force is positively correlated with a first product, the first product being the product of the depth and the first value, and the first value being a positive number; When the depth is greater than the length of the first reference path, the magnitude of the second force is determined based on the magnitude of the first force, the depth, and the second value; wherein the magnitude of the second force is positively correlated with a second product, the second product being the product of the depth and the second value, and the second value being greater than the first value.

3. The method according to claim 1, characterized in that, The first sensing information was collected before the second ablation device entered the target phantom; Before estimating the target deformation of the target model based on the information of the third force and the target model, the method further includes: Acquire second sensing information, which is sensing information collected by the sensing sensor inside the target phantom, and the second sensing information is collected after the second ablation device enters the target phantom; The estimation of the target deformation variables of the target model based on the information of the third force and the target model includes: If the difference between the first perceived information and the second perceived information is greater than a first threshold, the target deformation is estimated based on the magnitude of the third force and the target model.

4. The method according to claim 3, characterized in that, The step of modifying the second reference path based on the target deformation to obtain the first reference path includes: Based on the second perceived information, a three-dimensional reconstruction is performed on the target phantom to obtain the target's three-dimensional information; Based on the target deformation, the starting position of the second reference path is corrected to obtain the corrected starting position; Based on the target's three-dimensional information, the termination position of the second reference path is corrected to obtain the corrected termination position. The first reference path is obtained based on the corrected start position and the corrected end position.

5. A control device based on dual robotic arms and force feedback, characterized in that, The control device based on dual robotic arms and force feedback includes: The acquisition unit is used to acquire first sensing information after controlling the first robotic arm to drive the sensing sensor into the target model. The first sensing information is the sensing information collected by the sensing sensor inside the target model. The acquisition unit is used to acquire a first control command, which instructs the first ablation device to move from outside the target phantom to inside the target phantom based on a first reference path. The target phantom includes a region to be ablated. The first ablation device is used to ablate the region to be ablated. The first control command is obtained through a movement control device. The acquisition unit is further configured to acquire information about a first force after controlling the second robotic arm to drive the first ablation device into the target phantom based on the first control command and the first sensing information. The information about the first force includes the magnitude of the first force, which is the force experienced by the first ablation device after entering the target phantom. The processing unit is configured to determine the magnitude of a second force based on the magnitude of the first force, the second force being used to impede the movement of the control device, and the magnitude of the second force being positively correlated with the magnitude of the first force. The acquisition unit is also used to acquire information about a third force, which is the force experienced by the second ablation device after entering the target phantom. The second ablation device is used to ablate the area to be ablated. The second ablation device enters the target phantom before the first ablation device enters the target phantom. The processing unit is further configured to estimate the target deformation of the target model based on the information of the third force and the target model, wherein the target model is configured to estimate the deformation of the target model based on the force acting on the target model; The acquisition unit is further configured to acquire a second reference path, the second reference path being the path by which the first ablation device enters the target phantom from outside the target phantom, and the second reference path being planned before the second ablation device enters the target phantom; The acquisition unit is further configured to acquire the first reference path; the acquisition of the first reference path includes: modifying the second reference path based on the target deformation to obtain the first reference path; The processing unit is further configured to obtain the position of the first ablation device after controlling the second robotic arm to drive the first ablation device into the target phantom based on the first control command and the first sensing information; The processing unit is further configured to determine the depth at which the first ablation device enters the target phantom based on the first reference path and the position of the first ablation device. The processing unit is further configured to generate a second control command based on the magnitude of the second force, the second control command instructing a damping device in the control device to generate the second force, the damping device being configured to generate the force that hinders the movement of the control device; the determination of the magnitude of the second force based on the magnitude of the first force includes: determining the magnitude of the second force based on the magnitude of the first force and the depth, the magnitude of the second force being positively correlated with the depth; The processing unit is further configured to send the second control command to the damping device to cause the damping device to generate the second force for resisting the movement of the control device.

6. A surgical robot, characterized in that, Includes the control device based on dual robotic arms and force feedback as described in claim 5.

7. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store computer program code, the computer program code including computer instructions, wherein, when the processor executes the computer instructions, the electronic device performs the method as described in any one of claims 1 to 4.

Images