An automatic processing method based on six-dimensional force signals

Abstract

The present invention relates to an automated processing method based on six-dimensional force signals, including: collecting a signal y of the cutting force during osteotomy operation, discretizing the signal y, using a number of fuzzy state descriptions to obtain a state set; obtaining a processing action, dividing the processing action into a fuzzy set, and obtaining an action set; the state and the action constitute a Q table; therefore, for each state of the corresponding state set, its fuzzy rules are designed respectively, and the weight of each fuzzy rule is defined; the Q value in the Q table is updated iteratively through reinforcement learning to obtain the optimal Q table; the Q table is applied to realize automated processing. Compared with the prior art, the present invention realizes perceptual processing through fuzzy reinforcement learning, and can realize the surgical robot learning the doctor's processing operation to complete the operation. The reaction speed is due to the operation of ordinary people, which effectively improves the safety of the operation. In the process of automated processing, the experience of a large number of learning experts can be used to avoid damaging important tissue structures while completing the operation.

Description

Technical Field

[0001] The present invention relates to the field of medical surgical robot automation technology, and in particular to an automated processing method based on six-dimensional force signals. Background Art

[0002] With the continuous advancement of surgical robot-related technologies, doctors need robots to assist in completing difficult operations in many scenarios. At present, the force control solution for the robotic arm is mainly to install a six-degree-of-freedom force sensor at the end of the surgical robot to sense the force exerted on the surgical tool and operate the robotic arm through force position control, that is, the movement of the robotic arm is consistent with the operator's hand traction action. However, in osteotomy surgery, such a force control solution cannot guarantee surgical accuracy and has high requirements for the operator.

[0003] Taking the vascularized fibula transplant surgery as an example, the doctor needs to use surgical tools to cut the fibula segment, but the blood vessels of the fibula cannot be damaged. This requires the robot arm to control the movement according to force sense during the operation. Since the force required for the operation is large, the operator's hand shakes a lot, which not only reduces the accuracy of the operation, but also requires a lot of physical strength from the operator. Summary of the invention

[0004] The purpose of the present invention is to provide an automated processing method based on six-dimensional force signals in order to overcome the defects of the above-mentioned prior art.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] An automated processing method based on six-dimensional force signals comprises the following steps:

[0007] The cutting force signal y during the osteotomy operation is collected, including the force F in the direction of the osteotomy line and the position x in the direction of the osteotomy line. The signal y = [Fx] is discretized and described using several fuzzy states to obtain the state set S = {s i |i=1,…,N}; obtain the processing action, divide the processing action into fuzzy sets, and obtain the action set A={a j ,j=1,…,M}; the state and action constitute a Q table, and the behavior state set of the Q table is listed as the action set;

[0008] The values ​​in the Q table correspond to the formulation of fuzzy rules, so for each state of the state set, a fuzzy rule is designed and the weight of each fuzzy rule is defined;

[0009] By iteratively updating the Q value in the Q table through reinforcement learning, the optimal Q table is obtained, that is, the optimal fuzzy rule is obtained;

[0010] Apply Q table to realize automatic processing.

[0011] Preferably, the “signal y for collecting the cutting force during osteotomy operation” is:

[0012] A cutting force collection platform was built, and the force sensor was fixed on the table. A clamp was installed on the force sensor to clamp the bone segment model. Reference frames with reflective balls were installed on the bone segment model and the surgical tool respectively. The relative position between the bone segment model and the surgical tool, as well as the movement speed of the surgical tool and the reciprocating saw were determined in real time under the optical navigation system, and the measurement data of the force sensor was obtained in real time, which was then converted into a cutting force signal y.

[0013] Preferably, the force sensor is a six-dimensional force / torque sensor.

[0014] Preferably, the “determining in real time the relative position between the bone segment model and the surgical tool, and the movement speed of the surgical tool and the reciprocating saw under the optical navigation system” specifically includes:

[0015] Obtain the relative position between the bone segment model and surgical tools under the optical navigation system Assume that within a time period T, the relative positions at the beginning and end of the period are and have:

[0016]

[0017] in, is the attitude change amount, there are:

[0018]

[0019] Map it to the Lie algebra ζe(3) by logarithmic transformation, that is:

[0020]

[0021] The superscript V refers to the definition of converting an antisymmetric matrix into a vector. is the velocity of the surgical tool tip, and is solved as follows:

[0022]

[0023] Preferably, the signal y=[Fx] is discretized using a Gaussian membership function as follows:

[0024]

[0025] Among them, f(x,σ,c) is a Gaussian function, c is the center of the function curve, and σ is a scale parameter that controls the width of the function, which fuzzifies the signal y=[Fx]:

[0026]

[0027] Among them, Z l (y i ) represents the state y i The fuzzy value of the lth layer, l is the number of fuzzy layers, that is, F and x are fuzzified into l layers respectively, and the number of states in the state set is l×l.

[0028] Preferably, the value of l is 4.

[0029] Preferably, the processing actions are divided into fuzzy sets {NB, N, Z, P, PB}, where NB, N, Z, P, PB represent five fuzzy levels of negative large, negative, zero, positive, and positive large, respectively.

[0030] Preferably, the i-th fuzzy rule R i , that is, the corresponding state s i Fuzzy rule R ii as follows:

[0031] If i then y isa(i,1)withQ(i,1)

[0032] or isa(i,2)withQ(i,2)

[0033] …

[0034] or isa(i,j)withQ(i,j)

[0035] Where a(i,j) represents the action executed, i and j identify the state and action respectively, Q(i,j) represents the evaluation value of the action executed under the state, that is, the Q value of the i-th row and j-th column in the Q table, i and j identify the state and action respectively;

[0036] The i-th fuzzy rule R i The weight of is defined as:

[0037] α i (s)=min(Z(y1),Z(y2),…)

[0038] Among them, Z(y1), Z(y2),… represent the fuzzy values ​​of states y1, y2,… respectively.

[0039] Preferably, in the “iteratively updating the Q value in the Q table by reinforcement learning”, during the iteration process, in the state s i Next, select the row in the Q table corresponding to the fuzzy rule, and the execution action with the smallest Q value is the optimal execution action:

[0040]

[0041] The action output of the defuzzified calculation system is obtained by defuzzifying the optimal execution action in each fuzzy rule:

[0042]

[0043] The corresponding Q value is:

[0044]

[0045] Q(s,a) represents the Q value under state s and action a;

[0046] The smaller the speed error of the surgical tool, the larger the reward function value, and the reward function is defined as:

[0047] r=(vv * ) 2

[0048] Among them, (vv * ) represents the speed error of the surgical tool;

[0049] The Q value update formula is:

[0050] Q t (s,a)=Q t-1 (s,a)+β(r+γQ t-1 (s′,a′)-Q t-1 (s,a))

[0051] Among them, t represents the moment in the reinforcement learning iteration; β is the learning rate of the reinforcement learning algorithm; γ is the discount coefficient of the reinforcement learning algorithm; s′ represents the next state, and a′ represents the next action.

[0052] Preferably, the termination condition of the reinforcement learning iteration is: the updated value of the Q value is less than a preset threshold epsilon.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] The present invention realizes perception processing through fuzzy reinforcement learning, which enables the surgical robot to learn the doctor's processing operations and complete the operation. The reaction speed is faster than that of ordinary people, which effectively improves the safety of the operation. In the process of automated processing, it can avoid damaging important tissue structures while completing the operation by learning a lot of experts' experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flow chart of the present invention;

[0056] Figure 2 A schematic diagram of a cutting force collection platform constructed for the present invention;

[0057] Figure 3 To strengthen the training process of learning;

[0058] Figure numerals: 1. six-dimensional force / torque sensor, 2. reference frame, 3. surgical tool, 4. fibula model. DETAILED DESCRIPTION

[0059] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and a detailed implementation method and a specific operation process are given. It should be understood that these descriptions are only for further illustrating the features and advantages of the present invention, and are not limitations on the claims of the present invention. The description of this part is only for several typical embodiments, but the protection scope of the present invention is not limited to the following embodiments. The same or similar prior art means and some technical features in the embodiments are replaced with each other within the scope of the description and protection of the present invention.

[0060] As used herein, "one embodiment" or "embodiment" refers to a specific feature, structure or characteristic that may be included in at least one implementation of the present invention. In the description of the present invention, it is to be understood that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0061] Embodiment 1:

[0062] An automated processing method based on six-dimensional force signals comprises the following steps:

[0063] The cutting force signal y during the osteotomy operation is collected, including the force F in the direction of the osteotomy line and the position x in the direction of the osteotomy line. The signal y = [Fx] is discretized and described using several fuzzy states to obtain the state set S = {s i |i=1,…,N}; obtain the processing action, divide the processing action into fuzzy sets, and obtain the action set A={a j ,j=1,…,M}; the state and action constitute a Q table, and the behavior state set of the Q table is listed as the action set;

[0064] The values ​​in the Q table correspond to the formulation of fuzzy rules, so for each state of the state set, its fuzzy rules are designed respectively, and the weight of each fuzzy rule is defined;

[0065] By iteratively updating the Q value in the Q table through reinforcement learning, the optimal Q table is obtained, that is, the optimal fuzzy rule is obtained;

[0066] Apply Q table to realize automatic processing.

[0067] Specifically, as follows:

[0068] In order to realize the control of surgical robot system simulating the osteotomy of surgical experts, a cutting force acquisition platform was built, such as Figure 2 As shown in the figure, the key data of the simulated clinical treatment scenario is collected. The platform uses the same hardware devices as the surgical robot system to avoid the impact of equipment differences on the data. At the same time, the operator's operation is also as close to the model clinical scenario as possible.

[0069] The force sensor used in the platform is a six-dimensional force / torque sensor 1, the bone segment model is a fibula model 4, and the surgical tool 3 is a reciprocating saw. The six-dimensional force / torque sensor 1 is fixed on the table, and a clamp installed on the six-dimensional force / torque sensor 1 is designed to clamp the fibula model 4 to keep the fibula model 4 stable. A reference frame 2 with a reflective ball is installed on the fibula model 4 and the surgical tool 3 respectively, so that the relative position between the two can be determined in real time under the optical navigation system. The speed of the reciprocating saw with the surgical tool. Assume that within a time period T, the relative positions at the beginning and end of the period are and have:

[0070]

[0071] in, is the attitude change amount, there are:

[0072]

[0073] Map it to the Lie algebra ζe(3) by logarithmic transformation, that is:

[0074]

[0075] The superscript V refers to the definition of converting an antisymmetric matrix into a vector. is the velocity of the surgical tool tip, and is solved as follows:

[0076]

[0077] Through force analysis, gravity compensation, coordinate system alignment and other methods, the measurement signals of the six-dimensional force / torque sensor and the power source pedal force sensor signal are combined and filtered to obtain the cutting force.

[0078] This application proposes a model based on fuzzy reinforcement learning to realize perception processing. The fuzzy reinforcement system can convert the state set S = {s i |i=1,…,N} is mapped to the action set A={a j ,j=1,…,M}, such as Figure 3As shown, in reinforcement learning, the agent performs actions in a state to switch to other states, the surgical robot performs processing, and determines the reward value based on the speed error of the surgical tool and feeds it back to the agent.

[0079] Considering its safety, only the force F in the direction of the osteotomy line and the position x in the direction of the osteotomy line are analyzed. Since F and x are continuous quantities obtained in real time, the state quantity of their combination will be infinite, and the Q table cannot express it. Therefore, it is necessary to fuzzify the continuous variables to make them finite state quantities. Therefore, it is necessary to fuzzify the real-time signal and use the fuzzified signal as the input of the fuzzy rule. The Gaussian membership function is a common function used for fuzzy processing of continuous quantities:

[0080]

[0081] Among them, f(x,σ,c) is a Gaussian function, c is the center of the function curve, σ is a scale parameter, usually positive, controlling the width of the function, and fuzzifying the continuous variation y=[Fx] in this application:

[0082]

[0083] Among them, Z l (y i ) represents the state y i The fuzzy value of the lth layer, l is the number of fuzzy layers, that is, F and x are fuzzified into l layers respectively, and the number of states in the state set is l×l. In this embodiment, the value of l is 4, then F and x are fuzzified into 4 layers respectively, and there are 16 fuzzy states in the processing model.

[0084] The processing actions are divided into fuzzy sets {NB, N, Z, P, PB}, where NB, N, Z, P, PB represent five fuzzy levels: negative, negative, zero, positive, and positive. Therefore, the state and action can form a 16×5 Q table.

[0085] In traditional fuzzy control, fuzzy rules are usually compiled by operators or experts. In complex or unknown scenarios, fuzzy rules need to be learned through pre-collected data, and the values ​​in the Q table replace the formulation of fuzzy rules. i , that is, the corresponding state s i Fuzzy rule R i as follows:

[0086] If i then y isa(i,1)withQ(i,1)

[0087] or isa(i,2)withQ(i,2)

[0088] …

[0089] or isa(i,j)withQ(i,j)

[0090] Where a(i,j) represents the action executed, i and j identify the state and action respectively, Q(i,j) represents the evaluation value of the action executed under the state, that is, the Q value of the i-th row and j-th column in the Q table, i and j identify the state and action respectively;

[0091] The i-th fuzzy rule R i The weight of is defined as:

[0092] α i (s)=min(Z(y1),Z(y2),…)

[0093] Among them, Z(y1), Z(y2),… represent the fuzzy values ​​of states y1, y2,… respectively.

[0094] The values ​​in the Q table determine the specific formulation of fuzzy rules. In order to obtain the best control effect, it is necessary to formulate the best fuzzy rules, that is, to obtain the optimal Q table. The update and optimization of the Q value is calculated through continuous iterative updates through reinforcement learning.

[0095] During the iteration, in state s i Next, select the row in the Q table corresponding to the fuzzy rule, and the execution action with the smallest Q value is the optimal execution action:

[0096]

[0097] The action output of the defuzzified calculation system is obtained by defuzzifying the optimal execution action in each fuzzy rule:

[0098]

[0099] The corresponding Q value is:

[0100]

[0101] Q(s,a) represents the Q value under state s and action a;

[0102] In order to fit the experience of experts, the smaller the speed error of the surgical tool, the larger the reward function value, and the reward function is defined as:

[0103] r=(vv * ) 2

[0104] Among them, (vv * ) represents the speed error of the surgical tool;

[0105] The Q value update formula is:

[0106] Qt (s,a)=Q t-1 (s,a)+β(r+γQ t-1 (s′,a′)-Q t-1 (s,a))

[0107] Among them, t represents the moment in the reinforcement learning iteration; β is the learning rate of the reinforcement learning algorithm; γ is the discount coefficient of the reinforcement learning algorithm; s′ represents the next state, and a′ represents the next action.

[0108] The termination condition of the reinforcement learning iteration is: the updated Q value is less than the preset threshold epsilon.

[0109] The present invention realizes perception processing through fuzzy reinforcement learning, which enables the surgical robot to learn the doctor's processing operations and complete the operation. The reaction speed is faster than that of ordinary people, which effectively improves the safety of the operation. In the process of automated processing, it can avoid damaging important tissue structures while completing the operation by learning a lot of experts' experience.

[0110] Step 1: Perform osteotomy on the force collection platform and collect cutting force signals multiple times;

[0111] Step 2: Discretize the collected signals as training data;

[0112] Step 3: Construct the action network and the target action network. Both networks contain an input layer, two hidden layers and an output layer.

[0113] Step 4: Use the value function Q(t) of each action generated by the action network, select action a, and complete the training.

[0114] It should be noted that the present application can be implemented in software and / or a combination of software and hardware, for example, can be implemented using an application specific integrated circuit (ASIC), a general purpose computer or any other similar hardware device. In one embodiment, the software program of the present application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of the present application (including relevant data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present application can be implemented using hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0115] In addition, a part of the present application may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present application through the operation of the computer. The program instruction for calling the method of the present application may be stored in a fixed or removable recording medium, and / or transmitted through a data stream in a broadcast or other signal-bearing medium, and / or stored in a working memory of a computer device that runs according to the program instruction. Here, according to an embodiment of the present application, a device is included, the device including a memory for storing computer program instructions and a processor for executing program instructions, wherein, when the computer program instruction is executed by the processor, the device is triggered to run the method and / or technical solution based on the aforementioned multiple embodiments according to the present application.

[0116] The description and application of the present invention here are illustrative, and it is not intended to limit the scope of the present invention to the above-mentioned embodiments. The effects or advantages involved in the embodiments may not be embodied in the embodiments due to interference from various factors, and the description of the effects or advantages is not used to limit the embodiments. The deformation and change of the embodiments disclosed here are possible, and the replacement of the embodiments and the various equivalent parts are well known to those of ordinary skill in the art. It should be clear to those skilled in the art that the present invention can be implemented in other forms, structures, arrangements, proportions, and with other components, materials and parts without departing from the spirit or essential features of the present invention. Other deformations and changes can be made to the embodiments disclosed here without departing from the scope and spirit of the present invention.

Claims

1. An automated cutting method based on six-dimensional force signals, characterized in that: The following steps are involved: The cutting force signal y during the osteotomy operation is collected, including the force F in the direction of the osteotomy line and the position x in the direction of the osteotomy line. The signal y = [F x ] is discretized and described using several fuzzy states to obtain the state set ={ }; Get the cutting action, divide the cutting action into fuzzy sets, and get the action set ={ }; The states and actions form a Q table, and the behavior state set of the Q table is listed as the action set; The values ​​in the Q table correspond to the formulation of fuzzy rules, so for each state of the state set, a fuzzy rule is designed and the weight of each fuzzy rule is defined; By iteratively updating the Q value in the Q table through reinforcement learning, the optimal Q table is obtained, that is, the optimal fuzzy rule is obtained; Apply Q table to realize automatic cutting; The "signal y of cutting force during osteotomy operation" is: A cutting force acquisition platform is built, and a force sensor is fixed on the table. A clamp is installed on the force sensor, and the clamp is used to clamp the bone segment model. Reference frames with reflective balls are installed on the bone segment model and the surgical tool respectively. The relative position between the bone segment model and the surgical tool, as well as the movement speed of the surgical tool reciprocating saw are determined in real time under the optical navigation system, and the measurement data of the force sensor is obtained in real time, which is then converted into a cutting force signal y. The “real-time determination of the relative position between the bone segment model and the surgical tool, and the movement speed of the surgical tool and the reciprocating saw under the optical navigation system” is specifically: Obtain the relative position between the bone segment model and surgical tools under the optical navigation system , set in a time period The relative positions at the beginning and end of this cycle are and ,have: , in, is the attitude change amount, there are: , Map it to Lie algebra by logarithmic transformation ,Right now: , Among them, the upper right Refers to the definition of converting an antisymmetric matrix into a vector, is the velocity of the surgical tool tip, and is solved as follows: 。 2. The automatic cutting method based on six-dimensional force signal according to claim 1, characterized in that: The force sensor is a six-dimensional force / torque sensor.

3. The automatic cutting method based on six-dimensional force signal according to claim 1, characterized in that: The signal y=[F x] is discretized using the Gaussian membership function as follows: , in, is a Gaussian function, is the center of the function curve, is the scale parameter, which controls the width of the function and fuzzifies the signal y=[F x]: , in, Indicates status No. The fuzziness value of the layer, is the number of fuzzy layers, that is, fuzzify F and x into layer, the number of states in the state set is × .

4. The automatic cutting method based on six-dimensional force signal according to claim 3, characterized in that: The value of is 4.

5. The automatic cutting method based on six-dimensional force signal according to claim 1, characterized in that: Divide the cutting actions into fuzzy sets ,in They represent five fuzzy levels: negative, negative, zero, positive, and positive.

6. The automatic cutting method based on six-dimensional force signal according to claim 3, characterized in that: No. Fuzzy rules , that is, the corresponding state Fuzzy rules as follows: If then y is with , or is with , … or is with , in, Indicates the action to be performed. Identify states and actions respectively. Indicates the evaluation value of the action executed in the state, that is, Line The Q value of the column, Identify states and actions respectively; No. Fuzzy rules The weight of is defined as: , in, Respectively indicate the status The fuzzy value of .

7. The automatic cutting method based on six-dimensional force signals according to claim 6, characterized in that: In the “Iteratively updating the Q value in the Q table through reinforcement learning”, during the iteration process, in the state Next, select the row in the Q table corresponding to the fuzzy rule, and the execution action with the smallest Q value is the optimal execution action: , The action output of the defuzzified calculation system is obtained by defuzzifying the optimal execution action in each fuzzy rule: , The corresponding Q value is: , express State Actions The Q value under The smaller the speed error of the surgical tool, the larger the reward function value, and the reward function is defined as: , in, represents the velocity error of the surgical tool; The Q value update formula is: , Where t represents the moment in the reinforcement learning iteration; is the learning rate of the reinforcement learning algorithm; is the discount factor of the reinforcement learning algorithm; Indicates the next state, Indicates the next action.

8. The automatic cutting method based on six-dimensional force signal according to claim 1, characterized in that: The termination condition of the reinforcement learning iteration is: the updated value of the Q value is less than the preset threshold epsilon.

Images