Robot arm operation system and control method thereof

Abstract

To provide a robot arm operation system capable of obtaining appropriate assist when a medical activity is performed using a medical instrument from a remote location, and a control method thereof.SOLUTION: A robot arm operation system 1 includes: a storage part 120 that stores a large number of pieces of model data in which state information indicating a state of a body portion of a patient for which a medical instrument is used and operation information regarding an operation of a robot arm 230 corresponding to the state are associated with each other; an extraction part 140 that extracts model data corresponding to a current state of the body portion of the patient on the basis of the state information in the large number of pieces of model data of the storage part 120; an operation part 110 that an operator operates; and an arm control part 160 that operates the robot arm 230 on the basis of the operation on the operation part 110. The arm control part 160 causes a behavior of the robot arm 230 corresponding to the operation performed on the operation part 110 to approach to the operation of the robot arm 230 that operation information of the model data extracted by the extraction part 140 indicates.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a robot arm operation system and a control method thereof.

Background Art

[0002] Conventionally, a robot arm operation system has been proposed in which a robot arm equipped with a surgical instrument is operated by an operator to perform surgery (see, for example, Patent Document 1). With such a robot arm operation system, for example, even if the operator is at a remote location far from the patient to be operated on, surgery can be performed on the patient.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Here, in the system described in Patent Document 1, the surgical technique of the operator is of utmost importance. Moreover, since remote surgery is performed while grasping the surgical site of the patient through an image, it is more difficult than actual surgery and it becomes difficult to perform an appropriate surgery.

[0005] In addition, the system described in Patent Document 1 moves the robot arm as intended by the operator. Despite using a device called a robot, there is also a problem that it is not possible to obtain assistance for performing surgery more appropriately using the device.

[0006] Note that such problems are not limited to the case of performing surgery from a remote location. For example, when performing an injection from a remote location, or when a dentist grinds a tooth from a remote location, etc., they are common problems in other medical procedures performed using medical instruments from a remote location.

[0007] The present invention has been made to solve such conventional problems, and an object thereof is to provide a robot arm operation system capable of obtaining appropriate assistance when performing a medical act using a medical instrument from a remote location, and a control method therefor.

Means for Solving the Problems

[0008] The robot arm operation system according to the present invention is a robot arm operation system that performs a medical act using a medical instrument from a remote location by operating a robot arm having a medical instrument in accordance with an operation of an operator at a remote location, and includes a storage means that stores a large number of template data in which state information indicating a state of a body part of a patient using the medical instrument is associated with operation information regarding an operation of the robot arm corresponding to the state, an extraction means that extracts the template data corresponding to the state of the current patient's body part based on the state information among the large number of template data stored by the storage means, an operation means operated by the operator, and a control means that operates the robot arm based on an operation performed on the operation means. The control means makes the behavior of the robot arm according to the operation performed on the operation means approach the operation of the robot arm indicated by the operation information of the template data extracted by the extraction means.

[0009] A control method for a robot arm operation system according to the present invention operates a robot arm having a medical instrument in response to an operation of an operator at a remote location, thereby performing a medical act using the medical instrument from the remote location. The control method is for a robot arm operation system including a storage means that stores a large number of reference data in which state information indicating the state of a body part of a patient using the medical instrument is associated with operation information regarding the operation of the robot arm according to the state. The control method includes an extraction step of extracting the reference data according to the state of the current patient's body part based on the state information among the large number of reference data stored by the storage means, an operation step of the operator, and a control step of operating the robot arm according to the operation performed in the operation step. In the control step, the behavior of the robot arm according to the operation performed in the operation step is made to approach the operation of the robot arm indicated by the operation information of the reference data extracted in the extraction step.

Advantages of the Invention

[0010] According to the present invention, it is possible to provide a robot arm operation system capable of obtaining appropriate assistance.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Embodiments for Carrying Out the Invention

[0012] Hereinafter, the present invention will be described along with preferred embodiments. Note that the present invention is not limited to the embodiments shown below, and can be appropriately changed without departing from the gist of the present invention. Also, in the embodiments shown below, there are some places where the illustration and description of some configurations are omitted. However, regarding the details of the omitted technology, it goes without saying that publicly known or well-known technologies are appropriately applied within the range that does not conflict with the content described below.

[0013] FIG. 1 is a configuration diagram showing a robot arm operation system according to an embodiment of the present invention. As shown in FIG. 1, the robot arm operation system 1 includes an operation side device 100 and a robot arm device 200, and these are connected via a network N.

[0014] The operation side device 100 is provided at a remote location away from the robot arm device 200 and operates the robot arm device 200. The robot arm device 200 receives an operation signal from the operation side device 100 and operates a robot arm (refer to reference numeral 230 described later) based on the operation signal.

[0015] FIG. 2 is a block diagram showing the details of the robot arm operation system 1 shown in FIG. 1. As shown in FIG. 2, the operation side device 100 includes an operation unit (operation means) 110, a storage unit (storage means) 120, an input unit 130, an extraction unit (extraction means) 140, a generation unit (generation means) 150, an arm control unit (control means) 160, and a monitor 170. The robot arm device 200 includes an arm operation mechanism unit 210, a camera 220, and a robot arm 230. In this embodiment, the robot arm 230 is configured to include a robot hand capable of realizing the movement of the fingertips, and the robot arm operation system 1 is configured to be able to realize the movement of the fingertips operated by the operation unit 110.

[0016] The operation unit 110 performs operations for operating the robot arm 230 of the robot arm device 200. This operation unit 110 may be in the shape of a stick or the like, but preferably, it is configured to detect the three-dimensional movements of the fingers, hands, and arms of a doctor or the like who is an operator.

[0017] The storage unit 120 stores a large number of template data such as surgical procedures during past medical treatments (particularly, data of doctors who are considered to be good at surgery is preferable). FIG. 3 is a reference diagram showing an example of the template data stored in the storage unit 120 shown in FIG. 2. The large number of template data shown in FIG. 3 relates to the "act of first inserting the scalpel into the affected part" (an example of a predetermined act, which is an act of using a medical instrument on a body part) indicating the initial movement when first cutting the affected part with a scalpel. The storage unit 120 stores a large number of such template data (n or more in the example shown in FIG. 3: n is a natural number of 2 or more). The template data includes state information D1 and motion information D2, and these are associated with each other.

[0018] The state information D1 is information indicating the state of the body part of a patient using medical instruments such as a scalpel, scissors, needle holder, hook, forceps, syringe, speculum, etc. The medical instruments are not particularly limited to the above as long as they are held by a doctor or the like and used on the patient's body for medical purposes. For example, taping, etc. is also included, and an AED (automated external defibrillator), etc. is also included. The state information D1 is composed of a plurality of items (see, for example, the reference numerals D11 to D13 in FIG. 3), and the items vary according to the medical instruments used, the disease name, the drugs to be administered, etc. In the example shown in FIG. 3, the medical instrument is a scalpel. The first item D11 is the "affected part", the second item D12 is the "excised part", and the third item D13 is the "excised size". Specifically, in the template data "1", for the state information D1, the data of "stomach" is stored in the "affected part" which is the first item D11, the data of "lower part of the stomach" is stored in the "excised part" which is the second item D12, and the data of "circular with a diameter of 〇 cm (exposed × cm)" is stored in the "excised size" which is the third item D13. Thus, the state information D1 in the template data "1" is a disease such as gastric cancer, where a tumor with a diameter of 〇 cm is formed circularly in the lower part of the stomach, and × cm of the height of the tumor is exposed. For the three-dimensional ones such as the third item D13 among such state information D1, they are stored in the storage unit 120 by three-dimensional coordinate data (range data) in a state where a certain specific point of the patient's body is fixed.

[0019] Such state information D1 may be calculated and constructed by a computer based on the photographed image data near the affected part, image data such as X-rays, MRI (Magnetic Resonance Imaging) image data, and test values, etc. However, if possible, it is preferably constructed from the content previously judged and input by a doctor, or constructed after the doctor adds a correction value to the data calculated by the computer and adjusts it.

[0020] The operation information D2 is information regarding the operation of the robotic arm 230 according to the state information D1. The operation information D2 is composed of a plurality of components (see, for example, codes D21 to D24). The components of this operation information D2 also differ according to the medical device to be used, the disease name, the medicine to be administered, and the like. In the example shown in FIG. 3, the first component D21 is the "scalpel speed", the second component D22 is the "scalpel angle with respect to the affected part", the third component D23 is the "scalpel movement direction", and the fourth component D24 is the "scalpel depth". Although not shown in the figure, it is preferable that the components include data on the detailed movement of the fingertip and data on the pressure of the fingertip.

[0021] Specifically, in the example shown in FIG. 3, regarding the operation information D2 of the template data "1", data of "〇〇〇〇" is stored in the "scalpel speed" which is the first component D21, and data of "△△△△" is stored in the "scalpel angle with respect to the affected part" which is the second component D22. Data of "◇◇◇" is stored in the "scalpel movement direction" which is the third component D23, and data of "◇" is stored in the "scalpel depth (cutting depth)" which is the fourth component D24. These pieces of information are preferably composed of coordinate data or vector data of three-dimensional coordinates in a state where a specific point on the patient's body is fixed.

[0022] Here, the doctor selects the appropriate scalpel speed, angle, movement direction, and depth judged to be appropriate according to the affected part, the resection site, and the resection target (resection size), and uses the scalpel. The operation information D2 is configured as the operation in the past operation (an example of a medical act) judged to be appropriate by such a doctor replaced by the movement of the robotic arm 230 (it may also be the operation information to the operation unit 110 of the past robotic arm operation system 1). The operation information D2 is preferably constructed through image analysis by camera shooting, but is not limited thereto, and may be constructed based on a sensor signal by, for example, having a sensor built into the medical device itself. Then, such operation information D2 and state information D1 are set to construct the template data.

[0023] Similarly, the memory unit 120 also stores, for the reference data "2" and subsequent ones, the state information D1 and the operation information D2 in correspondence, and stores the details of each of the information D1 and D2. Further, although not shown in the figure, the memory unit 120 also stores reference data and the like regarding the operation following the act of "first inserting the scalpel into the affected part", which is "the next act after first inserting the scalpel into the affected part" (an example of a predetermined act). That is, in the example shown in FIG. 3, the memory unit 120 stores a large number of reference data for each operation until the operation is completed (that is, stores a large number of reference data in time series).

[0024] Regarding the reference data, even if a large amount of it is generated by the use of AI (artificial intelligence) from one piece of reference data while the items D11 to D13 and the components D21 to D24 are gradually changed.

[0025] FIG. 4 is a reference diagram showing another example of the reference data stored in the memory unit 120 shown in FIG. 2. The large number of reference data shown in FIG. 4 relates to the act of "inserting the injection needle" (an example of a predetermined act) indicating the initial movement when performing vaccination or drug administration with a syringe. The memory unit 120 stores a large number of such reference data (in the example shown in FIG. 4, m or more: m is a natural number of 2 or more).

[0026] In the example shown in FIG. 4, the medical device is a syringe. The state information D1 is composed of the "injection site" of the first item D11, the "injection position" of the second item D12 indicating the details of the injection site, and the "skin tension" of the third item D13. Specifically, in the reference data "1", in the state information D1, the data of "left arm" is stored in the "injection site" which is the first item D11, the data of "inside the elbow" is stored in the "injection position" which is the second item D12, and the data of "strong" is stored in the "skin tension" which is the third item D13. That is, it is information such that the inside of the elbow of the left arm is swollen and the skin tension is strong. Note that the "skin tension" of the third item D13 and the like may be expressed by numerical values.

[0027] Regarding such state information D1, similar to the above, it may also be calculated and constructed by a computer based on imaging image data near the injection site, test values, etc. However, if possible, it is preferably constructed from the content judged and input by a doctor in advance, or adjusted by adding a correction value by the doctor to the data calculated by the computer and then constructed.

[0028] The operation information D2 is composed of the "insertion position of the needle tip" of the first component D21, the "insertion speed" of the second component D22 indicating the details of the injection site, the "injection needle angle with respect to the affected part" of the third component D23, the "depth of the needle" of the fourth component D24, and the "direction of the needle tip (direction of the cut surface)" of the fifth component D25. Although not shown in FIG. 4, it is preferable that the components include data on the detailed movement of the fingertip and data on the pressure of the fingertip.

[0029] Specifically, in the example shown in FIG. 4, regarding the operation information D2 of the model data "1", the data of "ulnar cutaneous vein (position ◎ cm from the ulnar median cutaneous vein to the shoulder side)" is stored in the "insertion position of the needle tip", which is the first component D21, the data of "〇〇〇〇" is stored in the "insertion speed", which is the second component D22, the data of "△△△△" is stored in the "injection needle angle with respect to the affected part", which is the third component D23, the data of "◇◇◇" is stored in the "depth of the needle", which is the fourth component D24, and the data of "◇" is stored in the "direction of the needle tip (direction of the cut surface)", which is the fifth component D25. It is preferable that these pieces of information are also composed of coordinate data, vector data, etc. of three-dimensional coordinates in a state where a specific point on the patient's body is fixed.

[0030] Here, the injection needle has a shape such that its tip is cut obliquely. Generally, it is inserted along the skin with the cut surface at the tip facing away from the skin surface. However, when the injection needle is inserted in this way, it tends to stretch the skin, and for patients whose skin is already swollen and taut, the pain tends to increase. Therefore, it is necessary to adjust the injection angle and the orientation of the cut surface (if the orientation of the cut surface is perpendicular to the skin surface, it can be inserted as if cutting the skin, making it difficult to stretch the skin). Also, it is preferable to adjust the insertion speed and the depth of the needle according to the injection site, the tightness of the skin, etc. The operation information D2 is constituted by replacing the operations at the time of past injections (an example of a medical act) that have been judged to be appropriate by such a doctor with the movements of the robot arm 230 (operation information is also acceptable in the same way as above). Similar to the above, the operation information D2 is preferably constructed through image analysis by camera shooting, but it is not limited to this, and it is preferably constructed based on sensor signals by incorporating sensors in the medical device itself, etc. Then, such operation information D2 and state information D1 are set to construct template data.

[0031] Furthermore, similar to when described with reference to FIG. 3, the storage unit 120 also stores the state information D1 and the operation information D2 in correspondence for the template data after "2" and stores the details of each of the information D1 and D2. Furthermore, although not shown in the figure, the storage unit 120 also stores template data and the like regarding "the next action after inserting the injection needle" (an example of a predetermined action), which is the action following the "action of inserting the injection needle". That is, also in the example shown in FIG. 4, the storage unit 120 stores a large number of template data for each action until the injection is completed in chronological order. Furthermore, the template data stored in the storage unit 120 may be generated in large quantities using AI or the like from one piece of template data.

[0032] Referring again to FIG. 2, the input unit 130 of the operating-side device 100 inputs information regarding the current state of the patient's body part. Here, the input unit 130 may input by analyzing the video of the patient's body part captured by the camera 220 of the robot arm device 200, or the doctor as the operator may input through a keyboard or the like from the video displayed through the monitor 170 from the camera 220. Further, the information regarding the current state of the patient's body part may be in a form of reading the information that has been previously input and stored in the storage unit 120 by a doctor or the like on the robot arm device 200 side. Also, the information regarding the current state of the patient's body part may be information transmitted from outside the system 1. Furthermore, combinations of these may also be possible.

[0033] The extraction unit 140 extracts model data corresponding to the current state of the patient's body part based on the state information D1 from among a number of model data stored in the storage unit 120. This extraction unit 140 extracts model data based on the information regarding the current state of the patient's body part input by the input unit 130. At this time, the extraction unit 140 performs extraction using one or a plurality of items D11 to D13 as extraction criteria.

[0034] An example of extraction will be described with reference to FIG. 4. For example, when the extraction criterion is one of the "injection site" of the first item D11, and the current patient requires an injection in the "left arm", the extraction unit 140 extracts all of the reference data with the injection site being the left arm. Also, when the extraction criterion is two of the "injection site" and "injection position" of the first and second items D11 and D12, and the current patient requires an injection in the "left arm" and "the back of the elbow", the extraction unit 140 extracts all of the reference data with the injection site being the left arm and the injection position being the back of the elbow. Similarly, when the extraction criterion is one of the "injection position" of the second item D12, and the current patient requires an injection in the "back of the elbow", the extraction unit 140 extracts all of those reference data as long as the injection position is the back of the elbow regardless of whether the injection site is the left arm or the right arm. Also, when the extraction criterion is three of the "injection site", "injection position", and "skin tension" of the first to third items D11 to D13, and the skin tension of the current patient is "strong" and requires an injection in the "left arm" and "the back of the elbow", the extraction unit 140 extracts all of the reference data that meet all of these conditions. In this way, the extraction unit 140 uses part or all of the state information D1 as the extraction criterion, and extracts the reference data in which the detailed information within the extraction criterion matches the state of the body part of the current patient. Note that the extraction unit 140 may use the body part of the current patient as simply "arm" instead of "left arm", and extract the reference data of both the right arm and the left arm. Also, regarding how to set the extraction criterion, it may be selectable by a doctor, or may be preset.

[0035] Here, regarding the state information D1 of the reference data shown in FIG. 3, the "resection size" includes a numerical value. Regarding such a numerical value, a numerical range may be set, and the reference data within the numerical range may be extracted as matching. Also, since the "resection size" includes information on the shape, the reference data may be extracted based only on the shape. That is, the reference data may be extracted based on the match of only a further part of the items D11 to D13.

[0036] In addition, although not shown in the figure, the storage unit 120 stores patient status information such as the patient's gender, age, and occupation for the template data, and in addition to the status information D1, it may extract template data for which the patient status information matches.

[0037] As described above, the extraction unit 140 extracts template data corresponding to the current state of the patient's body part.

[0038] The generation unit 150 generates imitation data including imitation information indicating the operation to be imitated by the robot arm 230 based on the operation information D2 of the template data extracted by the extraction unit 140. The imitation data is the standard for the robot arm device 200 to imitate in this medical procedure. The generation unit 150 generates imitation data by, for example, the following four methods.

[0039] First, the generation unit 150 calculates representative values (absolute values, three-dimensional coordinate values, three-dimensional coordinate ranges, three-dimensional vectors, etc.) from the operation information D2 of all (multiple) extracted template data to generate imitation data having imitation information. For example, it is assumed that the template data "1" to "4" in FIG. 3 is extracted by the extraction unit 140. In this case, the generation unit 150 calculates representative values such as the average value or median value of "〇〇〇〇", "〇〇〇〇", "〇〇〇×", and "〇〇〇■" for the "scalpel speed", which is the first component D21. The generation unit 150 similarly calculates representative values for the "scalpel angle with respect to the affected part", "scalpel movement direction", and "scalpel depth", which are the second to fourth components D22 to D24. Note that for the "scalpel movement direction", the average of vectors obtained from three-dimensional coordinate values is used as the representative value. Thereby, the generation unit 150 generates imitation data including imitation information.

[0040] Second, the generation unit 150 may calculate a representative value based on a predetermined number of pieces of operation information D2 from among all (plural) of the extracted reference data having similar state information D1, and generate imitation data. Here, those having similar state information D1 are, for example, for the resection size shown in FIG. 3, since numerical values are set, those with close numerical values are regarded as such. On the other hand, as in the case of the "left arm", "inside of the elbow", and "strong" of the first to third items D11 to D13 shown in FIG. 4, when there are a large number of those that all match and exceed a predetermined number, based on information other than the state information D1, for example, the patient's status information, imitation data may be generated from those with a closer status. Note that by performing generation based on this predetermined number, reference data including outliers can be excluded.

[0041] Third, also, when the state information D1 of two or more pieces of extracted reference data can be made to match or approximate the state of the current patient's body part by weighting and adding the two or more pieces of reference data, the generation unit 150 may generate imitation data by weighting. To give an example, for instance, assume that for the state of the current patient's body part, the "affected part" is the "stomach", the "resection site" is the "lower part of the stomach", and the "resection size" is "circular with a diameter of ■ cm (exposed × cm)". Here, assume that ■ is a numerical value such that ■ = 1 / 3 × 〇 + 2 / 3 × ◎. In this case, the generation unit 150 weights the reference data "1" shown in FIG. 3 with "1 / 3", weights the reference data "2" with "2 / 3", and adds them, thereby obtaining data where the "resection size" is "circular with a diameter of ■ cm (exposed × cm)". That is, from the state information D1 of the reference data "1" and "2", the same state as the state of the current patient's body part (or an approximate state within a predetermined allowable value) can be obtained. Therefore, the generation unit 150 weights the operation information D2 of the reference data "1" with "1 / 3", weights the operation information D2 of the reference data "2" with "2 / 3", and adds them, thereby generating imitation data including imitation information (absolute values, three-dimensional coordinate values, three-dimensional coordinate ranges, and three-dimensional vectors, etc.).

[0042] Fourthly, the generation unit 150 may generate imitation data that has the same state as the state of the current patient's body part (even an approximate state within a predetermined allowable value) by, for example, using AI or the like to gradually change conditions from one template data. The fourth method is effective when only one piece of template data is extracted by the extraction unit 140.

[0043] In addition, the generation unit 150 may generate imitation data by combining two or more of the first to fourth methods. Furthermore, status information of the affected part may be considered in the generation.

[0044] The arm control unit 160 operates the robot arm 230 according to the operation of the operation unit 110. First, the arm control unit 160 receives an operation signal corresponding to the operation of the operation unit 110, executes assist processing on the received operation signal, and transmits a control signal, which is the operation signal after the assist processing, to the robot arm device 200. As a result, the arm operation mechanism unit 210 of the robot arm device 200 realizes the operation of the robot arm 230 including the assist content (there may be a case where no assist content is included as a result of the assist processing) based on the control signal.

[0045] First, the arm control unit 160 determines whether it is a predetermined action based on the operation signal. The arm control unit 160 determines whether it is a predetermined action based on the information in the storage unit 120. For example, when the operation of the robot arm 230 is only to bring the scalpel close to the patient's body, the arm control unit 160 determines that it is not a predetermined action. In this case, the arm control unit 160 uses the operation signal as the control signal as it is and transmits the control signal to the robot arm device 200.

[0046] On the other hand, when the scalpel reaches the body based on the operation signal as in the storage unit 120, the arm control unit 160 determines that it corresponds to the action of "first inserting the scalpel into the affected part" and determines that it is a predetermined action.

[0047] Furthermore, when it is a predetermined action, the arm control unit 160 reads the imitation data generated by the generation unit 150 for the current action. Next, the arm control unit 160 makes the behavior of the robot arm 230 based on the operation signal approach the operation of the robot arm 230 indicated by the imitation information of the imitation data.

[0048] Specifically, regarding the action of "first inserting the scalpel into the affected part", the arm control unit 160 compares the "scalpel speed" of the first component D21 indicated by the operation signal, the "scalpel angle with respect to the affected part" of the second component D22, the "scalpel movement direction" of the third component D23, and the "scalpel depth" of the fourth component D24 with the imitation information for each of the components D21 to D24. As a result of the comparison, if there are components D21 to D24 that deviate by more than a predetermined amount (for example, the coordinate value or vector quantity is a predetermined value), the arm control unit 160 assists the operation signal regarding those components D21 to D24 to generate a control signal. For example, when the "scalpel speed" of the first component D21 indicated by the operation signal is significantly different from the imitation information, but the other components D22 to D24 are substantially the same, only the "scalpel speed" is assisted. Note that the predetermined amount is set in advance for each of the components D21 to D24.

[0049] When performing the assist, the arm control unit 160 implements one of the following methods. For example, the arm control unit 160 changes the components D21 to D25 that deviate from the imitation information by more than a predetermined amount to the same values as the imitation information. Also, the arm control unit 160 may impose a limit so that the value becomes an allowable value set within a predetermined range from the value indicated by the imitation information. For the former, since it becomes the same value as the imitation information, it becomes a form of tracing the imitation information more. On the other hand, for the operator, the sense of discomfort increases. For the latter, the sense of discomfort is less than that of the former, and assistance can be performed. Note that the allowable value in the above may be a predetermined range or a variable value.

[0050] In addition, the assist is not limited to the above. For example, various methods can be adopted, such as being implemented as an intermediate value between the imitation information and the value indicated by the operation signal. Further, it is not limited to the case where each of the components D21 to D25 is assisted. Even if only one of the components D21 to D25 is the assist target, the values indicated by the imitation information may be approximated for the other components D21 to D25.

[0051] As described above, the arm control unit 160 performs assist processing based on the operation signal and transmits a control signal to the robot arm device 200.

[0052] Thereby, the arm operation mechanism unit 210 of the robot arm device 200 operates the robot arm 230 based on the control signal. And the operation is photographed by the camera 220 provided in the robot arm device 200 and is displayed on the monitor 170 of the operation side device 100.

[0053] Next, a flowchart showing the processing of the robot arm operation system 1 according to the present embodiment will be described. FIG. 5 is a flowchart showing the processing of the operation side device 100 of the robot arm operation system 1 according to the present embodiment. It is assumed that the video photographed by the camera 220 of the robot arm device 200 continues to be displayed on the monitor 170 of the operation side device 100 during the execution of this processing.

[0054] First, the operation side device 100 determines whether information on the state of the body part of the current patient has been input through the input unit 130 (S1). If not input (S1: NO), this process is repeated until it is determined that the input has been made.

[0055] When the information on the state of the body part is input (S1: YES), the extraction unit 140 extracts template data from the storage unit 120 (S2). In this process, the extraction unit 140 compares the information input in step S1 with the state information D1 of the template data stored in the storage unit 120 to determine whether to extract. The extraction here is performed for each predetermined action.

[0056] After that, the generation unit 150 generates imitation data based on the sample data extracted in step S2 (S3). In this process, the generation unit 150 generates imitation information serving as a reference to be imitated from the operation information D2 of the sample data. This imitation information is generated for each predetermined action. That is, in the present embodiment, the imitation data includes at least action information and imitation information. Further, here, the generated imitation data is generated for the part until the medical action is completed.

[0057] Next, the arm control unit 160 determines whether there is an operation via the operation unit 110 (S4). If there is no operation (S4: NO), the process proceeds to step S9.

[0058] On the other hand, if there is an operation (S4: YES), the arm control unit 160 determines whether the operation corresponds to a predetermined action (target action) (S5). If the operation does not correspond to the target action (S5: NO), the process proceeds to step S8.

[0059] On the other hand, if the operation corresponds to the target action (S5: YES), the arm control unit 160 compares each component D21 to D25 of the operation with each component D21 to D25 constituting the imitation information, and determines whether there are components D21 to D25 that are separated by a predetermined amount or more (S6).

[0060] If there are no components D21 to D25 that are separated by a predetermined amount or more (S6: NO), the process proceeds to step S8. On the other hand, if there are components D21 to D25 that are separated by a predetermined amount or more (S6: YES), the arm control unit 160 sets a restriction on those components D21 to D25 (S7). In this process, the arm control unit 160 restricts the values shown by the imitation information to be within an allowable value set within a predetermined range.

[0061] After that, the arm control unit 160 generates a control signal and transmits it to the robot arm device 200 (S8). As a result, the arm operation mechanism unit 210 of the robot arm device 200 operates the robot arm 230 based on the control signal.

[0062] After that, the operating-side device 100 determines whether the medical procedure has ended, that is, if it is a surgery, whether the surgery has been completed (S9). In this process, the operating-side device 100 determines the end, for example, when a predetermined operation is performed on the operation unit 110. However, not limited to this, the operating-side device 100 may automatically determine the end based on the image transmitted from the camera 220 of the robotic arm device 200. In this case, for example, it is determined by the fact that the transmitted image has become dark (that is, the electricity in the operating room has been turned off), the use of a suture needle or a needle holder, and then placing these medical instruments, etc.

[0063] If it is determined that the medical procedure has not ended (S9: NO), the process proceeds to step S4. On the other hand, if it is determined that the medical procedure has ended (S9: YES), the process shown in FIG. 5 ends.

[0064] Note that in the above description, the robotic arm operation system 1 is such that the operating-side device 100 includes the generation unit 150 and performs assist processing based on the imitation data, but it is not particularly limited to this. For example, the robotic arm operation system 1 may extract only one piece of reference data determined to be optimal by the extraction unit 140, and the arm control unit 160 may transmit a control signal to the robotic arm device 200 so as to approximate the behavior of the robotic arm 230 to the operation indicated by the operation information D2 of the one piece of reference data extracted.

[0065] In this way, according to the robot arm operation system 1 and its control method according to this embodiment, a large number of model data in which the state information D1 indicating the state of the body part is associated with the operation information D2 related to the operation of the robot arm are stored, the model data corresponding to the current state of the body part is extracted, and the behavior of the robot arm 230 according to the operation is made to approach the behavior of the robot arm 230 indicated by the imitation information of the imitation data (the operation information D2 of the model data is also acceptable). For this reason, for example, when the operator inserts a scalpel into the affected part at an incorrect speed or inserts the injection needle at an incorrect angle, etc., this will be corrected based on the model data. Therefore, it is possible to provide a robot arm operation system 1 that can obtain appropriate assistance when performing a medical act with a high degree of difficulty while checking the video from a remote location.

[0066] Also, regarding the behavior of the robot arm 230 at a predetermined time, in order to make it approach the behavior of the robot arm 230 indicated by the operation information D2 of the model data, for example, when moving the scalpel close to the body part or turning it upward to remove air from the syringe, etc., for the behavior of the robot arm 230 at the timing when no medical instrument is used for the body part, the discomfort that assistance always occurs regarding the behavior of the robot arm 230 without approaching the imitation information of the imitation data (the operation information D2 of the model data is also acceptable) can be reduced.

[0067] Also, imitation data is generated based on the extracted model data, and the behavior of the robot arm 230 according to the operation is made to approach the behavior of the robot arm 230 indicated by the imitation information of the generated imitation data. For this reason, for example, imitation data having imitation information that adopts the average movement of two or more pieces of extracted model data can be generated, and by making the behavior of the robot arm 230 approach this, it is possible to provide a robot arm operation system 1 that can obtain more appropriate assistance.

[0068] Also, when components D21 to D25 included in the imitation information of the imitation data are separated from the operation of the robotic arm 230 indicated by the imitation information by a predetermined amount or more, the behavior of the robotic arm 230 is brought closer to the operation of the robotic arm 230 indicated by the imitation information of the imitation data. For this reason, for example, when there are a number of components D21 to D25 such as the speed when inserting the scalpel into the body part, the inclination of the scalpel, the direction of inserting the scalpel, and the depth of inserting the scalpel, if the direction of inserting the scalpel is separated by a predetermined amount (predetermined angle) or more, there is a possibility of incorrect movement. Therefore, it is brought closer to the operation of the robotic arm 230 indicated by the imitation information of the imitation data. As a result, only the greatly incorrect components D21 to D25 are assisted, and the components D21 to D25 that are close to the operation of the robotic arm 230 indicated by the imitation information to a certain extent will move as intended by the operator. Therefore, it is possible to provide the robotic arm operation system 1 capable of obtaining more appropriate assistance.

[0069] Also, for components D21 to D25 that are separated by a predetermined amount or more, in order to provide a limit so that the allowable value is set within a predetermined range from the value indicated by the imitation information, for example, the direction of inserting the scalpel is restricted to the allowable value, and it is possible to perform assistance with less discomfort without approaching the operation of the robotic arm 230 indicated by the imitation information more than necessary.

[0070] As described above, the present invention has been described based on the embodiments. However, the present invention is not limited to the above embodiments, and modifications may be made or known or well-known technologies may be combined without departing from the spirit of the present invention.

[0071] For example, in the above embodiment, a part of the configuration such as the arm control unit 160 may be included in the robotic arm device 200, and the processing may be executed through communication.

[0072] Furthermore, in the above embodiment, the arm control unit 160 makes the behavior of the robot arm 230 according to the operation of the operation unit 110 approach the operation indicated by the imitation information (operation information D2 of the model data) for a predetermined behavior. However, not limited to the predetermined behavior, the imitation information (operation information D2 of the model data) may be made to approach the operation indicated for all operation behaviors.

[0073] Also, even if there are no components D21 to D25 that are separated by a predetermined amount or more between the behavior of the robot arm 230 indicated by the operation signal and the operation of the robot arm 230 indicated by the imitation information, the arm control unit 160 may assist. That is, the arm control unit 160 may make the behavior of the robot arm 230 indicated by the operation signal approach the operation indicated by the imitation information regardless of the presence or absence of the components D21 to D25 that are separated by a predetermined amount or more.

[0074] Furthermore, although not described in the above embodiment, the generation unit 150 preferably adjusts the imitation data according to the current state of the medical procedure. For example, in the act of "first inserting the scalpel into the affected area", it is assumed that the cut is deeper than the optimal value indicated by the imitation data assisted for the "scalpel depth". In this case, the generation unit 150 adjusts the imitation data for the next act, such as inserting the scalpel next from a shallower position, and so on. Furthermore, additional processing may be executed, such as displaying a message to pay attention to hemostasis due to cutting too deep on the monitor 170, or adjusting the imitation data regarding suturing in the latter half of the surgery. Furthermore, if the cut is too shallow, since it is necessary to cut to near the optimal value once in the next act and then perform the next act, an act of cutting to near the optimal value may be added as the target act. At this time, the generation unit 150 may generate imitation data in the same manner as described above.

[0075] Furthermore, the storage unit 120 may store the entire medical procedure as a series of data without making distinctions such as "the act of first inserting the scalpel into the affected area." In this case, it is preferable for the system 1 to be able to recognize which action is being performed at an intermediate point in the series of data based on criteria such as how each medical instrument contacts the patient.

[0076] Furthermore, in a medical procedure, when a certain action or the like is performed, it may be beneficial for certain matters but disadvantageous for others. For example, when the depth of the scalpel indicated by the imitation data reaches a deep value, the risk of bleeding may increase. Also, in order to completely remove cancer cells, if the resection is made larger, it may become difficult to maintain the appropriate size of the body part. In such cases, it is preferable for the imitation data to be adjusted by the automatic negotiation AI technology. This is because the implementation of the automatic negotiation AI technology can lead to future automated surgeries. In particular, when using a laser scalpel instead of a physical scalpel, since the resection depth can be adjusted by the energy amount, it is possible to lead to the automation of the resection act by laser irradiation.

Explanation of Signs

[0077] 1: Robot arm operation system 100: Operating side device 110: Operating unit (operating means) 120: Storage unit (storage means) 130: Input unit 140: Extraction unit (extraction means) 150: Generation unit (generation means) 160: Arm control unit (control means) 170: Monitor 200: Robot arm device 210: Arm movement mechanism unit 220: Camera 230: Robot arm D1: State information D2: Movement information D21~D25: Components

Claims

1. A robot arm operation system for performing a medical act using a medical instrument from a remote location by operating a robot arm having the medical instrument in response to an operation of an operator at a remote location, a storage means storing a large number of reference data in which state information indicating the state of a patient's body part using the medical instrument is associated with operation information regarding the operation of the robot arm according to the state; an extraction means for extracting the reference data according to the state of the current patient's body part based on the state information among the large number of reference data stored by the storage means; an operation means operated by the operator; a control means for operating the robot arm based on an operation performed on the operation means, and the control means approximates the behavior of the robot arm according to the operation performed on the operation means to the behavior of the robot arm indicated by the operation information of the reference data extracted by the extraction means. A robot arm operation system characterized by the above.

2. The control means approximates the behavior of the robot arm according to the operation performed on the operation means at a predetermined time to the behavior of the robot arm indicated by the operation information of the reference data extracted by the extraction means. The robot arm operation system according to claim 1, characterized by the above.

3. Further comprising a generation means for generating imitation data including imitation information indicating an operation to be imitated based on the operation information of the reference data extracted by the extraction means, and the control means approximates the behavior of the robot arm according to the operation performed on the operation means to the behavior of the robot arm indicated by the imitation information of the imitation data generated by the generation means. The robot arm operation system according to claim 1, characterized by the above.

4. When the behavior of the robot arm according to the operation performed on the operation means includes a component that is separated from the behavior of the robot arm indicated by the imitation information of the imitation data generated by the generation means by a predetermined amount or more, with respect to the component, the control means approximates the behavior of the robot arm to the behavior of the robot arm indicated by the imitation information of the imitation data. The robot arm operation system according to claim 3, characterized by the above.

5. When making the behavior of the robotic arm corresponding to the operation performed on the operation means approach the operation of the robotic arm indicated by the imitation information of the imitation data, for components separated by more than the predetermined amount, it is restricted so as to be an allowable value set within a predetermined range from the value indicated by the imitation information. The robotic arm operation system according to claim 4, characterized in that.

6. A control method for a robotic arm operation system including a storage means that stores a large number of reference data in which state information indicating the state of a patient's body part using a medical device and operation information regarding the operation of the robotic arm corresponding to the state are associated with each other by operating a robotic arm having a medical device in response to an operation by an operator at a remote location, and performing a medical act using the medical device from the remote location, An extraction step of extracting the reference data corresponding to the state of the current patient's body part based on the state information among the large number of reference data stored by the storage means, An operation step operated by the operator, A control step of operating the robotic arm according to the operation performed in the operation step, and In the control step, the behavior of the robotic arm according to the operation performed in the operation step is made to approach the operation of the robotic arm indicated by the operation information of the reference data extracted in the extraction step A control method for a robotic arm operation system, characterized in that.

Images