Controller for multi-flexibility input device

Abstract

To provide a controller for a multi-flexibility input device which can study inputs by a user and can reflect only intended inputs.SOLUTION: A controller 10 for a multi-flexibility input device has a machine learning unit 11 and a processing unit 12. The machine learning unit 11 carries out, from maximum input axis information obtained by maximum input by an operator for learning to a predetermined axis with using a multiple-flexibility input device 20, machine learning of relationship between input axis information relative to the predetermined axis intended by the operator and noise input axis information relative to an axis other than the predetermined axis not intended by the operator. The machine-learned machine learning unit 11 predicts prediction input axis information relative to the predetermined axis intended by the operator from the input axis information. The processing unit 12 selects whether the input axis information is input to the machine learning unit 11 for learning or for prediction, and outputs the prediction input axis information predicted by the machine-learned machine learning unit 11 as operation information to an operation object in an orthogonal coordinate system by the operator.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a controller for a multi-degree-of-freedom input device, and more particularly to a controller for a multi-degree-of-freedom input device for outputting operation information from the multi-degree-of-freedom input device to an object to be operated in order to obtain input axis information for two or more axes in a Cartesian coordinate system by an operator. [Background technology]

[0002] Joysticks and game pads are known for operating objects such as unmanned aerial vehicles (represented by drones) and robot arms. These multi-degree-of-freedom input devices output operation information in a Cartesian coordinate system by an operator to the object. In general, it is difficult to intuitively operate such multi-degree-of-freedom input devices.

[0003] In view of this, an example of a multi-degree-of-freedom input device that allows intuitive operation is disclosed in Patent Document 1, which is filed by the same applicant as the present application. In this device, the XYZ coordinate center point of the translation mechanism is located in the grip, while the roll / pitch / yaw rotation center point of the rotation mechanism is located in the wrist joint of the operator, enabling intuitive remote operation of an object to be operated with movements similar to those controlled by the musculoskeletal system. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2023-045516 A Summary of the Invention [Problem to be solved by the invention]

[0005] Although the multi-degree-of-freedom input device of the above-mentioned Patent Document 1 allows intuitive operation, the skeletal structure differs depending on the operator, and the movements of the wrist and arm are also individual to each operator, so that depending on the operator, unintended input may be input, and the operation may not be as intended. Therefore, there has been a demand for the development of a multi-degree-of-freedom input device that can reflect only the input intended by each operator.

[0006] In view of the above circumstances, the present invention aims to provide a controller for a multi-degree-of-freedom input device that can learn inputs by an operator and reflect only the intended input. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object of the present invention, the controller for a multi-degree-of-freedom input device according to the present invention includes a machine learning unit that performs machine learning to learn the relationship between input axis information for a specified axis intended by the operator and noise input axis information for an axis other than the specified axis that is not intended by the operator, from maximum input axis information that an operator inputs maximum for learning purposes for a specified axis using the multi-degree-of-freedom input device, and the machine learning unit that has completed the machine learning predicts predicted input axis information for the specified axis intended by the operator from the input axis information input by the operator using the multi-degree-of-freedom input device, and a processing unit that selects whether the input axis information input by the operator using the multi-degree-of-freedom input device is to be input for learning purposes or for prediction purposes to the machine learning unit, and outputs the predicted input axis information predicted by the machine learning unit that has completed the machine learning from the input axis information input for prediction as operation information for an object to be operated by the operator in a Cartesian coordinate system.

[0008] Here, the machine learning unit may further learn by machine learning the relationship between input axis information for a specified axis intended by the operator and noise input axis information for an axis other than the specified axis that is not intended by the operator, from minimum input axis information that the operator inputs for learning purposes using the multi-degree-of-freedom input device for the specified axis.

[0009] In addition, when there is a noise axis having noise input axis information greater than a predetermined amount among the noise input axis information for an axis other than the specified axis that is not intended by the operator, the machine learning unit may treat a composite axis of the specified axis and the noise axis as a new specified axis, and learn by machine learning the relationship between the input axis information for the new specified axis intended by the operator and the noise input axis information for an axis other than the new specified axis that is not intended by the operator, from maximum input axis information that the operator inputs maximum for learning purposes using the multi-degree-of-freedom input device for the new specified axis.

[0010] In addition, when there is a noise axis having noise input axis information greater than a predetermined amount among the noise input axis information for an axis other than the specified axis that is not intended by the operator, the machine learning unit may treat a composite axis of the specified axis and the noise axis as a new specified axis, and learn by machine learning the relationship between the input axis information for the new specified axis intended by the operator and the noise input axis information for an axis other than the new specified axis that is not intended by the operator, from minimum input axis information that the operator inputs for learning purposes using the multi-degree-of-freedom input device for the new specified axis. Effect of the Invention

[0011] The controller for a multi-degree-of-freedom input device of the present invention has the advantage that it can learn inputs by an operator and reflect only the intended inputs. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram for explaining a controller for a multi-degree-of-freedom input device according to the present invention. [Diagram 2] FIG. 2 is a diagram for explaining axes of a multi-degree-of-freedom input device used in a controller for a multi-degree-of-freedom input device of the present invention. [Diagram 3] FIG. 3 is a flowchart for explaining an example of a learning input procedure by the machine learning unit of the controller for a multi-degree-of-freedom input device of the present invention. [Figure 4]FIG. 4 is a graph for explaining input axis information input by an operator using a multi-degree-of-freedom input device for six axes, namely, the X-axis, Y-axis, Z-axis, roll rotation axis, pitch rotation axis, and yaw rotation axis. [Diagram 5] FIG. 5 is a conceptual diagram of a neural network that performs machine learning of a composite axis in the machine learning unit of the controller for a multi-degree-of-freedom input device of the present invention. [Figure 6] FIG. 6 is a schematic perspective view for explaining an example of a multi-degree-of-freedom input device suitable for use with the multi-degree-of-freedom input device controller of the present invention. [Figure 7] FIG. 7 is a graph comparing flight times of the present invention and comparative examples. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the illustrated examples. FIG. 1 is a block diagram for explaining a controller for a multi-degree-of-freedom input device of the present invention. The controller 10 for a multi-degree-of-freedom input device of the present invention is for outputting operation information from a multi-degree-of-freedom input device 20 to an object to be operated in order to obtain input axis information for two or more axes in a Cartesian coordinate system by an operator. The object to be operated may be an unmanned aerial vehicle such as a drone, a robot arm, or the like. The input axis information refers to information on the position of a number line in the X-axis direction from the origin in an XY coordinate system and the position of a number line in the Y-axis direction.

[0014] As shown in the figure, the multi-degree-of-freedom input device controller 10 of the present invention is composed of a machine learning unit 11 and a processing unit 12. The multi-degree-of-freedom input device 20 is an interface for operating an object to be operated, and is not limited to a specific one. The multi-degree-of-freedom input device 20 used in the multi-degree-of-freedom input device controller 10 of the present invention may be, in the simplest case, a device capable of outputting two-axis XY coordinate input axis information such as a game pad such as a joystick. It may also be a haptic interface capable of outputting input axis information for six axes, namely, the X axis, the Y axis, the Z axis, the roll rotation axis, the pitch rotation axis, and the yaw rotation axis.

[0015] The axes of a multi-degree-of-freedom input device used in the multi-degree-of-freedom input device controller of the present invention will be described with reference to Fig. 2. Fig. 2(a) shows an example of an axis of an input device capable of inputting with two degrees of freedom, such as a joystick. Fig. 2(b) shows an example of an axis of an input device capable of inputting with six degrees of freedom, including an X-axis, a Y-axis, a Z-axis, a roll rotation axis, a pitch rotation axis, and a yaw rotation axis. In the following description, the operator will input to the axes of such an orthogonal coordinate system.

[0016] The machine learning unit 11 learns by machine learning the relationship between input axis information for a specific axis intended by the operator and noise input axis information for an axis other than the specific axis that is not intended by the operator. When using the multi-degree-of-freedom input device 20 for input, even if the operator operates it intentionally, input may be made not only to the axis that the operator intended to input, but also to an axis that the operator did not intend. For example, even if the operator intends to input accurately only to the X-axis, there may be cases where the input is unintentionally slightly shifted to the Y-axis. The machine learning unit 11 learns by machine learning the noise input axis information for such an unintended axis.

[0017] For example, a neural network may be used as the machine learning unit 11. More specifically, for example, a Long / Short Term Memory (LSTM) or a Gated Recurrent Unit (GRU) may be used.

[0018] The machine learning unit 11 uses maximum input axis information that the operator inputs maximum for learning purposes to a predetermined axis using the multi-degree-of-freedom input device 20. Specifically, when the controller 10 for a multi-degree-of-freedom input device of the present invention is set to machine learning mode, the operator is prompted to input maximum for learning purposes to a predetermined axis. The operator is prompted to input maximum for each of the X-axis, Y-axis, Z-axis, roll rotation axis, pitch rotation axis, and yaw rotation axis, for example. Note that the maximum input means the positive input limit and negative input limit for each axis of the multi-degree-of-freedom input device 20. The input axis information is time-series data.

[0019] 3 is a flow chart for explaining an example of a learning input procedure by the machine learning unit of the controller for a multi-degree-of-freedom input device of the present invention. As shown in the figure, the machine learning unit 11 in the machine learning mode has the operator input maximum positive input to a predetermined axis, for example, the X-axis. Then, the machine learning unit 11 returns to neutral and has no input. Then, the machine learning unit 11 has the operator input maximum negative input to a predetermined axis, for example, the X-axis. Finally, the machine learning unit 11 returns to neutral and has no input. This is performed multiple times for each input axis, for example, the X-axis, Y-axis, Z-axis, roll rotation axis, pitch rotation axis, and yaw rotation axis, to perform machine learning.

[0020] In the machine learning unit 11, a model may be created for each input axis. Specifically, intended input is labeled as an input state, and unintended input is labeled as a no-input state, and machine learning is performed. For example, in maximum input axis information for maximum input for learning on the X axis, the X value is labeled as an input state, and the Y value, Z value, roll value, pitch value, and yaw value are labeled as no-input states. That is, inputs on axes other than the X axis are learned as noise. Similarly, in maximum input axis information for maximum input for learning on the Y axis, the Y value is labeled as an input state, and the X value, Z value, roll value, pitch value, and yaw value are labeled as no-input states. These may be performed for each input axis of the multi-degree-of-freedom input device 20, and machine learning may be performed.

[0021] The machine learning unit 11 that has undergone machine learning in this way is capable of predicting predicted input axis information for a specific axis intended by the operator from the input axis information input by the operator using the multi-degree-of-freedom input device 20. Here, the predicted input axis information is input axis information that the machine learning unit 11 that has undergone machine learning predicts to be input axis information for a specific axis intended by the operator by removing predicted noise input axis information. In this way, the machine learning unit 11 that has undergone machine learning is capable of receiving input axis information input using the multi-degree-of-freedom input device and predicting input axis information truly intended by the operator.

[0022] Specifically, for example, when the input axis information by the operator includes not only an X-axis input value but also an Y-axis input value, whether the Y-axis input is the input intended by the operator is predicted using the machine learning unit 11 that has undergone machine learning. As a result, when the machine learning unit 11 that has undergone machine learning determines that the Y-axis input is, for example, an input unintended by the operator, the input value of the Y-axis is set to 0 as noise input axis information, and the input axis information predicted to be truly intended by the operator is the input value of the X-axis other than the Y-axis, which is output as predicted input axis information.

[0023] Even if the input axis information is intentionally input to both the X-axis and the Y-axis, there may be cases where an unintended input is input closer to the Y-axis. Even in such a case, the machine learning unit 11 that has already undergone machine learning can predict whether or not the input closer to the Y-axis is an input unintended by the operator. If it is determined that the input is unintended by the operator, the input value closer to the Y-axis is removed as noise input axis information from the input value of the Y-axis, and the input value of the Y-axis is corrected to obtain predicted input axis information that is predicted to be truly intended by the operator.

[0024] The processing unit 12 selects whether the input axis information input by the operator using the multi-degree-of-freedom input device 20 is to be input to the machine learning unit 11 as described above for learning or prediction. Furthermore, the processing unit 12 outputs predicted input axis information predicted by the machine learning unit 11 that has completed machine learning from the input axis information input for prediction as operation information for an object to be operated by the operator in a Cartesian coordinate system. That is, the processing unit 12 selects whether the machine learning unit 11 is in machine learning mode or prediction mode. In the case of the machine learning mode, the processing unit 12 inputs maximum input axis information that is maximum input for learning to the machine learning unit 11 as described above. In the case of the prediction mode, the processing unit 12 obtains and outputs predicted input axis information from the machine learning unit 11.

[0025] In the machine learning mode, the processing unit 12 may convey the instruction for the learning input to the operator via a monitor, voice, etc. Also, a switch or the like may be provided as appropriate so that the machine learning mode and the prediction mode can be switched.

[0026] In this way, in the case of the machine learning mode, the processing unit 12 causes the machine learning unit 11 to perform machine learning, and in the case of the prediction mode, the predicted input axis information predicted by the machine learning unit 11, which has already completed machine learning, to be the operator's true intention is output as operation information to the object to be operated, such as an unmanned aerial vehicle.

[0027] By using the controller 10 for a multi-degree-of-freedom input device of the present invention configured in this way, it is possible to machine-learn inputs by an operator and reflect only the intended inputs. Since machine learning is possible for each operator, it is possible to understand the operator's habits according to the individuality of the operator and output only the truly intended inputs as operation information. Therefore, it is possible to operate the object to be operated without stress.

[0028] The multi-degree-of-freedom input device controller 10 of the present invention may be connected between an existing multi-degree-of-freedom input device 20 and an object to be operated. In the multi-degree-of-freedom input device controller 10 of the present invention, the machine learning unit 11 may have the function of the processing unit 12. Furthermore, the machine learning unit 11 and the processing unit 12 may be realized by a program executable on a computer, a mobile information terminal, or the like, or may be realized by dedicated hardware such as an electronic circuit or an integrated circuit. Also, a storage unit may be provided as necessary to store learning information for each operator. In addition, the multi-degree-of-freedom input device controller 10 of the present invention may be built into the multi-degree-of-freedom input device 20.

[0029] In the illustrated example described above, the machine learning unit 11 performs machine learning using maximum input axis information that is maximum input for learning. Although this can be effective in a simple manner, machine learning using only maximum input axis information may only learn patterns in which the input increases strongly. Therefore, the controller 10 for a multi-degree-of-freedom input device of the present invention may also perform machine learning on weak inputs. By performing machine learning on weak inputs, patterns in which the input increases weakly can also be learned, and it is possible to configure the controller 10 to more accurately reflect only the input intended by the operator. An example in which the minimum input is also performed for learning will be described below.

[0030] The machine learning unit 11 may perform machine learning by having the operator perform not only maximum input for learning but also minimum input. That is, the machine learning unit 11 may further perform machine learning of the relationship between input axis information for a predetermined axis intended by the operator and noise input axis information for an axis other than the predetermined axis that the operator does not intend, from minimum input axis information that the operator inputs minimum for learning for a predetermined axis using the multi-degree-of-freedom input device 20. The input procedure for learning by the machine learning unit 11 may be performed in the same manner as the procedure for performing the maximum input described above. That is, after machine learning of the maximum input in the machine learning mode, the operator is subsequently made to perform minimum input with positive input, for example, for the X-axis. Then, the operator is returned to neutral and no input is performed. Then, the operator is made to perform minimum input with negative input, for example, for the X-axis. Finally, the operator is returned to neutral and no input is performed. This may be performed multiple times for each input axis, for example, the X-axis, Y-axis, Z-axis, roll rotation axis, pitch rotation axis, and yaw rotation axis, to perform machine learning. This makes it possible to perform machine learning not only for the maximum input, but also for the minimum input. Therefore, by performing machine learning on a larger number of patterns, it becomes possible to more accurately predict the input axis information that the operator truly intends.

[0031] However, there may be a noise axis that has noise input axis information larger than a predetermined amount among noise input axis information for an axis not intended by the operator other than a predetermined axis. That is, there may be an axis that has a large interference with a specific axis. FIG. 4 is a graph for explaining input axis information input by an operator to six axes, namely, the X-axis, the Y-axis, the Z-axis, the roll rotation axis, the pitch rotation axis, and the yaw rotation axis, using the multi-degree-of-freedom input device 20. The upper part is the maximum input axis information, and the lower part is the minimum input axis information. The vertical axis represents the magnitude of the input value. The magnitude of the input value of the input axis information for each axis is assigned to the magnitude of the input value of the noise input axis information for each axis other than each axis. As can be seen from FIG. 4, in the case of this operator, for example, when focusing on the Y-axis of the maximum input axis information, it can be seen that although the input should have been made only to the Y-axis, a large input also appears on the yaw rotation axis due to interference. Similarly, it can be seen that a large input appears on the pitch rotation axis due to interference with the input to the Z-axis. It can also be seen that the Y-axis of the minimum input axis information interferes with the yaw rotation axis. That is, when the specified axis is the Y axis, the noise axis becomes the yaw rotation axis, and when the specified axis is the Z axis, the noise axis becomes the pitch rotation axis.

[0032] In this way, when there is a noise axis having noise input axis information larger than a predetermined amount, the composite axis of the predetermined axis and the noise axis may be machine-learned again by the machine learning unit 11 as a new predetermined axis. That is, for example, maximum input and minimum input for learning may be performed simultaneously for two axes, the Y axis and the yaw rotation axis. More specifically, for example, in a state where the maximum input is performed for the Y axis, the maximum input for learning may be performed for the yaw rotation axis. Using these input axis information, the relationship between the input axis information for the composite axis of the Y axis and the yaw rotation axis and the noise input axis information for an axis not intended by the operator other than the composite axis of the Y axis and the yaw rotation axis may be machine-learned. In this way, it is possible to perform machine learning for even more patterns. Note that, for the input for learning, not only the maximum input but also the minimum input may be combined as described above even for the composite axis.

[0033] FIG. 5 shows a conceptual diagram of a neural network for machine learning a composite axis in a machine learning unit of a controller for a multi-degree-of-freedom input device of the present invention. As shown in the figure, in the input layer, for example, input axis information for the Y axis and the yaw rotation axis is received. In addition, in the intermediate layer, noise input axis information is machine learned. For example, in the input axis information input for learning for the composite axis of the Y axis and the yaw rotation axis, the value of the composite axis may be labeled as an input state, and values ​​other than the composite axis may be labeled as a no-input state. Then, in the output layer, using the machine learning unit 11 that has already been trained, predicted input axis information for a specific axis intended by the operator is predicted from the input axis information input by the operator using the multi-degree-of-freedom input device 20. For example, whether the input of the yaw rotation axis is an input intended by the operator is predicted using the machine learning unit 11 that has already been trained, and if it is determined to be an unintended input, the input value of this yaw rotation axis is removed from the input value of the Y axis as noise input axis information, and the predicted input axis information is predicted to be truly intended by the operator.

[0034] It is not necessary to perform machine learning of the composite axis for all operators. That is, machine learning of the composite axis may be additionally performed only when the noise input axis information is large due to the individuality of the operator. For example, in the machine learning mode, the operator first inputs each axis for learning, and obtains noise input axis information. If all noise input axis information is smaller than a predetermined amount, the machine learning mode may be terminated at this stage. On the other hand, if there is noise input axis information larger than a predetermined amount among the noise input axis information, the composite axis of the predetermined axis and the noise axis may be further input for learning. In this way, if the noise input axis information is small due to the individuality of the operator, it is possible to shorten the learning time.

[0035] In the controller for a multi-degree-of-freedom input device of the present invention, even more input axis information may be used to train the machine learning unit 11. However, if many patterns are input for learning, the learning time required for the machine learning unit 11 increases accordingly, so it is only necessary to use a combination of various types of input axis information for learning in consideration of accuracy and convenience.

[0036] The multi-degree-of-freedom input device controller of the present invention is particularly useful for multi-degree-of-freedom input devices such as haptic interfaces that can accurately reflect wrist movements. FIG. 6 is a schematic perspective view for explaining an example of a multi-degree-of-freedom input device suitable for using the multi-degree-of-freedom input device controller of the present invention. In the figure, parts with the same reference numerals as in FIG. 1 represent the same objects. The multi-degree-of-freedom input device 20 shown in the figure is similar to the device of Patent Document 1 by the same applicant as the present application. The multi-degree-of-freedom input device 20 is composed of a grip unit 21, a translation mechanism unit 22, a translation motion detection unit 23, a rotation mechanism unit 24, a rotation motion detection unit 25, and a control unit 26.

[0037] The grip portion 21 is for an operator to hold. The grip portion 21 may have a shape that allows it to be held in the palm of the operator's hand, as in the illustrated example.

[0038] The translation mechanism 22 is connected to the grip unit 21 and configured so that the center point of the XYZ coordinates is located on the grip unit 21. The translation mechanism 22 may perform translational motion in accordance with the translational motion of the operator's fist, centered on the center point of the XYZ coordinates.

[0039] The translational motion detection unit 23 detects the amount of translational motion of the translational mechanism 22. The translational motion detection unit 23 detects the amount of translational motion caused by the wrist movement of the operator on the grip unit 21. That is, the translational motion detection unit 23 detects the amount and speed of movement of the translational mechanism 22 in parallel along the X-axis, Y-axis, and Z-axis.

[0040] The rotation mechanism 24 is connected to the grip 21 and configured so that the roll / pitch / yaw rotation center point is located at a position different from the XYZ coordinate center point and offset from the XYZ coordinate center point toward the wrist joint of the operator. The rotation mechanism 24 may rotate around the roll / pitch / yaw rotation center point in accordance with the rotational movement of the wrist joint of the operator.

[0041] The rotational motion detection unit 25 detects the amount of rotational motion of the rotation mechanism unit 24. The rotational motion detection unit 25 detects the amount of rotational motion caused by the wrist movement of the operator on the grip unit 21. In other words, the rotational motion detection unit 25 detects the amount and speed of rotation of the rotation mechanism unit 24 around the roll rotation axis, pitch rotation axis, and yaw rotation axis.

[0042] The control unit 26 uses the translational motion amount detected by the translational motion detection unit 23 and the rotational motion amount detected by the rotational motion detection unit 25 to output input axis information for the six axes.

[0043] In the controller 10 for a multi-degree-of-freedom input device of the present invention, the relationship between the input axis information for a predetermined axis and noise input axis information for an axis other than the predetermined axis that is not intended by the operator can be machine-learned by the machine learning unit 11 using the input axis information thus input by the multi-degree-of-freedom input device 20. Then, using the trained machine learning unit 11, predicted input axis information for the predetermined axis intended by the operator can be predicted, and output as operation information to the operation target object.

[0044] The controller for a multi-degree-of-freedom input device of the present invention using the multi-degree-of-freedom input device 20 shown in Fig. 6 will be described in more detail below. Note that the numerical values ​​exemplified below are merely examples, and the present invention is not limited to these.

[0045] The translation mechanism 22 of the multi-degree-of-freedom input device 20 has a linear slide structure and follows three axes (X-axis, Y-axis, and Z-axis) of an orthogonal coordinate system with the operator's grip 21 as the minus point. The range of motion in each axis direction is from -25 mm to +25 mm. In addition, the multi-degree-of-freedom input device 20 is equipped with a motor, and it is possible to apply a tension of up to 12.5 N in each axis direction.

[0046] The rotation mechanism unit 24 of the multi-degree-of-freedom input device 20 follows the directions of the rotation axes (roll rotation axis, pitch rotation axis, and yaw rotation axis) around three axes of an orthogonal coordinate system centered on the operator's wrist. The range of motion of each axis is from -50 degrees to +50 degrees. Each axis is also equipped with two torsion springs to return to a neutral position. The load spring constant of the torsion spring is 0.067 N / deg for the roll rotation axis and yaw rotation axis, and 0.064 N / deg for the pitch rotation axis.

[0047] The input values ​​of each axis obtained from the multi-degree-of-freedom input device 20 are normalized to fall within the range of -1 to +1 before use. The resolution is 0.02 mm for translational movement and 0.02 deg for rotational movement.

[0048] In the machine learning mode, the machine learning unit 11 uses the multi-degree-of-freedom input device 20 as described above to obtain maximum input axis information and minimum input axis information for the six axes, namely the X-axis, Y-axis, Z-axis, roll rotation axis, pitch rotation axis, and yaw rotation axis, for learning. An example of the obtained input axis information is shown in FIG. 4 above. In the case where the Y-axis and yaw rotation axis, and the Z-axis and pitch rotation axis interfere with each other as in this example, input axis information for learning can be further obtained for these composite axes.

[0049] Specifically, an input is made to the Y axis, and then, while maintaining that position, an input is made to the yaw rotation axis. The input to the Y axis is set to four positions: maximum positive input, minimum positive input, minimum negative input, and maximum negative input. At these four input positions, two types of input, maximum input and minimum input, are made to the yaw rotation axis. Similarly, at the four input positions for the Z axis, two types of input are made to the pitch rotation axis. In this way, machine learning is performed on compound axes as well.

[0050] A learning model is created for each axis. As mentioned above, the X-axis, Y-axis, Z-axis, and roll rotation axis do not have a significant impact on the other axes, so only the input values ​​of each axis are used as explanatory variables. On the other hand, for the yaw rotation axis and pitch rotation axis, interference is observed with the input of the Y-axis and Z-axis, respectively. Therefore, the learning model for the yaw rotation axis uses the input value of the compound axis of the Y-axis and yaw rotation axis as an explanatory variable, and the learning model for the pitch rotation axis uses the input value of the compound axis of the Z-axis and pitch rotation axis as an explanatory variable. Additionally, the objective variable for each axis in the learning model is the presence or absence of input for that axis.

[0051] The learning model uses neural network algorithms LSTM and GRU. Two types of neural networks, LSTM and GRU, are tried and the one with the highest score is selected. The neural networks LSTM and GRU use sequence data as input, so the learning model predicts the presence or absence of input at time t from n input values ​​from time t-n+1 to time t. Here, n represents the length of the sequence data and is adjusted as a hyperparameter. The sequence data used in the learning model uses input value data every 30 ms.

[0052] Each input axis information collected in machine learning mode is labeled with 1 for the input state and 0 for the no-input state, and then each input axis information is converted into sequence data. Next, the data is combined for each axis to create a dataset. Finally, the dataset for each axis is split in a 9:1 ratio to create a dataset for hyperparameter tuning and a dataset for testing.

[0053] The optimal hyperparameters are determined by grid search. There are 60 combinations of hyperparameters: sequence length 30, 60, 90, 120, patch size 512, 1024, 2048, and number of hidden layer units 5, 10, 15, 20. Each combination is verified for LSTM and GRU models. Using the data for tuning hyperparameters, 5-fold cross-validation is performed for each combination of hyperparameters to search for the combination that maximizes the average F1 score, an evaluation index. The F1 score is one of the evaluation indexes for binary classification problems, and is a value ranging from 0 to 1, with the closer to 1 the better. From the results of the grid search, the optimal combinations of hyperparameters and models on each axis, as well as the F1 scores during verification, are shown in Table 1. [Table 1]

[0054] As can be seen from Table 1, the optimal sequence lengths for the X-axis, Y-axis, Z-axis, roll rotation axis, and pitch rotation axis models were 120 and 150, respectively, while the optimal sequence length for the yaw rotation axis model was 30. In other words, it can be seen that the performance of the yaw rotation axis model is improved by classifying based on a short period of time.

[0055] A model for each axis was created using the combination of the hyperparameters and learning model determined in this way. To evaluate the versatility of the models, the F1 score for the test dataset for each model was calculated. The results are shown in Table 2. [Table 2]

[0056] As can be seen from Table 2, the F1 score for each model exceeded 0.97, demonstrating high accuracy in classification performance.

[0057] The machine learning unit 11 may be tuned in this manner in the machine learning mode. Then, in the prediction mode, the machine learning unit 11 that has undergone machine learning is used to predict predicted input axis information for a specific axis intended by the operator from input axis information input by the operator using the multi-degree-of-freedom input device 20. In fact, whether or not the predicted input axis information is correctly the input intended by the operator was verified as follows.

[0058] Specifically, we verified the operation of a drone, which is an object to be operated, using the multi-degree-of-freedom input device shown in Figure 6. The drone was flown out of sight while being controlled by watching the image from the camera mounted on the drone. The evaluation was carried out as follows: This invention: When the operation information from the multi-degree-of-freedom input device of FIG. 6 is output to a drone using the controller for the multi-degree-of-freedom input device of this invention, Comparative Example 1: When operation information is output to a drone using only the multi-degree-of-freedom input device of FIG. 6 without using the controller for the multi-degree-of-freedom input device of the present invention, Comparative Example 2: When outputting operation information to a drone using a game pad instead of the controller for a multi-degree-of-freedom input device of the present invention, Regarding the control method of the drone, the movement speed and rotation speed of the drone were controlled according to the input amount of the multi-degree-of-freedom input device.

[0059] In the case of the present invention, when the machine learning unit 11 predicts that the input axis information is noise, the speed is set to 0 as a no-input state, and when the machine learning unit 11 predicts that the input axis information is for a specific axis intended by the operator, the speed is controlled as an input state so as to change exponentially according to the input from the multi-degree-of-freedom input device.

[0060] In the case of Comparative Example 1, the thresholds for the positive and negative inputs for each axis were set to the average of the minimum input axis information, and were controlled so as to change linearly from the thresholds according to the amount of input from the multi-degree-of-freedom input device.

[0061] In the case of Comparative Example 2, the thresholds for positive and negative inputs for each axis were set to ±0.1, and were controlled so as to change linearly from the thresholds according to the amount of input from the game pad.

[0062] In both operation methods, the maximum speed was 0.6 m / s and the maximum rotation speed was 45 deg / s.

[0063] The drone was operated under these conditions, and three comparative experiments were conducted for the present invention and each comparative example. The operators were 10 right-handed people (9 men and 1 woman) aged between 22 and 25, all of whom were beginners in drone operation. The flight course used in the experiment was to have the drone take off from the takeoff and landing position, pass through the first square hoop 1, pass through two square hoops 2 with different heights and positions, pass through hoop 1 again, and land at the takeoff and landing position. In this flight course, the time it took to pass through hoop 1, pass through hoop 2, return, and pass through hoop 1 again was measured.

[0064] FIG. 7 shows a comparison graph of the flight time of the present invention and each comparative example. The vertical axis is the flight time, the box represents the variance, and X represents the median. As shown in the figure, it can be seen that the flight time of the present invention is significantly reduced compared to Comparative Example 1 and Comparative Example 2. This is considered to be due to the presence or absence of noise input axis information for axes other than the predetermined axis unintended by the operator and the presence or absence of noise input axis information for the composite axis. That is, in the present invention, the machine learning unit 11 uses predicted input axis information for the predetermined axis intended by the operator, so that there is no noise input axis information and only the input intended by the operator can be reflected, so that the occurrence of unintended input is suppressed and the flight time is reduced.

[0065] Incidentally, the controller for a multi-degree-of-freedom input device of the present invention is not limited to the above-mentioned illustrated example, and it goes without saying that various modifications can be made within the scope of the gist of the present invention. [Explanation of symbols]

[0066] 10. Controller for multi-degree-of-freedom input device 11 Machine Learning Department 12 Processing section 20 Multi-degree-of-freedom input device 21 Grip section 22 Translation mechanism section 23 Translational motion detector 24 Rotation mechanism 25 Rotational motion detector 26 Control Unit

Claims

1. A controller for a multi-degree-of-freedom input device for outputting operation information from a multi-degree-of-freedom input device for obtaining input axis information for two or more axes in a Cartesian coordinate system by an operator to an operation target object, the controller for the multi-degree-of-freedom input device comprising: a machine learning unit that performs machine learning to learn a relationship between input axis information for a predetermined axis intended by an operator and noise input axis information for an axis other than the predetermined axis that is not intended by the operator, from maximum input axis information that an operator inputs maximum for learning purposes for the predetermined axis using a multi-degree-of-freedom input device, the machine learning unit that has completed machine learning predicts predicted input axis information for the predetermined axis intended by the operator from the input axis information input by the operator using the multi-degree-of-freedom input device; a processing unit that selects whether input axis information input by an operator using a multi-degree-of-freedom input device is input for learning or prediction to the machine learning unit, and outputs predicted input axis information predicted by the machine learning unit that has completed machine learning from the input axis information input for prediction as operation information for an operation target object in a Cartesian coordinate system by the operator; A controller for a multi-degree-of-freedom input device, comprising:

2. 2. The controller for a multi-degree-of-freedom input device according to claim 1, wherein the machine learning unit further performs machine learning to learn a relationship between input axis information for a specified axis intended by the operator and noise input axis information for an axis other than the specified axis that is not intended by the operator, from minimum input axis information that an operator inputs for learning purposes using the multi-degree-of-freedom input device.

3. 2. The controller for a multi-degree-of-freedom input device according to claim 1, wherein when there is a noise axis having noise input axis information greater than a predetermined amount among noise input axis information for axes other than the specified axis that are not intended by the operator, the machine learning unit sets a composite axis of the specified axis and the noise axis as a new specified axis, and machine-learns the relationship between the input axis information for the new specified axis intended by the operator and the noise input axis information for an axis other than the new specified axis that is not intended by the operator, from maximum input axis information that the operator inputs maximum for learning purposes using the multi-degree-of-freedom input device.

4. 3. The controller for a multi-degree-of-freedom input device according to claim 2, wherein when there is a noise axis having noise input axis information greater than a predetermined amount among noise input axis information for axes other than the specified axis that are not intended by the operator, the machine learning unit sets a composite axis of the specified axis and the noise axis as a new specified axis, and machine-learns the relationship between the input axis information for the new specified axis intended by the operator and the noise input axis information for an axis other than the new specified axis that is not intended by the operator, from minimum input axis information that the operator inputs for learning purposes using the multi-degree-of-freedom input device.

Images