Task segmentation device, remote operation system, task segmentation method, and program

Abstract

A task segmentation device (22) according to the present disclosure segments work executed by a remote robot (2) into segment tasks that are simple tasks, said remote robot (2) being remotely operated in accordance with an operation command from an operator terminal (1). The task segmentation device (22) comprises a clustering unit (222) that, using at least a state variable including an operation command value included in the operation command and a sensor value indicating a detection result acquired by a sensor (25) for detecting the state of the remote robot (2) as target data for classification, clusters target data for each of a plurality of time points corresponding to the work via an unsupervised learning method that does not require designation of the number of clusters, thereby segmenting the work into segment tasks.

Description

TASK SEGMENTATION DEVICE, REMOTE OPERATION SYSTEM, TASK SEGMENTATION METHOD, AND PROGRAM

[0001] The present disclosure relates to a task segmentation device, a remote operation system, a task segmentation method, and a program for segmenting the operation content of an operator who remotely operates a robot.

[0002] In the remote operation of robots, a technology is known in which the operator's actions are understood by segmenting the operation content based on sensor values ​​detected by sensors. Non-Patent Document 1 below discloses that when an operator remotely controls a robot arm to perform an assembly task such as "pulling out a peg inserted into a hole and inserting it into another hole," the operation content is segmented using data obtained by sensors such as force sensors, joint angle sensors, and cameras. Hereinafter, the segmentation of the operator's operation content, i.e., the work of a remotely operated robot, into simple tasks (simple actions) will be referred to as task segmentation.

[0003] Kota Imai, Yasutake Takahashi, Satoki Tsukijihara, Masaki Haruna, "Movement Segmentation Based on Operation Data of a Remotely Controlled Robot," Hokuriku Shinetsu Symposium (2021)

[0004] However, in the technology described in Non-Patent Document 1, the assembly task of "pulling out a peg inserted into a hole and inserting it into another hole" is manually divided into simple tasks such as "Step 1: Move to the peg," "Step 2: Grab the peg," "Step 3: Pull out the peg," "Step 4: Move to the hole," and "Step 5: Insert the peg into the hole" by a user in advance, and the conditions for transitioning from one action to another (firing conditions) are also manually determined by the user in advance. This poses a problem in that task segmentation requires the effort of an operator or a user other than the operator. Note that the user may be an operator, a person other than the operator, or both an operator and a person other than the operator.

[0005] The present disclosure has been made in view of the above, and aims to provide a task segmentation device that can perform task segmentation while reducing the effort required by the user.

[0006] In order to solve the above-mentioned problems and achieve the objectives, the task segmentation device disclosed herein is a task segmentation device that segments work to be performed by a robot that is remotely controlled in accordance with operation commands from an operator terminal into segmented tasks, which are simple tasks, and is equipped with a clustering unit that uses at least state variables including an operation command value included in the operation command and a sensor value indicating a detection result acquired by a sensor that detects the state of the robot as target data for classification, and clusters the target data at each of multiple times corresponding to the work using an unsupervised learning method that does not require the number of clusters to be specified, thereby segmenting the work into segmented tasks.

[0007] The task segmentation device according to the present disclosure has the effect of being able to segment tasks while reducing the amount of work required by the user.

[0008] FIG. 1 shows a configuration example of a remote operation system according to the first embodiment. FIG. 1 shows a configuration example of a task segmentation device according to the first embodiment. FIG. 2 shows a flowchart showing an example of a processing procedure in the task segmentation device according to the first embodiment. FIG. 2 shows another configuration example of a task segmentation device according to the first embodiment. FIG. 3 shows another configuration example of a remote operation system according to the first embodiment. FIG. 4 shows a configuration example of a task segmentation device according to the fourth embodiment. FIG. 5 shows a configuration example of a remote operation system according to the fifth embodiment.

[0009] A task segmentation device, a remote operation system, a task segmentation method, and a program according to embodiments will be described in detail below with reference to the accompanying drawings.

[0010] First Embodiment Fig. 1 is a diagram showing an example of the configuration of a remote control system according to a first embodiment. The remote control system 100 of this embodiment includes an operator terminal 1 and a remote robot 2. The operator terminal 1 is a terminal that can be operated by an operator. The operator can remotely control the remote robot 2 by operating the operator terminal 1.

[0011] The remote-controlled robot 2 is a robot that is remotely controlled by operation commands from the operator terminal 1. The remote-controlled robot 2 may be, for example, but is not limited to, a manipulator (robot arm), a combination of a cart and a manipulator, or a humanoid robot. The remote-controlled robot 2 may be, for example, an industrial robot or industrial machine used in the industrial field, but may also be, for example, a nursing robot, a housework robot, a transport robot, etc., and the uses of the remote-controlled robot 2 are not limited to these.

[0012] A method is known in which the remote-controlled robot 2 learns the operations performed by an operator in advance in order to have the remote-controlled robot 2 perform a task. For example, in "evolutionary teleoperation," which realizes the autonomy and automation of the remote-controlled robot 2's tasks, task segmentation technology, which understands the operator's actions and segments the operator's series of actions, i.e., the operation contents, into simple tasks, is important. In "evolutionary teleoperation," for example, the remote-controlled robot 2 learns its operations by operating based on the operator's operations, and while the remote-controlled robot 2 operates automatically based on the learning results, the operator intervenes if the remote-controlled robot 2's operation is inappropriate. If task segmentation is not performed when the operator intervenes, it is impossible to know which task (task) is currently being performed and which task has failed, and therefore each task comprising the task must be replayed sequentially. For example, when performing the task of "pulling out a peg that has been inserted into a hole and inserting it into another hole," it is necessary to sequentially perform all of the following steps: "Step 1: Move to the peg," "Step 2: Grab the peg," "Step 3: Pull out the peg," "Step 4: Move to the hole," and "Step 5: Insert the peg into the hole." On the other hand, by segmenting the task, the operator can grasp which step the remote-controlled robot 2 is currently performing and can also grasp the history of the steps that have been performed, making it easier to understand the cause of the failure, and the operator can simply operate the remote-controlled robot 2 from the step that needs to be recovered.

[0013] Task segmentation can be performed by defining segmented tasks in advance or by setting firing conditions in advance, but in this case, manual preparation is required. Therefore, the remote operation system 100 of this embodiment performs task segmentation using machine learning, thereby realizing task segmentation with reduced effort on the part of the user.

[0014] 1, the operator terminal 1 includes a display unit 11, an input I / F (Interface) 12, and a transceiver unit 13. The display unit 11 displays various types of information to the operator. The display unit 11 displays information received from the remote robot 2 via the transceiver unit 13. The display unit 11 may be, for example, a display, a monitor, a head-mounted display, or VR (Virtual Reality) goggles, but is not limited to these.

[0015] The input I / F 12 accepts operations by the operator. The input I / F 12 may be, but is not limited to, a joystick, a touchpad, a button, a keyboard, a mouse, a game controller, or a device that accepts input by detecting an operator's gesture based on a captured image of the operator or the operator's muscle movements. The input I / F 12 may also be a device that accepts audio input, such as a microphone.

[0016] The transmitter / receiver 13 communicates with other devices. For example, the transmitter / receiver 13 transmits an operation command indicating the content of an operation received by the input I / F 12 to the remote robot 2 via the network. The operation command includes an operation command value indicating the amount of operation to operate the remote robot 2. The transmitter / receiver 13 also outputs information received from the remote robot 2 to the display unit 11. Specifically, for example, the transmitter / receiver 13 outputs a sensor value received from the remote robot 2 to the display unit 11, and the display unit 11 displays the sensor value, allowing the operator to operate the input I / F 12 while checking the sensor value. Note that while FIG. 1 illustrates an example in which one or more of the input I / F 12, the display unit 11, and the transmitter / receiver 13 are provided within the operator terminal 1, one or more of the input I / F 12, the display unit 11, and the transmitter / receiver 13 may be provided separately from the main body of the operator terminal 1 and connected to the operator terminal 1 via a wired or wireless connection. That is, at least some of the input I / F 12, the display unit 11, and the transmitting / receiving unit 13 may be external devices, and hereinafter, the term "operator terminal 1" will be used to refer to such external devices as well.

[0017] The remote robot 2 includes a transmitter / receiver 21, a task segmentation device 22, a classification device 23, a task execution unit 24, and a sensor 25. The transmitter / receiver 21 communicates with other devices. For example, the transmitter / receiver 21 can output an operation command value included in an operation command received from the operator terminal 1 to the task segmentation device 22, the classification device 23, and the task execution unit 24. The transmitter / receiver 21 also transmits a sensor value, which is a detection result by the sensor 25, to the operator terminal 1. The transmitter / receiver 21 may also transmit at least one of the calculation result in the task segmentation device 22 and the calculation result in the classification device 23 to the operator terminal 1.

[0018] The task segmentation device 22 performs machine learning for task segmentation and outputs the learning result to the classification device 23. Details of the task segmentation device 22 will be described later. The classification device 23 classifies the currently performed work into segmented tasks using the learning result by the task segmentation device 22, the sensor values, and the operation command values. In other words, the classification device 23 performs task segmentation of the currently performed work using the learning result by the task segmentation device 22, the sensor values, and the operation command values.

[0019] The task execution unit 24 executes a task on the operation target 3 using the operation command received from the operator terminal 1 via the transceiver 21 and the detection result by the sensor 25. The task execution unit 24 includes, for example, a movable unit such as a manipulator (robot arm) or a moving mechanism, and a motion control unit that controls the movable unit. The motion control unit controls the operation of the movable unit by generating control commands for operating the movable unit based on the operation command and outputting the control commands to the movable unit. For example, if the task execution unit 24 includes a manipulator, the motion control unit generates control commands for controlling the operation of each joint and gripper (hand) of the manipulator. The operation target 3 may be, for example, an object grasped by the remote robot 2, an object transported by the remote robot 2, or an object to which the remote robot 2 applies force. Note that FIG. 1 illustrates an example in which a task is executed on the operation target 3; however, the work performed by the remote robot 2 is not limited to the example involving the operation target 3. In other words, the operation target 3 is not essential. The specific configuration and operation of the task execution unit 24 are not limited thereto. Furthermore, the task executing unit 24 may generate a control command using the classification result from the classification device 23 .

[0020] The sensor 25 detects the state of the remote robot 2 and outputs the detection result as a sensor value. In other words, the sensor value indicates the detection result obtained by the sensor 25. The sensor value is output, for example, to the transmitter / receiver 21, the task segmentation device 22, the classification device 23, and the task execution unit 24. The sensor 25 may include, but is not limited to, at least one of a sensor that detects the gripping force of a gripper in the remote robot 2, a sensor that detects the distance between the remote robot 2 and the operation target object 3, a sensor that detects the angle of the joints of the remote robot 2, and a camera that captures images of the surroundings of the remote robot 2. Note that while FIG. 1 illustrates one sensor 25, multiple sensors 25 may be used. Furthermore, at least some of the sensors 25 may be provided external to the remote robot 2. For example, the sensor 25 may include a camera that captures images of the remote robot 2 from outside, or the sensor 25 may be provided on the operation target object 3.

[0021] Fig. 2 is a diagram showing an example of the configuration of the task segmentation device 22 according to this embodiment. As shown in Fig. 2, the task segmentation device 22 includes a data processing unit 221, a clustering unit 222, a result storage unit 223, and a labeling unit 224. Hereinafter, operation command values ​​and sensor values ​​will be collectively referred to as state variables. The state variables indicate the operating state of the remote robot 2, and the task segmentation device 22 uses the state variables to perform task segmentation.

[0022] The data processing unit 221 processes the processing target variables by performing multiple types of calculations using the processing target variables, which are at least a part of the state variables. In particular, the data processing unit 221 processes at least a part of the operation command values ​​received from the transmitter / receiver 21, i.e., the operation command values ​​included in the operation command transmitted from the operator terminal 1, and the sensor values ​​received from the sensor 25, and outputs the processed data, which is the processed data, to the clustering unit 222. The processed data output by the data processing unit 221 is data in which at least a part of the state variables input to the data processing unit 221 has been processed, and the processed data output by the data processing unit 221 may also include unprocessed data. In other words, the processed data output by the data processing unit 221 may include unprocessed state variables and processed state variables (results of the calculations performed by the data processing unit 221).

[0023] For example, when t is an integer indicating a discrete time, the data processing unit 221 outputs L-dimensional data at each time in a time series, such as L (L is an integer of 2 or more) types of data (state variables and data processed from the state variables) at t = 0 and L types of data at t = 1.

[0024] The processing performed by the data processing unit 221 may include, but is not limited to, multiple types of calculations. It may also be a single calculation. For example, the data processing unit 221 may perform calculations using a sign function or elementary function that outputs a value corresponding to the positive or negative sign of an input value. It may also perform differentiation, integration, or a second- or higher-order differentiation or integration, or a combination of two or more of these. The elementary function may be, for example, a trigonometric function such as sin or cos, an algebraic function, a logarithmic function, or an exponential function, or a combination of these. For example, the processed data output by the data processing unit 221 may include a sensor value, an operation command value, a differential value of the sensor value, and a calculation result using a sign function of the sensor value. The processed data output by the data processing unit 221 is not limited to this example.

[0025] When performing machine learning for task segmentation, appropriate learning results may be obtained by using data obtained by processing state variables, such as differential values ​​of sensor values, as input data. On the other hand, it may be difficult to determine what type of processing is effective without actually performing machine learning. Therefore, if the data processing unit 221 performs multiple types of calculations, data corresponding to each of the multiple types of calculations can be used as input data, thereby increasing the likelihood that the input data contains valid data. For example, by calculating the differential values ​​of the sensor values ​​and the results of calculations using a sign function of the sensor values, more appropriate clustering may be achieved than when only the differential values ​​of the sensor values ​​are calculated. Note that, while an example in which the data processing unit 221 performs multiple calculations will be described below, this is not limiting and the data processing unit 221 may perform only one calculation. Furthermore, the state variables may be input directly to the clustering unit 222 without providing the data processing unit 221.

[0026] The clustering unit 222 uses at least the state variables as target data for classification to cluster the target data at each of multiple time periods corresponding to tasks performed by the remote robot 2 using an unsupervised learning method that does not require the number of clusters to be specified, thereby segmenting the tasks into simple tasks (hereinafter also referred to as segmented tasks). Specifically, the clustering unit 222 uses the results of the calculations performed by the data processing unit 221 and the state variables as target data to perform clustering using an unsupervised learning method that does not require the number of clusters to be specified, thereby segmenting the tasks into segmented tasks, and stores the clustering results in the result storage unit 223 as learning results. That is, the clustering unit 222 classifies the processed data, which is time-series data received from the data processing unit 221, into multiple clusters. As described above, the data corresponding to each clustered point is L-dimensional data. The clustering result includes, for example, processed data corresponding to each point to which information indicating which cluster it belongs to (hereinafter also referred to as labeled processed data). The learning result may also include indicators indicating the center of gravity and spread of each cluster. The index indicating the spread is, for example, a variance value or data obtained by processing the variance value, but is not limited to these.

[0027] Examples of unsupervised learning methods that do not require the number of clusters to be specified include, but are not limited to, the x-means method, the g-means method, star clustering, etc. In the following, an example will be described in which the x-means method is used as an unsupervised learning method that does not require the number of clusters to be specified, but the unsupervised learning method that does not require the number of clusters to be specified is not limited to the x-means method.

[0028] The x-means method is a clustering method that recursively invokes clustering using the k-means method. The k-means method divides data into multiple clusters, calculates the center of gravity of each cluster, and repeatedly reassigns each data to a cluster based on the distance between the center of gravity and each data until the cluster changes, i.e., converges. The x-means method recursively applies 2-means clustering to each cluster after the k-means clustering has converged. The application of 2-means clustering is stopped when a specified evaluation criterion is met, allowing data to be classified into clusters without setting the number of clusters in advance. Examples of evaluation criteria that can be used include, but are not limited to, the Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC).

[0029] The labeling unit 224 uses the clustering results stored in the result storage unit 223 to assign a label to each cluster clustered by the clustering unit 222, associates the labeled clustering results with the labeled clustering results, and stores them in the result storage unit 223. The label is information indicating the segmented tasks, such as information indicating the content of the operation of each segmented task. The labeling unit 224 may assign a label to each cluster by accepting an input of a label corresponding to each cluster from a user, or may estimate a label using supervised machine learning, image processing, or the like, and assign the estimated label to each cluster.

[0030] When the labels of each cluster are estimated using supervised machine learning, for example, a trained model is generated in advance using supervised learning using multiple datasets consisting of image data capturing the operation of the remote robot 2 and corresponding labels, which are ground truth data. This trained model is a trained model for inferring labels from image data, which are features. The image data may be, for example, video data captured over a certain period of time, and may be acquired by a sensor 25 capturing the operation of the remote robot 2, or may be acquired by a camera other than the sensor 25. Furthermore, the features are not limited to the image data itself, but may also be data extracted by processing the image data. The label assignment unit 224 estimates labels by inputting image data (or features extracted from the image data) corresponding to each cluster into the trained model and inferring the labels. In this case, the sensor 25 includes a camera capturing the operation of the remote robot 2, and the label assignment unit 224 selects image data corresponding to each cluster and inputs the selected image data (or features extracted from the image data) into the trained model, thereby inferring labels corresponding to each cluster. The labeling unit 224 may select, for example, one or more pieces of imaging data from each cluster. As the supervised learning algorithm, for example, neural network genetic programming, functional logic programming, SVM (Support Vector Machine), etc. may be used, but the supervised learning algorithm is not limited to these.

[0031] Even when labels are estimated by image processing, for example, corresponding photographic data is prepared for each label, and the label assignment unit 224 estimates a label corresponding to each cluster based on, for example, the similarity between features extracted from the photographic data prepared in advance and features extracted from the photographic data corresponding to each cluster that has been clustered.

[0032] The method of estimating labels is not limited to the above-described example. Note that, although an example has been described in which the task segmentation device 22 includes the label assignment unit 224, assigning labels is not essential, and the task segmentation device 22 does not have to include the label assignment unit 224. Labels are assigned so that a user, which is at least one of an operator and a person other than the operator, can easily understand the content of the segmented tasks, but the segmentation itself has been completed by the clustering unit 222, and identification information such as a number corresponding to each cluster may be used as a label.

[0033] FIG. 3 is a flowchart showing an example of a processing procedure in the task segmentation device 22 according to this embodiment. As shown in FIG. 3 , the task segmentation device 22 acquires state variables (step S1). Specifically, the data processing unit 221 acquires state variables by receiving sensor values ​​from the sensor 25 and operation command values ​​from the transmission / reception unit 21. The sensor values ​​and operation command values, which are state variables, are acquired periodically, for example, at the same cycle. When some state variables are acquired at different cycles, the data processing unit 221 may, for example, determine a state variable that serves as a time reference, select another state variable that corresponds to the time closest to the time at which the state variable is acquired, and consider the state variable that serves as the time reference and the selected other state variable as a set of data corresponding to the same time. Alternatively, the data processing unit 221 may interpolate the other state variables to calculate the other state variables that correspond to the time at which the state variable that serves as the time reference is acquired.

[0034] Next, the task segmentation device 22 processes the state variables (step S2). Specifically, the data processing unit 221 processes the data by performing multiple types of calculations on at least some of the acquired state variables, and outputs the processed data to the clustering unit 222. Note that steps S1 and S2 are performed sequentially, for example, during a collection period during which data to be clustered is collected. As a result, time-series data corresponding to the collection period is acquired.

[0035] Next, the task segmentation device 22 performs clustering (step S3). In detail, the clustering unit 222 performs task segmentation by clustering using an unsupervised learning method that does not require specifying the number of clusters, using the processed data received from the data processing unit 221, and stores the clustering result as a learning result in the result storage unit 223.

[0036] Next, the task segmentation device 22 assigns labels (step S4) and ends the process. Specifically, in step S4, the label assignment unit 224 assigns labels to each cluster clustered by the clustering unit 222, using the clustering results stored in the result storage unit 223. Note that, as described above, label assignment does not necessarily have to be performed.

[0037] The clustering results obtained by the task segmentation device 22, i.e., the learning results, are input to, for example, the classification device 23. The classification device 23 uses the learning results and processed data obtained by processing the current state variables in the same manner as the data processing unit 221 to classify the currently performed operations into segmented tasks. For example, when applied to "evolutionary teleoperation," state variables acquired while the remote-controlled robot 2 is operating automatically are input to the classification device 23, thereby enabling task segmentation of the work being performed by the remote-controlled robot 2. The results of this task segmentation are transmitted to the operator terminal 1 via the transceiver 21, and the operator terminal 1 displays the classification results, allowing an operator, manager, or the like to understand the task currently being performed. As a result, if the remote-controlled robot 2 fails in a task, the operator can understand which step the remote-controlled robot 2 is currently performing and can efficiently intervene in the operation of the remote-controlled robot 2. Furthermore, displaying the classification results on the operator terminal 1 can also be used to monitor the work status.

[0038] The command values ​​for the remote-controlled robot 2 to operate automatically may be generated in any manner. For example, they may be generated based on information including operation command values ​​from previous remote operations by an operator, or may be generated by other methods. Furthermore, when an operator intervenes, the command values ​​for the remote-controlled robot 2 to operate automatically may be modified based on the operation command values ​​from the remote operations by the operator.

[0039] Furthermore, the clustering results, i.e., learning results, obtained by the task segmentation device 22 are not limited to being used to classify current actions, but may also be used to automate and automate the remote robot 2.

[0040] In addition, in the example shown in Figure 2, an example was described in which the task segmentation device 22 performs clustering using an unsupervised learning method that does not require specifying the number of clusters, but this is not limited to this, and task segmentation may also be performed using supervised learning.

[0041] FIG. 4 is a diagram showing another example configuration of the task segmentation device 22 according to this embodiment. In FIG. 4, the task segmentation device 22 performs task segmentation through supervised learning. The task segmentation device 22 shown in FIG. 4 includes a data processing unit 221, a model generation unit 225, and a result storage unit 226. The data processing unit 221 processes the state variables by performing multiple types of calculations on at least some of the state variables, as in the example shown in FIG. 2, and outputs the processed data to the model generation unit 225. The model generation unit 225 generates trained models through supervised learning using multiple data sets each consisting of processed data, which are feature quantities, and labels, which are ground truth data, and stores the generated trained models in the result storage unit 226.

[0042] In this case, the classification device 23 can infer labels corresponding to the segmented tasks by inputting processed data obtained by processing the current state variables in the same manner as the data processing unit 221 into the trained model. Examples of supervised learning algorithms that can be used include, but are not limited to, neural network genetic programming, functional logic programming, and support vector machines. Furthermore, even when the task segmentation device 22 shown in FIG. 4 is used, only the correct answer data corresponding to the processed data needs to be prepared in advance. This eliminates the need to manually model the actions or manually predetermine the firing conditions, thereby reducing the user's workload. Furthermore, by having the data processing unit 221 perform multiple types of calculations, the likelihood that the input data contains valid data can be increased, as in the example shown in FIG. 2.

[0043] Furthermore, in the example shown in FIG. 1, an example has been described in which the task segmentation device 22 is provided inside the remote robot 2, but the location in which the task segmentation device 22 is provided is not limited to this. FIG. 5 is a diagram showing another example of the configuration of the remote operation system of this embodiment. The remote operation system 100a shown in FIG. 5 includes an operator terminal 1a and a remote robot 2a. Components similar to those shown in FIG. 1 are assigned the same reference numerals as in FIG. 1, and redundant explanations will be omitted. In the example shown in FIG. 5, the task segmentation device 22 is provided inside the operator terminal 1a.

[0044] The task segmentation device 22 may be provided separately from the operator terminal 1, 1a and the remote robot 2, 2a. In this case, the task segmentation device 22 acquires, for example, operation command values ​​from the operator terminal 1 similar to that shown in Fig. 1 and sensor values ​​from the remote robot 2a similar to that shown in Fig. 5.

[0045] Next, the hardware configuration of the task segmentation device 22 of this embodiment will be described. The task segmentation device 22 of this embodiment shown in FIG. 1 functions as the task segmentation device 22 when a computer system executes a computer program that describes the processing to be performed by the task segmentation device 22. FIG. 6 is a diagram showing an example of the configuration of a computer system that realizes the task segmentation device 22 of this embodiment. As shown in FIG. 6, this computer system includes a control unit 101, an input unit 102, a memory unit 103, a display unit 104, a communication unit 105, and an output unit 106, which are connected via a system bus 107. The control unit 101 and the memory unit 103 form a processing circuit.

[0046] In FIG. 6 , the control unit 101 is a processor such as a CPU (Central Processing Unit) that executes a program describing the processing in the task segmentation device 22 of this embodiment. Note that a portion of the control unit 101 may be realized by dedicated hardware such as a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array). The input unit 102 includes buttons, a keyboard, a mouse, a touchpad, and the like. The storage unit 103 includes various types of memory such as RAM (Random Access Memory) and ROM (Read Only Memory) and a storage device such as a hard disk, and stores programs to be executed by the control unit 101, necessary data obtained during processing, and the like. The storage unit 103 is also used as a temporary storage area for programs. As described above, the display unit 104 is, for example, a display. Note that the display unit 104 and the input unit 102 may be integrated and realized by a touch panel or the like. The communication unit 105 is a receiver and transmitter that perform communication processing. The output unit 106 is, for example, a speaker. Note that Fig. 6 is an example, and the configuration of the computer system is not limited to the example of Fig. 6. For example, in this embodiment, the computer system that realizes the task segmentation device 22 does not need to include the display unit 104 and the output unit 106.

[0047] Here, an example of the operation of the computer system until the program of this embodiment is ready to be executed will be described. In a computer system having the above configuration, for example, a computer program is installed into storage unit 103 from a CD-ROM or DVD-ROM inserted in a CD (Compact Disc)-ROM drive or DVD (Digital Versatile Disc)-ROM drive (not shown). Then, when the program is executed, the program is read from storage unit 103 and stored in the main storage area of ​​storage unit 103. In this state, control unit 101 executes processing as task segmentation device 22 of this embodiment in accordance with the program stored in storage unit 103.

[0048] In the above explanation, a program describing the processing in the task segmentation device 22 is provided using a CD-ROM or DVD-ROM as a recording medium, but this is not limited to this, and it is also possible to use a program provided via a transmission medium such as the Internet, depending on the configuration of the computer system, the capacity of the program to be provided, etc.

[0049] The data processing unit 221, clustering unit 222, and labeling unit 224 shown in Fig. 2 are realized by the control unit 101 shown in Fig. 6 executing a computer program stored in the storage unit 103 shown in Fig. 6. The storage unit 103 shown in Fig. 6 is also used to realize the data processing unit 221, clustering unit 222, and labeling unit 224 shown in Fig. 2. The result storage unit 223 shown in Fig. 2 is part of the storage unit 103 shown in Fig. 6.

[0050] As described above, the task segmentation device 22 of this embodiment performs task segmentation using machine learning. This makes it possible to achieve task segmentation with reduced user effort. Furthermore, by having the task segmentation device 22 perform task segmentation using clustering based on an unsupervised learning method that does not require the specification of the number of clusters, the number of clusters does not need to be specified, and task segmentation can be fully automated. This not only reduces the amount of effort that has previously been required for tuning by the user, i.e., a human, but also makes it possible to cluster tasks that are likely to be overlooked when simply set up in advance by a human, such as "classification of task failures," using the task segmentation device 22 of this embodiment, thereby achieving segmentation with fewer omissions.

[0051] Furthermore, when input data to be input into clustering is predetermined, such as sensor values ​​and differential values ​​of the sensor values, it is difficult to collect all appropriate data in advance. For example, by combining data processed using several operators, including differentials, with sensor values, it is possible to increase the likelihood that this composite input data will contain data that is effective for task segmentation.

[0052] Second Embodiment Fig. 7 is a diagram showing an example of the configuration of a task segmentation device according to a second embodiment. The remote operation system of this embodiment is similar to the remote operation system 100 of the first embodiment, except that it includes a task segmentation device 22a instead of the task segmentation device 22. Components having the same functions as those of the first embodiment are given the same reference numerals as those of the first embodiment, and redundant explanations will be omitted. Below, differences from the first embodiment will be mainly explained.

[0053] The task segmentation device 22a includes an additional firing condition identification unit 227 in addition to the components of the task segmentation device 22 of the first embodiment. The firing condition identification unit 227 re-clusters the target data clustered by the clustering unit 222 using a method capable of obtaining a separating hyperplane, and identifies thresholds for each state variable when the segmented task transitions from the separating hyperplane obtained by the re-clustering. Specifically, the firing condition identification unit 227 re-clusters the clustering results stored in the result storage unit 223 using a method capable of obtaining a separating hyperplane (e.g., SVM), calculates thresholds for each state variable when the task transitions from the obtained separating hyperplane, and stores the calculated thresholds as firing conditions in the result storage unit 223. The firing conditions are output to the classification device 23 as part of the learning results. The classification device 23 uses the firing conditions to perform classification, thereby speeding up the classification process. The ignition condition identification unit 227 may output the ignition condition to the transmitting / receiving unit 21 , and the transmitting / receiving unit 21 may transmit the ignition condition to the operator terminal 1 .

[0054] The task segmentation device 22a of this embodiment is also realized by a computer system, similar to the task segmentation device 22 of embodiment 1. The ignition condition identification unit 227 is realized, for example, by the control unit 101 shown in FIG. 6 executing a computer program stored in the storage unit 103 shown in FIG. 6. The storage unit 103 shown in FIG. 6 is also used to realize the ignition condition identification unit 227. Similarly to the task segmentation device 22 of embodiment 1, the task segmentation device 22a of this embodiment may be provided within the operator terminal 1a, or may be provided separately from the operator terminals 1, 1a and the remote robots 2, 2a.

[0055] Third Embodiment Fig. 8 is a diagram showing an example of the configuration of a task segmentation device according to a third embodiment. The remote operation system of this embodiment is similar to the remote operation system 100 of the first embodiment, except that it includes a task segmentation device 22b instead of the task segmentation device 22. Components having the same functions as those of the first embodiment are assigned the same reference numerals as those of the first embodiment, and redundant explanations will be omitted. Differences from the first embodiment will be mainly explained below.

[0056] As described in the first embodiment, the possibility of appropriate clustering increases when the data processing unit 221 performs multiple calculations, but on the other hand, the clustering takes time as the input data increases. For this reason, in this embodiment, the task segmentation device 22b evaluates the clustering results to reduce the input data for clustering and speed up task segmentation.

[0057] The task segmentation device 22b includes a clustering accuracy evaluation unit 228 and a data reduction unit 229 in addition to the components of the task segmentation device 22 of the first embodiment. The clustering accuracy evaluation unit 228 uses the clustering results stored in the result storage unit 223 to evaluate the clustering accuracy (how well the clustering has been performed) using silhouette calculation, intra-cluster error average sum, etc., and outputs an evaluation value indicating the clustering accuracy to the data reduction unit 229. The data reduction unit 229 sets at least one of the state variables and the processing target variables from the target data (processed data) input to the clustering unit 222 as candidates for reduced data, and notifies the clustering unit 222 of the set candidates, thereby causing the clustering unit 222 to perform clustering with the candidates reduced from the target data. In other words, the data reduction unit 229 causes the clustering unit 222 to perform clustering using the data from which the candidates have been reduced from the target data as input. Then, the data reduction unit 229 uses the evaluation value calculated by the clustering accuracy evaluation unit 228 based on the clustering results in a state where the candidates have been reduced, to determine reduction data to be reduced from the target data.

[0058] In this embodiment, for example, the data reduction unit 229 determines the reduction data to be reduced from the processed data output from the data processing unit 221 using a brute force method or a combinatorial optimization method. For example, when the data to be reduced is determined using a brute force method, the data reduction unit 229 sequentially changes the data to be reduced from the processed data, i.e., sequentially changes the combinations of data to be reduced from the data constituting the processed data, and causes the clustering unit 222 to perform clustering. Then, based on the evaluation results corresponding to each combination, the combination with the highest clustering accuracy or the combination whose clustering accuracy satisfies a predetermined accuracy is determined as the processed data after data reduction. After determining the processed data after data reduction, the data reduction unit 229 instructs the clustering unit 222 to perform clustering based on the determined results. This makes it possible to maintain clustering accuracy while reducing the time required for clustering.

[0059] When determining the data to be reduced (the combination of data to be reduced from the data that constitutes the processed data) using a combinatorial optimization method, the data reduction unit 229 searches for an optimal combination based on the algorithm of the combinatorial optimization method, using the evaluation values ​​received from the clustering accuracy evaluation unit 228. The combinatorial optimization method may be, for example, a branch and bound method, a local search method, or the like, but is not limited to these.

[0060] The task segmentation device 22b of this embodiment is also realized by a computer system, similar to the task segmentation device 22 of embodiment 1. The clustering accuracy evaluation unit 228 and the data reduction unit 229 are realized, for example, by the control unit 101 shown in FIG. 6 executing a computer program stored in the storage unit 103 shown in FIG. 6. The storage unit 103 shown in FIG. 6 is also used to realize the clustering accuracy evaluation unit 228 and the data reduction unit 229. Similarly to the task segmentation device 22 of embodiment 1, the task segmentation device 22b of this embodiment may be provided within the operator terminal 1a, or may be provided separately from the operator terminals 1, 1a and the remote robots 2, 2a.

[0061] Furthermore, a clustering accuracy evaluation unit 228 and a data reduction unit 229 may be added to the task segmentation device 22a described in the second embodiment, so that the operations described in this embodiment can be performed.

[0062] Fourth Embodiment. FIG. 9 is a diagram showing an example of the configuration of a remote control system according to a fourth embodiment. A remote control system 100b according to this embodiment includes an operator terminal 1b and remote robots 2a-1 to 2a-n (n is an integer equal to or greater than 2). Each of the remote robots 2a-1 to 2a-n is similar to the remote robot 2a shown in FIG. 5 of the first embodiment. When referring to each of the remote robots 2a-1 to 2a-n without distinguishing them individually, they are also referred to as remote robot 2a. Components having the same functions as those in the first embodiment are assigned the same reference numerals as those in the first embodiment, and redundant explanations will be omitted. Below, differences from the first embodiment will be mainly explained.

[0063] In this embodiment, the remote robot 2a remotely controlled by the operator terminal 1b can be switched. That is, in this embodiment, when there are multiple remote robots 2a to be operated and the remote robot 2a to be operated is changed, robust state variables are reused to speed up task segmentation. Note that reuse here refers to state variables that indicate what kind of variables are time-independent, and does not refer to the reuse of the values ​​of state variables in a fixed time period. The multiple remote robots 2a may be robots of the same type, or different types of robots may be mixed. Furthermore, the sensor values ​​acquired by the multiple remote robots 2a do not have to be the same.

[0064] The operator terminal 1b is similar to the operator terminal 1a shown in Fig. 5 of the first embodiment, except that it includes a task segmentation device 22c instead of the task segmentation device 22. Fig. 10 is a diagram showing an example of the configuration of the task segmentation device 22c of this embodiment. As shown in Fig. 10, the task segmentation device 22c has a state variable selection unit 230 added to the task segmentation device 22 of the first embodiment.

[0065] The state variable selection unit 230 separates the state variables in the target data clustered before the switching and the state variables acquired after the switching into first state variables that are robust state variables assumed to be unlikely to change before and after the switching, and second state variables that are state variables other than the first state variables, and notifies the data processing unit 221 of the separation results. For example, the state variable selection unit 230 may select robust state variables and notify the data processing unit 221 of information indicating the selected state variables as the separation results. For the robust state variables, the state variable selection unit 230 outputs to the data processing unit 221 information related to the separating hyperplane of the state variables related to the remote robot 2a before the change, which is stored in the result storage unit 223.

[0066] Based on the notified classification results, the data processing unit 221 extracts non-robust state variables from the newly input sensor values ​​and operation specification values, and outputs information regarding the separating hyperplane of the state variables for the remote robot 2a before the change and the extracted state variables to the clustering unit 222. Robust state variables are state variables that are expected to be unlikely to change even when the remote robot 2a to be operated is changed. Robust state variables may be predefined as state variables that are expected to be common between remote robots 2a. For example, state variables corresponding to the changed remote robot 2a that have been updated by at least one of addition and modification may be defined as non-robust state variables. Robust state variables may also be extracted automatically, for example, by the following method.

[0067] (1) The state variable selection unit 230 acquires the separating hyperplane, the centroid distance of each cluster, and the variance of the cluster distance within the state space for the remote robot 2a before the change.

[0068] (2) The state variable selection unit 230 plots the state variables (sensor values ​​and operation command values) corresponding to the modified remote robot 2a in the state space prepared for the unmodified remote robot 2a, and calculates an evaluation value that evaluates the suitability of the clustering based on the results obtained in (1) and the plot of the state variables of the modified remote robot 2a. For example, the state variable selection unit 230 calculates an evaluation value such as how far the modified data group is from the separating hyperplane obtained before the change (a quantity corresponding to a certain distance in space) or "what percentage of the modified data group falls within a 'hypersphere with a radius determined based on the variance (e.g., three times the variance value) centered on the center of gravity obtained before the change'." Note that the evaluation value is not limited to these examples, and may be an evaluation value that indicates how clearly each cluster can be separated.

[0069] (3) The state variable selection unit 230 determines robust state variables using the evaluation values ​​calculated in (2). For example, the state variable selection unit 230 determines state variables whose evaluation values ​​are equal to or greater than a threshold value as robust state variables.

[0070] The clustering unit 222 performs clustering using the state variables received from the data processing unit 221, i.e., the information on the separating hyperplane related to the reused state variables, and the extracted state variables. This allows for faster clustering compared to performing new clustering using state variables corresponding to the modified remote-controlled robot 2a. For example, consider a case where remote-controlled robots 2a-1 and 2a-2, each with a different structure, are remotely controlled to perform an assembly task of "pulling out a peg inserted into a hole and inserting the peg into another hole." Given that the remote-controlled robots 2a-1 and 2a-2 are different in structure but both have robot arms, it is assumed that state variables such as joint angles and current values ​​will differ between the remote-controlled robots 2a-1 and 2a-2, but the values ​​of state variables such as the relative position between the target position and the grip will not change significantly between the remote-controlled robots 2a-1 and 2a-2. In this way, state variables whose values ​​do not change significantly even when the remote-controlled robots 2a-1 and 2a-2, which are examples of edge devices, are changed can be defined as robust state variables. As described in the first embodiment, the data processing unit 221 does not have to be provided, and in this case, the clustering unit 222 can reuse the clustering results by, for example, fixing the separating hyperplane of the target data corresponding to the first state variable, and perform task segmentation by clustering the target data corresponding to the second state variable.

[0071] Furthermore, instead of reusing information on the separating hyperplane of the state variables, or in addition to reusing the state variables, the clustering unit 222 may reuse the clustering results. For example, clustering may be performed using, as initial values, clustering data, which is the processed data after classification obtained by clustering the remote robot 2a before the change. That is, for example, if the state variables do not change before and after the change of the operation target, clustering can be performed using the center of gravity of each cluster as the initial value, thereby speeding up clustering. Even if the state variables change before and after the change of the operation target, clustering can be speeded up by setting the value of the common state variable dimension among the centers of gravity in the multidimensional space to the value of the center of gravity obtained by clustering the remote robot 2a before the change of the operation target. Furthermore, firing conditions may be reused as the clustering results.

[0072] Like the task segmentation device 22 of the first embodiment, the task segmentation device 22c of the present embodiment is also realized by a computer system. The state variable selection unit 230 is realized, for example, by the control unit 101 of FIG. 6 executing a computer program stored in the storage unit 103 of FIG. 6. The storage unit 103 of FIG. 6 is also used to realize the state variable selection unit 230. Also, while FIG. 9 illustrates an example in which the task segmentation device 22c is provided in the operator terminal 1b, like the task segmentation device 22 of the first embodiment illustrated in FIG. 1, the task segmentation device 22c may be provided in the remote robot 2a, or may be provided separately from the operator terminal 1b and the remote robot 2a. When the task segmentation device 22c is provided in the remote robot 2a, it is sufficient that the task segmentation device 22c is provided in at least one of the multiple remote robots 2a.

[0073] Furthermore, the state variable selection unit 230 may be added to the task segmentation device 22a described in embodiment 2 or the task segmentation device 22b described in embodiment 3, so that the operations described in this embodiment are performed. Furthermore, the clustering accuracy evaluation unit 228 and data reduction unit 229 described in embodiment 3 may be added to the task segmentation device 22a described in embodiment 2, and further the state variable selection unit 230 may be added, so that the operations described in this embodiment are performed.

[0074] Fifth Embodiment. Fig. 11 is a diagram showing an example of the configuration of a remote control system according to a fifth embodiment. A remote control system 100c of this embodiment includes an operator terminal 1c and a remote robot 2a. The remote robot 2a is similar to the remote robot 2a shown in Fig. 5 of the first embodiment. Components having the same functions as those of the first and fourth embodiments are given the same reference numerals as those of the first and fourth embodiments, and redundant explanations will be omitted. Below, differences from the first and fourth embodiments will be mainly explained.

[0075] The operator terminal 1c includes the display unit 11, the transmitter / receiver 13, and the task segmentation device 22c similar to those in the fourth embodiment, as well as input I / Fs 12-1 to 12-m (m is an integer of 2 or greater). Each of the input I / Fs 12-1 to 12-m is similar to the input I / F 12 in the first embodiment. When referring to the input I / Fs 12-1 to 12-m without distinguishing them individually, they are also referred to as input I / F 12. At least some of the input I / Fs 12-1 to 12-m may be provided in different locations.

[0076] In the fourth embodiment, an example was described in which the remote-controlled robot 2a to be operated is changed. However, in the present embodiment, an example is described in which there are multiple input I / Fs 12, and the input I / F 12 used by the operator for operation is changed. That is, in the present embodiment, an example is described in which the input I / F 12 used to remotely control the remote-controlled robot 2a can be switched. For example, if the operator is operating using the input I / F 12 that is a joystick and then changes to an input I / F 12 that is a touch panel, the input I / F 12 will be changed. Even if the input I / F 12 is changed, there may be common operation command values, but if the input I / F 12 is changed, the operation command values ​​may also change.

[0077] In this embodiment, an example of an edge device is input I / F 12, and in this embodiment as well, a state variable whose value does not change significantly even when the edge device is changed is defined as a robust state variable, and as in embodiment 4, the state variable before the change is reused as the robust state variable.

[0078] The configuration of the task segmentation device 22c in this embodiment is the same as that in the fourth embodiment, but operation command values ​​can be input to the task segmentation device 22c from each of the input I / Fs 12-1 to 12-m. In this embodiment, the state variables that change when the edge device is changed are the operation command values.

[0079] The task segmentation device 22c of this embodiment reuses information about the target data clustered before switching, as in the fourth embodiment. That is, the state variable selection unit 230 of this embodiment classifies the state variables in the target data clustered before switching the input I / F 12 and the state variables acquired after switching into third state variables, which are robust state variables assumed to be unlikely to change before and after switching, and fourth state variables, which are state variables other than the third state variables, and notifies the data processing unit 221 of the classification results. Based on the notified classification results, the data processing unit 221 extracts non-robust state variables from the newly input sensor values ​​and operation specification values, and outputs information about the separating hyperplane of the state variables related to the remote robot 2a before the change and the extracted state variables to the clustering unit 222. The robust state variables may be predetermined state variables assumed to be common between the input I / Fs 12, or may be automatically extracted as in the fourth embodiment. This enables faster clustering. As described in the first embodiment, the data processing unit 221 does not have to be provided, and in this case, the clustering unit 222 can reuse the clustering results by, for example, fixing the separating hyperplane of the target data corresponding to the third state variable, and perform task segmentation by clustering the target data corresponding to the fourth state variable.

[0080] Furthermore, the task segmentation device 22c of this embodiment may reuse the results of clustering instead of or in addition to reusing state variables, as in the fourth embodiment.

[0081] Like the task segmentation device 22 of the first embodiment, the task segmentation device 22c of the present embodiment is also realized by a computer system. The state variable selection unit 230 is realized, for example, by the control unit 101 of FIG. 6 executing a computer program stored in the storage unit 103 of FIG. 6. The storage unit 103 of FIG. 6 is also used to realize the state variable selection unit 230. Also, while FIG. 11 illustrates an example in which the task segmentation device 22c is provided within the operator terminal 1c, like the task segmentation device 22 of the first embodiment illustrated in FIG. 1, the task segmentation device 22c may be provided within the remote robot 2a, or may be provided separately from the operator terminal 1c and the remote robot 2a.

[0082] Furthermore, the state variable selection unit 230 may be added to the task segmentation device 22a described in embodiment 2 or the task segmentation device 22b described in embodiment 3, so that the operations described in this embodiment are performed. Furthermore, the clustering accuracy evaluation unit 228 and data reduction unit 229 described in embodiment 3 may be added to the task segmentation device 22a described in embodiment 2, and further the state variable selection unit 230 may be added, so that the operations described in this embodiment are performed.

[0083] Furthermore, when both switching of the remote robot 2a and switching of the input I / F 12 are possible, the fourth and fifth embodiments may be combined so that the task segmentation device 22c determines robust state variables taking into account both changes in the remote robot 2a and changes in the input I / F 12, and reuses the robust state variables.

[0084] 11, the operator terminal 1c includes multiple input I / Fs 12. However, this is not limiting, and multiple operator terminals 1 according to the first embodiment may be provided. In this case, the task segmentation device 22c acquires operation command values ​​output from the multiple input I / Fs 12 of the multiple operator terminals 1. When the operator terminal 1 used for operation is changed, the task segmentation device 22c performs clustering by reusing the state variables corresponding to the input I / Fs 12 of the previous operator terminal 1 or by reusing the clustering results of the input I / Fs 12 of the previous operator terminal 1. The multiple operator terminals 1 may be provided in different locations. For example, the operation of this embodiment can be applied to a case where an operator in Tokyo is operating a remote robot 2a using an operator terminal 1, and then an operator in Osaka begins to operate the remote robot 2a using another operator terminal 1. In this example, the task segmentation device 22c may be provided in one or more of the multiple operator terminals 1, or may be provided within the remote robot, or may be provided separately from the operator terminal and the remote robot.

[0085] The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, or different embodiments may be combined with each other. It is also possible to omit or modify parts of the configurations as long as they do not deviate from the gist of the invention.

[0086] 1, 1a, 1b, 1c Operator terminal, 2, 2a Remote robot, 3 Operation object, 11 Display unit, 12 Input I / F, 13, 21 Transmitting / receiving unit, 22, 22a, 22b, 22c Task segmentation device, 23 Classification device, 24 Task execution unit, 25 Sensor, 100, 100a, 100b, 100c Remote operation system, 221 Data processing unit, 222 Clustering unit, 223, 226 Result storage unit, 224 Label assignment unit, 225 Model generation unit, 227 Firing condition identification unit, 228 Clustering accuracy evaluation unit, 229 Data reduction unit, 230 State variable selection unit.

Claims

1. A task segmentation device that segments an operation to be performed by a robot remotely operated in accordance with an operation command from an operator terminal into segmented tasks, which are simple tasks, comprising: a clustering unit that uses at least state variables including an operation command value included in the operation command and a sensor value indicating a detection result obtained by a sensor that detects the state of the robot as target data for classification, and segments the operation into the segmented tasks by clustering the target data at each of a plurality of times corresponding to the operation using an unsupervised learning method that does not require the number of clusters to be specified.

2. A data processing unit that processes the variables to be processed by performing multiple types of calculations using variables to be processed that are at least a part of the state variables, wherein the clustering unit uses the results of the calculations by the data processing unit and the state variables as the target data and segments the work into the segmented tasks by clustering.

3. A task segmentation device as described in claim 2, comprising: a data reduction unit which sets at least one of the state variables and the variables to be processed as candidates for reduced data from the target data input to the clustering unit; and a clustering accuracy evaluation unit which calculates an evaluation value indicating accuracy of clustering using the target data clustered by the clustering unit, wherein the data reduction unit executes the clustering using data obtained by reducing the candidates from the target data as input, and determines the reduced data to be reduced from the target data using the evaluation value calculated by the clustering accuracy evaluation unit.

4. A task segmentation device as described in any one of claims 1 to 3, characterized in that it is equipped with a trigger condition identification unit that re-clusters the target data clustered by the clustering unit using a method capable of obtaining a separating hyperplane, and identifies the threshold value of each of the state variables when the segmented task transitions from the separating hyperplane obtained by the re-clustering.

5. The task segmentation device of any one of claims 1 to 4, characterized in that the robot, which is remotely controlled by the operator terminal, can be switched, and the task segmentation device comprises: a state variable selection unit that separates the state variables in the target data clustered before the switching and the state variables acquired after the switching into first state variables which are robust state variables assumed to be unlikely to change before and after the switching, and second state variables which are state variables other than the first state variables, and the clustering unit reuses the clustering results of the target data corresponding to the first state variables and clusters the target data corresponding to the second state variables, thereby segmenting the work into segmented tasks.

6. A task segmentation device as described in any one of claims 1 to 5, characterized in that the input interface for remotely controlling the robot can be switched, and the task segmentation device comprises: a state variable selection unit that separates the state variables in the target data clustered before the switching of the input interface and the state variables acquired after the switching of the input interface into a third state variable which is a robust state variable that is assumed to be unlikely to change before and after the switching, and a fourth state variable which is a state variable other than the third state variable, and the clustering unit reuses the clustering result of the third state variable and clusters the target data corresponding to the fourth state variable, thereby segmenting the work into segmented tasks.

7. A remote operation system comprising: an operator terminal operable by an operator; a robot remotely operated in accordance with operation commands from the operator terminal; and a task segmentation device which segments work performed by the robot into segmented tasks which are simple tasks, wherein the task segmentation device comprises a clustering unit which uses at least state variables including an operation command value contained in the operation command and a sensor value indicating a detection result acquired by a sensor which detects the state of the robot as target data for classification, and clusters the target data at each of a plurality of times corresponding to the work by an unsupervised learning method which does not require the designation of the number of clusters, thereby segmenting the work into the segmented tasks.

8. A task segmentation method in a task segmentation device that segments work performed by a robot remotely operated in accordance with operation commands from an operator terminal into segmented tasks, which are simple tasks, comprising: a step of acquiring state variables including an operation command value included in the operation command and a sensor value indicating a detection result acquired by a sensor that detects the state of the robot; and a step of using at least the state variables as target data for classification, and clustering the target data at each of a plurality of times corresponding to the work by an unsupervised learning method that does not require the specification of the number of clusters, thereby segmenting the work into the segmented tasks.

9. A program for a computer system that segments an operation performed by a robot remotely controlled in accordance with an operation command from an operator terminal into segmented tasks, which are simple tasks, comprising: a step of acquiring state variables including an operation command value included in the operation command and a sensor value indicating a detection result acquired by a sensor that detects the state of the robot; and a step of segmenting the operation into the segmented tasks by using at least the state variables as target data for classification and clustering the target data at each of a plurality of times corresponding to the operation by an unsupervised learning method that does not require the specification of the number of clusters.

Images