Operation system and operation method

Abstract

To provide an operation system and an operation method which can improve operability.SOLUTION: An operation system according to the embodiment comprises a predicting part that determines a predicted value of motion of an operator at a time when a predetermined predicted time elapses after a current time, from a biological signal of the operator, using a predetermined machine learning model, and a control part that controls motion of a robot on the basis of the predicted value. An operation method according to the embodiment includes a step of determining a predicted value of motion of the operator at the time when the predetermined predicted time elapses after the current time, from the biological signal of the operator, using the predetermined machine learning model and a step of controlling motion of the robot on the basis of the predicted value.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an operation system and an operation method. [Background technology]

[0002] There has been proposed an operation system that receives an operator's action and drives a robot according to the received action. The operation system may be configured as a remote control system that transmits the operator's action to a robot located at a remote location by communication. In such an operation system, the robot's action is delayed from the operator's action due to communication delays, control delays, and the like. The robot's action is captured by a camera or the like and presented to the operator using a head-mounted display, a stationary monitor screen, or the like. In this case, image delays also occur due to sampling during image capture, communication delays, and the like. The operator visually checks the presented image and determines the next action. If a delay occurs in the action understood by visually checking the image, the operator may be forced to slow down his or her own operation. In such a case, operation efficiency decreases and the operator may not feel a sense of ownership in operating the robot.

[0003] In response to this, information obtained by measuring a person's movements may be extrapolated using a method such as a Kalman filter to predict future movements. For example, Patent Document 1 describes a remote control communication system in which a relay node arranged on a communication path connecting a master device and a slave device corrects sensor information based on delay information, the most recent sensor information, and the most recent operation command information. Motion prediction is effective for movements at a constant speed. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2001-54507 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, even if the technique described in Patent Document 1 is used, the robot's movements are delayed compared to the operator's movements. If the operator's movement speed changes, the robot's movements at a certain point in time will differ from the operator's movements. This can reduce operation efficiency and cause the user to lose a sense of agency. An aspect of the present invention has been made in consideration of the above-mentioned points, and an object of the present invention is to provide an operation system and an operation method that can improve operability. [Means for solving the problem]

[0006] (1) The present invention has been made to solve the above problems, and one aspect of the present invention is A detection unit that detects myoelectric signals of the upper arm and hand as biosignals of the operator, an image capturing unit that captures images showing the operating environment of the robot, Based on the biological signal, the operator's condition after a predetermined predicted time has elapsed from the present time is Hand The predicted value of the action is calculated using a given machine learning model. Predict A prediction unit and a prediction unit for predicting at least the image and the operator Hand an estimation unit that estimates a second motion of the motion that the operator can perform based on a motion signal indicating the motion; and the second operation A control unit that controls the operation of the robot based on the The estimation unit estimates, as the second motion, a motion that is more minute than the predicted value. It is an operating system. (2) Another aspect of the present invention is the operation system according to (1), further comprising a gaze direction detection unit that detects a gaze direction of the operator, and the estimation unit may detect an object of interest appearing in the gaze direction from the image, and further estimate the second action based on the object of interest. .

[0009] (3) Another aspect of the present invention is the operation system according to (1) or (2), further comprising: detecting an electroencephalogram as the biological signal. Electroencephalogram A detection unit may be provided.

[0010] (4) Another aspect of the present invention is the (3) The operation system may further include a transmission path for transmitting the predicted value.

[0011] (5) Another aspect of the present invention is the (4) Any of the operating systems ,beforeA display unit for displaying the image may also be provided.

[0012] (6) Another aspect of the present invention is the (5) The operation system may further include a model learning unit that determines parameters of the machine learning model so as to reduce a difference between the predicted value determined from the biological signal delayed by a delay amount equivalent to the predicted time and an actual measured value of the current movement of the operator.

[0013] (7) Another aspect of the present invention is The robot is provided with a detection unit that detects myoelectric signals from the upper arm and hand as biosignals of the operator, and an image capture unit that captures images showing the operating environment of the robot. 1. A method for operating an operating system, comprising: The above A predicted value of the operator's movement after a predetermined time has elapsed from the present is calculated based on the biosignal using a predetermined machine learning model. Predict The first step is to A second step of estimating a second movement of the movement that the operator may perform based on at least the image and a movement signal indicating a hand movement of the operator; The predicted value and the second operation The robot's movement is controlled based on the 3 Steps and The second step estimates a motion that is more minute than the predicted value as the second motion. This is how it works. Effect of the Invention

[0014] As mentioned above (1)、(7) According to the configuration, the robot's movement is controlled based on the operator's biosignal that precedes the operator's movement. This reduces the delay until it is possible to confirm that the operator's movement has been reflected in the robot's movement. This improves the operability of the robot.

[0015] As mentioned above (1)、(7) According to the configuration of the present invention, the action intended by the operator is estimated as the second action from the action of the operator and the operating environment of the robot. By supplementing the operation of the robot with the second action, the operator's intention is reflected in the robot's action. This further improves the operability of the robot.

[0016] As mentioned above (1)、(7) According to the configuration, myoelectric signals that occur prior to a movement can be acquired without the operator being aware of it, and can be used to predict the movement.

[0017] As mentioned above (3) According to this configuration, electroencephalograms that occur prior to a movement can be acquired without the operator being aware of them, and can be used to predict the movement.

[0018] As mentioned above (4) According to this configuration, even if the operation device is separated from the robot, it is possible to control the operation of the robot 20.

[0019] As mentioned above (5) According to the configuration of the present invention, an image showing the operating environment of the robot captured by the image capturing unit is displayed on the display unit, so that the operator can visually confirm the status of the robot even if the operator is separated from the robot.

[0020] As mentioned above (6) According to this configuration, a machine learning model for estimating a movement from a biosignal can be learned based on a movement performed by an operator and the biosignal that occurs prior to the movement. [Brief description of the drawings]

[0021] [Figure 1] 1 is a schematic block diagram showing an example of the configuration of an operation system according to a first embodiment. [Diagram 2] FIG. 13 is a diagram showing an example of EMG and operator motion. [Diagram 3] FIG. 11 is a schematic block diagram showing an example of the configuration of an operation system according to a second embodiment. [Figure 4] FIG. 1 is a schematic block diagram showing an example of the configuration of a model learning system. [Diagram 5] FIG. 1 is a diagram illustrating an example of the configuration of a machine learning model. [Figure 6] FIG. 2 illustrates an example of a hardware configuration. [Figure 7] 1A and 1B are diagrams illustrating examples of perception of biosignals, operator actions, and robot actions. [Figure 8] 13A and 13B are diagrams illustrating other examples of perception of biosignals, operator actions, and robot actions. [Figure 9]FIG. 13 is a diagram illustrating an example of an estimation of an operator's action. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] <First embodiment> The first embodiment will be described below with reference to the drawings. FIG. 1 is a schematic block diagram showing an example of the configuration of an operation system S1a according to this embodiment. The operation system S1a determines a predicted value of the operator's movement at a predicted time from the operator's biosignal using a predetermined machine learning model. The predicted time corresponds to a time after a predetermined predicted time has elapsed from that point in time (present). The operation system S1a controls the movement of the robot 20 based on the predicted value. The operator can control the position and shape of the robot 20 by moving himself.

[0023] The operation system S1a includes an operation device 10a, a robot 20, and a photographing unit 30. The operation device 10a and the robot 20 are connected via a transmission line C. p Various data can be transmitted and received wirelessly or by wire via the transmission line C. p is not limited to a data bus (base line), and may be configured to include a communication network in part or in whole. The image capturing unit 30 captures an image showing the robot 20 and its surroundings within its own field of view. The image shows the operating environment of the robot 20. The image capturing unit 30 transmits image data showing the captured image to the operation device 10a via a transmission line C. p The image capturing unit 30 is, for example, a digital video camera.

[0024] The operation device 10a directly or indirectly receives a motion of the operator, who is a user, and provides a motion signal indicating the received motion to the robot 20. The operation device 10a detects a biosignal generated in the operator, and generates a motion signal indicating a predicted value of the operator himself at a predicted time from the detected biosignal. The operation device 10a includes a detection unit 110, a prediction unit 120, and a display unit 140.

[0025] The operation device 10a may be configured as a so-called wearable device, including a wearing tool that can be worn on the human body. The wearing tool is provided with members that constitute the above-mentioned functional parts. For example, the operation device 10a may be provided with gloves and a head band as wearing tools. The gloves can be worn on the hand, which is a part of the human body. The gloves are provided with a detection unit 110, and configured as a data glove. The number of data gloves is one or two. Each data glove is used to operate a manipulator that constitutes the robot 20. The number of data gloves may be equal to the number of manipulators. The headband can be worn on the head, which is a part of the human body. The headband may be provided with a display unit 140, and configured as a head mounted display (HMD).

[0026] The detection unit 110 includes a biosignal detection unit that detects a biosignal generated in the operator. The detection unit 110 outputs the detected biosignal to the prediction unit 120. The detection unit 110 acquires, for example, an electromyogram (EMG) as the biosignal. The electromyogram indicates a time change in the myoelectric potential in the upper arm and hand that are part of the living body, that is, the myoelectric potential at each time. The electromyogram corresponds to one form of the myoelectric signal. In this embodiment, the detection unit 110 includes a surface electrode. The surface electrode can be attached to the surface of the living body to acquire the myoelectric signal non-invasively. The detection unit 110 may perform a predetermined pre-processing on the detected myoelectric potential and output a myoelectric signal indicating the processed myoelectric potential to the prediction unit 120.

[0027] The prediction unit 120 predicts the motion of the operator at a predicted time, which is a time after a predetermined predicted time has elapsed from the present time, as a predicted value using a predetermined machine learning model based on the biological signal input from the detection unit 110. The motion is represented as a time series of postures at each time. The prediction unit 120 transmits a motion signal indicating the predicted value to the robot 20. For example, a joint motion may be used as the motion of the operator. The joint motion is represented using an angle formed by two parts that are connected to each other among a plurality of parts of the human body at each time based on a musculoskeletal dynamics model. The musculoskeletal dynamics model is a type of human body model that indicates the posture or motion of the human body. The plurality of parts include the wrist, palm, and joints of each finger. The joints are parts that form part of a finger and are connected to other parts at one or both ends via joints. The operator's motion may include, instead of or in addition to the joint motion, the position of a part of the body where each data glove is worn (e.g., wrist) or the center of gravity between parts as a representative position. The machine learning model may be any kind of mathematical model, such as a neural network or a random forest, as long as it is a mathematical model that can estimate motion from a biological signal.

[0028] In general, a certain delay occurs between the generation (firing) of myoelectric potential and the exertion of muscle tension causing a movement. This delay time is called electromechanical delay (EMD). In the present application, this delay time may be called delay time ΔT5. In the example shown in FIG. 2, EMG starts generating myoelectric potential at time T, and movement starts at time T+ΔT5 after the delay time ΔT5 has elapsed (curve Rm). As will be described later, by analyzing biosignals, it is possible to predict a movement (curve Pm) before the actual movement occurs. The delay time ΔT5 is typically about 100 ms to 300 ms.

[0029] The display unit 140 receives image data from the photographing unit 30. The display unit 140 includes a display that displays an image based on the received image data. The operator can check the operating status of the robot 20 by visually checking the image displayed on the display unit 140.

[0030] The robot 20 operates based on an operation signal provided from the operating device 10a. The robot 20 includes a plurality of members, and is provided with an actuator that changes the angle between two mutually adjacent members (sometimes referred to as a "member pair" in this specification), thereby changing the position and posture of the robot 20 as a whole. An actuator is provided for each joint connecting each two members. The robot 20 reproduces an operation corresponding to the operator's operation indicated by the provided operation signal. The robot 20 may be a single-arm robot having one manipulator, or a dual-arm robot having two manipulators. Each manipulator includes one robot arm and one end effector. The end effector is also called a robot hand. For example, assume that the robot 20 is a dual-arm robot and the operator thrusts his / her right arm forward. In this case, the robot 20 operates the robot arm so that the end effector corresponding to the operator's right hand advances in the forward direction. When the operator makes a clenched grip with his / her left hand, the robot 20 deforms a plurality of fingers constituting an end effector corresponding to the operator's left hand so that the tips of the fingers face each other.

[0031] The robot 20 includes a control unit 210 and a driving unit 220 . The control unit 210 controls the operation of the robot 20 based on the operation signal provided from the operating device 10a. The control unit 210 controls the position and posture of the members constituting the robot 20 so that the end effector moves according to the operation indicated by the operation signal, for example. More specifically, the control unit 210 converts, for example, a position in the robot coordinate system corresponding to the representative position of the operator indicated by the operation signal, and sets it as a target value of the position of the end effector. The control unit 210 may set a position in the robot coordinate system corresponding to the position of each part indicated by the joint operation as a target value of the position of the member of the robot 20 corresponding to that part. The control unit 210 solves a known inverse kinematic problem to determine the angle formed by each pair of members during the period until the end effector or the member reaches the target position as a target value. The control unit 210 outputs a control signal indicating the determined target value to the drive unit 220.

[0032] The driving unit 220 includes an actuator and a controller for each member pair. The controller drives the actuator so that the current value of the angle between the member pair at that time approaches the target value of the angle indicated by the control signal. The controller can use any method as long as it controls so as to reduce the residual between the current value and the target value. The controller can be, for example, any of a PID controller, a PD controller, etc. The control unit 210 may set a torque as a target value for each member pair. In this case, the controller constituting the drive unit 220 drives the actuator so that the current value of the torque applied to the member pair at that time approaches the target value of the angle indicated by the control signal.

[0033] In the example shown in FIG. 1, the delay that occurs between the operator's action, the action being reflected in the action of the robot 20, and the image showing the action of the robot 20 being displayed on the display unit 140 can be attributed to the following factors: (1) Communication delay ΔT1: The communication delay ΔT1 is the time it takes for a motion signal indicating the motion of the operator to be transmitted from the prediction unit 120 to the control unit 210. In the conventional method, the detection unit 110 acquires a motion signal indicating the motion of the operator and provides the acquired motion signal to the robot 20. (2) Control delay ΔT2: The control delay ΔT2 is the time from when an operation signal is acquired until it is reflected in the operation of the robot 20. In the example shown in Fig. 7, the time from the operation of the operator to the operation of the robot corresponds to the sum of the transmission delay ΔT1 and the control delay ΔT2.

[0034] (3) Monitoring Delay T3: The monitoring delay ΔT3 is the time from when the image capturing unit 30 captures an image showing the operation of the robot 20 until when image data showing the captured image is generated. (4) Transmission delay ΔT4: The transmission delay ΔT4 is the time it takes for image data to be transmitted from the image capture unit 30 to the display unit 140. In the example shown in Fig. 7, the time from the robot motion to the recognition of the robot motion corresponds to the sum of the monitoring delay ΔT3 and the transmission delay ΔT4. Therefore, when the operation device 10a acquires an operation signal indicating the operator's motion as in the conventional method, the total delay time ΣT is the sum of these delays ΔT1+ΔT2+ΔT3+ΔT4.

[0035] In contrast, in this embodiment, the prediction unit 120 predicts the movement after a delay time ΔT5 from that point in time using the biosignal detected by the detection unit 110, and provides the robot 20 with a movement signal indicating the predicted movement. Therefore, the delay time until the operation is reflected in the robot movement and recognized by the operator is offset by the delay time ΔT5. In the example shown in FIG. 8, the period from the operator movement predicted from the EMG as the biosignal to the robot movement corresponds to a time ΔT1+ΔT2-ΔT5 obtained by subtracting the delay time ΔT5 from the total value of the transmission delay ΔT1 and the control delay ΔT2. The delay time ΔT5 corresponds to the EMD, and is, for example, 200 ms. Therefore, the total delay time ΣT is reduced to a time ΔT1+ΔT2+ΔT3+ΔT4-ΔT5 obtained by subtracting the delay time ΔT5 from the total delay time in the conventional method. Therefore, the operability by the operator is improved, and the sense of self-agency regarding the movement of the robot 20 can be improved.

[0036] <Second embodiment> Next, the second embodiment will be described, focusing mainly on the differences from the above embodiment. The same reference numerals will be used for the processes and configurations common to the above embodiment, unless otherwise specified, and the description thereof will be incorporated. FIG. 3 is a schematic block diagram showing an example of the configuration of an operation system S1b according to this embodiment. The operation system S1b includes an operation device 10b, a robot 20, and an image capturing unit 30. The operation device 10b includes a detection unit 110, a prediction unit 120, a display unit 140, an estimation unit 150, and an integration unit 160.

[0037] The detection unit 110 includes a motion detection unit that detects the motion of the operator in addition to the biosignal. The detection unit 110 outputs a motion signal indicating the detected motion to the estimation unit 150. The detection unit 110 includes, for example, an acceleration sensor for detecting the position of each part of the operator. The parts to be detected include at least the palm and the knuckles of each finger. The parts may further include the wrist. Each acceleration sensor may be installed in a data glove. Each acceleration sensor may be installed in a position that contacts the part to be detected for acceleration when the data glove is worn on the hand of the operator. The detection unit 110 may include, instead of the acceleration sensor, a photographing unit that photographs an image of the operator's body and a motion detection unit that optically detects the position of each part of the operator using the photographed image (motion capture). However, a marker that reflects light having a predetermined wavelength component is attached to each part of the operator.

[0038] The detection unit 110 may detect the gaze direction of the operator. The detection unit 110 outputs gaze information indicating the detected gaze direction to the prediction unit 120. The detection unit 110 includes, for example, an eye tracker for detecting the gaze direction of the operator. The gaze detection unit may be attached to, for example, a head mounted display. The gaze detection unit may be installed at a position where it can acquire images of the pupils of both eyes that are targets for gaze detection when the head mounted display is attached to the head of the operator.

[0039] The detection unit 110 may further include an acceleration sensor for detecting the orientation and position of the operator's head. The detection unit 110 outputs operator position information indicating the detected orientation and position to the prediction unit 120. The detection unit 110 includes, for example, a six-axis acceleration sensor for detecting the orientation and position of the operator's head. The six-axis acceleration sensor detects acceleration in each of three mutually orthogonal directions in a three-dimensional space and axial acceleration in each of the three directions. The detection unit 110 can determine the orientation by performing a second-order integration of the detected axial acceleration in the three directions. The detection unit 110 can determine the position by performing a second-order integration of the detected acceleration in the three directions. The acceleration sensor may be installed at a position that contacts the face of the head that is the target of detection of acceleration or axial acceleration when the head-mounted display is worn on the operator's head.

[0040] The estimation unit 150 receives an action signal from the detection unit 110 and image data from the imaging unit 30. The estimation unit 150 estimates a second action that the operator may perform based on the action signal and the image data. The second action is a more detailed action than the global action (e.g., movement of the hand) estimated mainly by the prediction unit 120. The second action includes, for example, the action of each finger, and fine adjustment of one or both of the position and direction of the hand. The estimation unit 150 outputs a second action signal indicating the estimated second action to the integration unit 160.

[0041] For example, when image data showing a situation in which the end effector of the robot 20 enters an area closer to an object than its size and an action signal showing the start of the action of multiple fingers are input, the estimation unit 150 estimates an action in which the tips of the fingers face each other as the second action. In such a situation, the operator is likely to make the end effector grasp the object by clenching his / her hand. For example, when image data showing a situation in which the end effector of the robot 20 grasps an object placed on a table and an action signal showing the start of the action of multiple fingers are input, the estimation unit 150 estimates an action in which the tips of the fingers repel each other as the second action. In such a situation, the operator is likely to make the end effector release the grasped object by spreading his / her fingers.

[0042] The estimation unit 150 may further use the line-of-sight information input from the detection unit 110 to determine the second action. The estimation unit 150 estimates the line-of-sight direction indicated in the line-of-sight information as the direction of interest. The direction of interest means the direction in which the operator is interested. The estimation unit 150 can convert the line-of-sight direction indicated in the line-of-sight information into a direction in a robot environment coordinate system based on the position and orientation of the image capture unit 30, based on the operator position information input from the detection unit 110 and the position, orientation, and field of view of the image capture unit 30, and determine the converted direction as the direction of interest. When the image indicated in the image data includes images of multiple objects, the estimation unit 150 can determine an object whose image appears in the direction of interest among the multiple objects as the object of interest. The estimation unit 150 may determine the second action based on the operation signal, the image data, and the object of interest, and ignore objects other than the object of interest. This avoids unnecessary estimation of the second action for objects other than the object of interest that the operator does not intend.

[0043] In the estimation unit 150, for example, second action data indicating a relationship between image data and an action signal as input information, and a second action as output information is set in advance. The estimation unit 150 can determine a second action corresponding to the input information to itself by referring to the second action data. In the estimation unit 150, a parameter set for estimating the second action as output information from image data and an action signal as input information using a second machine learning model may be set in advance. The estimation unit 150 can determine the second action based on the input information using the second machine learning model under the set parameter set. The input information may further include gaze information.

[0044] The prediction unit 120 outputs an action signal indicating the predicted value predicted by the prediction unit 120 to the integration unit 160 instead of the robot 20. The integration unit 160 receives an action signal from the prediction unit 120 and a second action signal from the estimation unit 150. The integration unit 160 integrates a first action indicated by the action signal with a second action indicated by the second action signal to form an integrated action. As long as the first action and the second action are consistent and do not contradict each other, the integrated action is also expressed as a joint action. The integration unit 160 outputs an action signal indicating the integrated action to the robot 20.

[0045] In addition, when a contradiction occurs between the first action and the second action, or when an adverse effect is expected from the execution of the first action and the second action, the integration unit 160 may select one of them and reject the other. For example, when the hand movement of the operator indicated by the first action corresponds to the movement of the end effector gripping the object toward the object in a direction away from the table, and the second action is the action of pushing the fingertips away from each other (corresponding to the release from the gripping state of the object), the integration unit 160 may select the first action and reject the second action. In that case, when the end effector grips the object and leaves the table, the object is prevented from falling off by the second action. In addition, when the first action indicates the start of the movement of the hand that was stationary (corresponding to the start of the end effector), and the second action indicates the middle of the action of bringing the fingertips toward each other (corresponding to the process of gripping the object), the integration unit 160 may reject the first action and give priority to the second action. In this case, it is possible to avoid moving the object before the end effector has completely grasped it.

[0046] The integration unit 160 may be set in advance with selection data indicating, for example, the first motion and the second motion and whether or not each of them is required. The integration unit 160 refers to the selection data to specify whether or not each of the first motion and the second motion is required. When the integration unit 160 specifies that each of the first motion and the second motion is required, it generates an operation signal indicating an integrated motion that integrates the first motion and the second motion, and outputs the generated operation signal to the robot 20. When the integration unit 160 specifies that one of the first motion and the second motion is required and specifies that the other motion is not required, it generates an operation signal indicating one of the motions, and outputs the generated operation signal to the robot 20. When the integration unit 160 specifies that both the first motion and the second motion are not required, it does not generate an operation signal indicating either motion.

[0047] The control unit 210 of the robot 20 controls the operation of the robot 20 based on the operation signal provided from the operating device 10b. The control unit 210 may receive an operation signal indicating an integrated operation. In this case, the control unit 210 is instructed to perform an operation that combines an operation estimated from a biosignal and a second operation estimated according to the operating environment, and commands the driving unit 220 to perform such an operation. For example, a global operation is instructed as the estimated operation, and a fine operation is instructed as the second operation. The global operation may be, for example, translational movement of the wrist, which has a relatively high moving speed. The second operation may be, for example, alignment of the individual fingers with respect to an object, grasping, and release from the grasping state. Therefore, the operator can instruct a fine operation that is more in line with the operator's intention in the operating environment of the robot 20.

[0048] (Model learning) Next, learning of the machine learning model will be described. Fig. 4 is a schematic block diagram showing an example of the configuration of a model learning system S2. The model learning system S2 is a system for learning the machine learning model used by the prediction unit 120. The model learning system S2 includes a motion detection unit 112, a biosignal detection unit 114, and a model learning unit 40. The motion detection unit 112 detects motion of the human body. The biosignal detection unit 114 detects biosignals generated in the human body. The human body to be detected for motion may be the body of a person other than the operator, but is preferably the body of the operator himself. The model learning unit 40 determines a parameter set of the machine learning model so as to reduce (minimize) the difference between the motion (output 2) estimated using the machine learning model based on the detected biosignal and the detected motion (output 1). With this configuration, the model learning unit 40 learns the machine learning model using supervised data including a plurality of data pairs consisting of known biosignals and motion signals (supervised training).

[0049] The motion detection unit 112 detects motion of the human body in the same manner as the detection unit 110. The motion detection unit 112 includes, for example, an imaging unit that captures an image showing the operator's torso, and a motion analysis unit that optically detects the position of each part of the operator using the image captured by the imaging unit. The motion detection unit 112 outputs a motion signal indicating the detected motion of the human body to the model learning unit 40.

[0050] The biosignal detection unit 114 detects a biosignal occurring in the human body in the same manner as the detection unit 110. The biosignal detection unit 114 includes, for example, a biosignal sensing unit 114a, a biosignal acquisition unit 114b, and a preprocessing unit 114c.

[0051] The biosignal sensing unit 114a is placed on or in close proximity to a predetermined part of the human body and senses a biosignal occurring at that part. The biosignal sensing unit 114a corresponds to, for example, the above-mentioned surface electrode. The biosignal acquiring unit 114b acquires the biosignal sensed by the biosignal sensing unit 114a. The biosignal acquiring unit 114b outputs the acquired biosignal to the preprocessing unit 114c. The biosignal acquiring unit 114b is, for example, an EMG capture. The EMG capture amplifies, for example, the potential generated in the biosignal sensing unit 114a and acquires a biosignal indicating the amplified potential.

[0052] The preprocessing unit 114c performs preprocessing on the biosignal input from the biosignal acquisition unit 114b, and outputs the processed biosignal obtained by the preprocessing to the model learning unit 40. The preprocessing includes, for example, one or both of rectification and filtering. Rectification is a process in which, when the potential at each time is equal to or greater than a predetermined potential threshold (for example, 0 V), the potential is set as the potential of the processed biosignal, and, when the potential at each time is less than the predetermined potential threshold, the threshold is set as the potential of the processed biosignal. Filtering is a process in which components of a significant frequency band determined in advance as the biosignal are maintained, and components of other frequency bands are suppressed or removed. This removes noise mixed into the biosignal.

[0053] The model learning unit 40 includes a delay unit 410 , a motion analysis unit 420 , and a parameter update unit 430 . The delay unit 410 delays the biological signal input from the biological signal detection unit 114 by a delay amount (for example, 200 ms) corresponding to a predetermined predicted time. The delay unit 410 outputs the delayed biological signal to the parameter update unit 430.

[0054] The motion analysis unit 420 analyzes the motion indicated by the motion signal input from the motion detection unit 112 using a known musculoskeletal dynamics model, and outputs a motion signal indicating the analyzed motion to the parameter update unit 430. For example, the motion analysis unit 420 identifies the positions of each of a plurality of parts constituting the human body from the input motion signal, and determines the angle between the two parts for each pair of two mutually connected parts. The motion analysis unit 420 can generate a motion signal indicating the determined angle.

[0055] The parameter update unit 430 uses the biological signal input from the delay unit 410 as input information, and regressively updates the parameter set of the machine learning model so that the magnitude of the difference between the estimated value of the human body movement estimated as output information using a predetermined machine learning model and the observed value of the human body indicated in the movement signal input from the motion analysis unit 420 becomes smaller. The observed value and the estimated value correspond to the output 1 and the output 2 shown in the figure, respectively. A loss function indicating the magnitude of the difference between the observed value and the estimated value may be, for example, any of the sum of squared differences (SSD), cross entropy, etc.

[0056] The parameter update unit 430 may repeat the update of the parameter set to be updated a preset number of times, or may repeat the update until the parameter set converges to a certain value. The parameter update unit 430 can determine whether or not the convergence has occurred based on, for example, whether or not the magnitude of the difference between the values ​​of some or all of the parameters constituting the parameter set before and after the update is equal to or less than a predetermined threshold value. The parameter update unit 430 sets the parameter set determined by the update in the prediction unit 120. The machine learning model used in the parameter update unit 430 may be a mathematical model of the same type as the machine learning model used in the prediction unit 120.

[0057] The above-mentioned detection unit 110 may be used as the motion detection unit 112 and the biological signal detection unit 114, or may be provided separately from the detection unit 110. The model learning unit 40 may be provided in either or both of the operation systems S1a and S1b, or may be realized in a device independent of either of the operation systems S1a and S1b. Furthermore, when the model learning unit 40 is provided in either one or both of the operation systems S1a and S1b, the model parameters of the machine learning model may be updated (online learning) using the biosignals and the motion signals input from the detection unit 110 during the operation of the robot 20. This allows the motion prediction value estimated based on the biosignals to be calibrated so as to approach the predicted value of the current motion, thereby further improving the prediction accuracy of the motion signals.

[0058] (Machine learning model) Next, an example of a machine learning model will be described. FIG. 5 shows an example of the configuration of a CNN (Convolutional Neural Network), which is a type of neural network. A CNN is a representative example of a machine learning model used in so-called deep learning. A CNN includes one input layer, multiple intermediate layers (hidden layers), and one output layer. The CNN illustrated in FIG. 5 includes an input layer Il, L layers (L is an integer equal to or greater than 2) of intermediate layers Ml-1 to Ml-L, and an output layer Ol. Each layer includes a predetermined number of nodes. The multiple intermediate layers include at least one convolution layer and one pooling layer.

[0059] Each node of the input layer Il outputs the input value input to itself to at least one node of the next layer. For example, in the prediction unit 120 and the parameter update unit 430, each sample value within a predetermined period (e.g., 10-100 ms) constituting a biological signal is input to a node corresponding to the sample value. Each node of the output layer Ol outputs an input value input from at least one node of the previous layer to the outside. The prediction unit 120 and the parameter update unit 430 output each element value expressing a movement (for example, an angle between parts) from the node corresponding to that element value.

[0060] A convolution layer is preset with a number of kernels. The number of kernels corresponds to the number of kernels used for processing (e.g., calculation) each input value. The number of kernels is usually smaller than the number of input values. A kernel refers to a processing unit for calculating one output value at a time. The output value calculated in a layer is used as an input value for the next layer. A kernel is also called a filter. The kernel size refers to the number of input values ​​used in one processing in the kernel. The kernel size is usually an integer of 2 or more.

[0061] A convolutional layer is a layer that calculates a convolved value by performing a convolution operation for each kernel on the input value input from the previous layer to each of multiple nodes, and calculates a corrected value by adding the calculated convolved value and a bias value. The convolutional layer calculates the function value of a predetermined activation function for the calculated corrected value, and outputs the calculated output value to the next layer. Note that one or more input values ​​are input to each node of the convolutional layer from the previous layer, and an independent convolution coefficient is used for each input value to calculate the convolved value at each node. The parameters of the convolutional coefficient, bias value, and activation function are part of one set of model parameters.

[0062] As the activation function, for example, a rectified linear unit, a sigmoid function, etc. can be used. The rectified linear unit is a function that sets a threshold (e.g., 0) as an output value for an input value below the threshold, and outputs an input value that exceeds the threshold as is. Therefore, this threshold can be part of a set of model parameters. In addition, for a convolutional layer, whether or not it is necessary to refer to an input value from a node in the previous layer, and whether or not it is necessary to output an output value to a node in the next layer can also be part of a set of model parameters.

[0063] A pooling layer is a layer that has nodes that determine one representative value from the input values ​​input from multiple nodes in the previous layer and output the determined representative value as an output value to the next layer. The representative value is, for example, a value that statistically represents multiple input values, such as the maximum value, average value, or mode. A stride is set in advance for the pooling layer. The stride indicates the range of adjacent nodes in the previous layer that refer to the input value for one node. Therefore, the pooling layer can also be considered as a layer that provides the output value to the next layer by reducing (down-sampling) the input value from the previous layer to a lower dimension.

[0064] Next, an example of estimating an operator's motion using a machine learning model will be described. FIG. 9 is a diagram showing an example of estimating an operator's motion. The vertical and horizontal axes in FIG. 9 indicate motion and time, respectively. Section A shows an estimated value of a motion estimated using a machine learning model from a biosignal used in model learning, and an observed value of the motion. The estimated value is approximately equal to the observed value. The estimated value precedes the observed value by a time equivalent to the predicted time. This indicates that the motion at the predicted time after the current time has elapsed is predicted. This is because a delay equivalent to the predicted time is added to the biosignal used as input information to the machine learning model in model learning, and the motion is synchronized with the motion used as output information. Section B shows an estimated value using a machine learning model from a biosignal not used in model learning, and an observed value of the motion. Although an error is added to the estimated value, it has the same tendency of change over time as the observed value. The estimated value precedes the observed value by a time equivalent to the predicted time, but its period and amplitude are the same as the observed value.

[0065] In the above description, the case where EMG is used as the biosignal is taken as an example, but the present invention is not limited to this. Any biosignal that has a correlation with the movement of the human body and occurs prior to the movement can be applied. For example, an electroencephalograph (EEG) can be used as such a biosignal. In this case, the detection unit 110 may be configured as a brain machine interface (BMI) for non-invasively detecting the brain wave. The BMI includes scalp electrodes for detecting changes in potential caused by brain activity. The scalp electrodes may be installed on the head mounted display. The scalp electrodes may be installed at a position that contacts the scalp of the operator when the operator wears the head mounted display.

[0066] As described above, the operation systems S1a and S1b according to the above embodiments include a prediction unit 120 that determines a predicted value of the operator's movement after a predetermined predicted time has elapsed from the present based on the operator's biosignal using a predetermined machine learning model, and a control unit 210 that controls the movement of the robot 20 based on the predicted value. According to this configuration, the movement of the robot 20 is controlled based on the operator's biological signal that occurs prior to the operator's movement. Since the delay until it is confirmed that the operator's movement is reflected in the movement of the robot 20 is reduced, the operability of the robot 20 can be improved. Furthermore, the operator's sense of self-agency can be enhanced.

[0067] In addition, the operation system S1b includes an estimation unit 150 that estimates a second action that the operator may perform based on at least an image showing the operating environment of the robot 20 and an action signal showing the operator's action, and the control unit 210 may further control the action of the robot 20 based on the second action. According to this configuration, the action intended by the operator is estimated as the second action from the action of the operator and the operating environment of the robot 20. By supplementing the operation of the robot 20 with the second action, the intention of the operator is reflected in the action of the robot 20. Therefore, the operability of the robot 20 is further improved.

[0068] The operation systems S1a and S1b may further include a detection unit 110 that detects a myoelectric signal as a biosignal. According to this configuration, myoelectric signals that occur prior to a movement can be acquired without the operator being aware of it, and can be used to predict the movement.

[0069] The operation systems S1a and S1b may further include a detection unit 110 that detects brain waves as a biological signal. According to this configuration, electroencephalograms that occur prior to a movement can be acquired without the operator being aware of them, and can be used to predict the movement.

[0070] The operation systems S1a and S1b may further include a transmission path Cp for transmitting to the control unit 210 a predicted value estimated from a biological signal. With this configuration, even if the operation devices 10a and 10b are separated from the robot 20, the operation of the robot 20 can be controlled.

[0071] The operation systems S1a and S1b may include an image capturing unit 30 that captures an image showing the operating environment of the robot 20, and a display unit 140 that displays the image. With this configuration, an image showing the operating environment of the robot 20 captured by the image capturing unit 30 is displayed on the display unit 140. Therefore, even if the operator is separated from the robot 20, the operator can visually confirm the state of the robot 20.

[0072] The operation systems S1a and S1b may be provided with a model learning unit 40 that determines parameters of a machine learning model so as to further reduce the difference between a predicted value of the operator's movement determined from a biological signal delayed by a delay amount corresponding to the predicted time and an actual measured value of the current operator's movement. With this configuration, it is possible to learn a machine learning model for estimating a movement from a biosignal, based on a movement performed by an operator and the biosignal that occurs prior to the movement.

[0073] Although an embodiment of the present invention has been described above with reference to the drawings, the specific configuration is not limited to the above, and various design changes and the like are possible within the scope that does not deviate from the gist of the present invention.

[0074] For example, in the operation systems S1a and S1b, one or both of the image capturing unit 30 and the display unit 140 may be omitted. p The communication network may not be included in the system. Such a configuration may be acceptable as long as the operator can be relatively close to the robot 20 and can visually observe the operating environment of the robot 20. Instead of including the control unit 210 in the operation device 10a, the control unit 210 may be omitted in the robot 20. As long as the prediction section 120 is connected to the detection section 110 via a wired or wireless connection so as to be capable of transmitting various types of data, the detection section 110 may be omitted from the operation devices 10a and 10b. The operation devices 10a and 10b may also include a model learning unit 40. The operating devices 10a and 10b do not necessarily have to be configured to be wearable on the human body. The operating devices 10a and 10b may be, for example, stationary types.

[0075] The above-mentioned devices (e.g., the operation devices 10a and 10b, the robot 20, and the model learning device) may be realized by hardware including dedicated components, or may be configured as a computer including general-purpose components. The computer 50 illustrated in FIG. 6 includes a processor 52, a drive unit 56, an input / output unit 58, a ROM 62, and a RAM 64.

[0076] The processor 52 controls processes for implementing the functions of each device and the functions of each part constituting the device. The processor 52 is, for example, a CPU (Central Processing Unit). The drive unit 56 may have the storage medium 54 fixed or removable. The drive unit 56 reads various data stored in the storage medium 54, or stores various data in the storage medium 54. The drive unit 56 is, for example, a solid state drive (SSD) or a hard disk drive (HDD). The storage medium 54 is, for example, a non-volatile memory such as a flash memory or a hard disk.

[0077] The input / output unit 58 inputs or outputs various data wirelessly or via wire to / from other devices. The input / output unit 58 may be connected to other devices via a communication network so that various data can be input / output. The input / output unit 58 may be, for example, an input / output interface, a communication interface, or a combination thereof.

[0078] The ROM (Read Only Memory) 62 is a storage medium that permanently stores a program in which commands for instructing various processes to be executed by each unit of each device are written, various data such as parameters for the execution of the program, and various data acquired by each unit. Note that in this application, executing a process instructed by commands written in the program may be referred to as "executing a program" or "running a program". The RAM (Random Access memory) 64 is a storage medium mainly used as a working area for the processor 52. When the processor 52 is started, it records the programs and parameters stored in the ROM 62 in the RAM 64. The processor 52 then temporarily records in the RAM 64 the results of calculations obtained by the execution of the programs, acquired data, and the like.

[0079] Each of the above devices may have a computer system therein. For example, the processor 52 may be a component of the computer system. The above-mentioned processes are stored in a computer-readable storage medium in the form of a program, and the computer reads and executes the program to perform the processes. The computer system includes software such as an OS (Operation System), device drivers, and utility programs, and hardware such as peripheral devices. The hardware shown in FIG. 6 corresponds to an example of such hardware. The "computer-readable storage medium" refers to a storage device such as a magnetic disk, a magneto-optical disk, a ROM, a RAM, a portable medium such as a semiconductor memory, and a hard disk built into a computer system. Furthermore, the computer-readable storage medium may include a storage device that dynamically holds a program for a short period of time, such as a communication line used when transmitting a program using a network such as the Internet or a communication line such as a telephone line, and a storage device that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. The above-mentioned program may be a program for realizing a part of the above-mentioned functions, or may be a program that can realize the above-mentioned functions in combination with a program already recorded in the computer system, that is, a so-called differential file (differential program).

[0080] In addition, some or all of the above devices may be realized as integrated circuits such as LSI (Large Scale Integration). Each functional block of the above devices may be individually processed, or some or all of them may be integrated into a processor. The integrated circuit method is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. In addition, if an integrated circuit technology that can replace LSI appears due to advances in semiconductor technology, an integrated circuit based on that technology may be used. [Explanation of symbols]

[0081] S1a, S1b...operation system, S2...model learning system, 10a, 10b...operation device, 20...robot, 30...imaging unit, 40...model learning unit, 50...computer, 52...processor, 54...storage medium, 56...drive unit, 58...input / output unit, 62...ROM, 64...RAM, 110...detection unit, 112...motion detection unit, 114...biological signal detection unit, 114a...biological signal sensing unit, 114b...biological signal acquisition unit, 114c...preprocessing unit, 120...prediction unit, 140...display unit, 150...estimation unit, 160...integration unit, 210...control unit, 220...drive unit, 410...delay unit, 420...motion analysis unit, 430...parameter update unit

Claims

1. A detection unit for detecting myoelectric signals of an upper arm and a hand as a biosignal of an operator; an image capturing unit that captures an image showing the operating environment of the robot; A prediction unit that predicts a predicted value of a hand movement of the operator after a predetermined predicted time has elapsed from the present based on the biological signal by using a predetermined machine learning model; an estimation unit that estimates a second motion that the operator may perform based on at least the image and a motion signal indicating a motion of a hand of the operator; a control unit that controls a movement of the robot based on the predicted value and the second movement, The estimation unit estimates, as the second motion, a motion that is more minute than the predicted value. Operation system.

2. A gaze direction detection unit for detecting the gaze direction of the operator, The estimation unit detects an object of interest appearing in the line of sight from the image, and estimating the second motion based on the object of interest. The operating system according to claim 1 .

3. a brain wave detection unit for detecting brain waves as the biological signal; The operating system according to claim 1 or 2.

4. a transmission path for transmitting the predicted value The operating system according to any one of claims 1 to 3.

5. A display unit for displaying the image. The operating system according to any one of claims 1 to 4.

6. a model learning unit that determines parameters of the machine learning model so as to reduce a difference between the predicted value determined from the biological signal delayed by a delay amount corresponding to the predicted time and an actual measured value of the current motion of the operator. The operating system according to any one of claims 1 to 5.

7. A detection unit for detecting myoelectric signals of an upper arm and a hand as a biosignal of an operator; An image capturing unit that captures an image showing the operating environment of the robot.

1. A method for operating an operating system, comprising: A first step of predicting a predicted value of the operator's movement after a predetermined time has elapsed from the present based on the biological signal using a predetermined machine learning model; a second step of estimating a second motion that the operator may perform based on at least the image and a motion signal indicating a motion of the operator's hand; and a third step of controlling a motion of the robot based on the predicted value and the second motion, The second step estimates a motion that is more minute than the predicted value as the second motion. How to operate.

Images