Control system
The control system addresses the divergence of voluntary and autonomous actions by dynamically adjusting weights based on brain wave detection and environment recognition, ensuring operations align with user intent, enhancing control precision and discretion.
Patent Information
- Application Number
- PCT/JP2024/018061
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-17
- Filing Date
- 2024-05-16
- Publication Date
- 2024-11-21
AI Technical Summary
Existing control systems using Brain Machine Interfaces (BMIs) face challenges in accurately reflecting user intention when voluntary and autonomous actions diverge, leading to reduced control discretion and potential operations differing from the user's intent.
A control system incorporating a detection section, first and second determination sections, and a weight determination section to dynamically adjust weights based on brain wave detection, environment recognition, and user intention, harmonizing voluntary and autonomous controls to ensure operations align with user intent.
The system effectively reduces the likelihood of operations differing from user intention by dynamically adjusting weights, thereby improving control precision and alignment with user intent, especially in tasks like robot arm movements and avatar speech.
Smart Images

Figure JP2024018061_21112024_PF_FP_ABST
Abstract
Description
Control System
[0001] The present invention relates to a control system.
[0002] There is known a technology for controlling a controlled object using a BMI (Brain Machine Interface). There is also known a technology for autonomously operating a controlled object. For example, Patent Literature 1 discloses a machine system that reflects, in determining the machine's operation, not only an intended operation based on a user's intention but also an autonomous operation based on the machine's own judgment of the situation. The machine system of Patent Literature 1 can reduce the hassle and burden on a user when operating a machine using a BMI.
[0003] Specifically, the mechanical system of Patent Document 1 uses a BMI to determine a user-intended action, which is an action intended by a user. The mechanical system of Patent Document 1 also determines the autonomous action of the robot based on the robot's motion information and information about the robot's surroundings (e.g., obstacle detection information) acquired via sensors. If the user-intended action and the autonomous action do not match, the mechanical system determines the action to be performed by the robot in accordance with predetermined rule information.
[0004] Specifically, in the mechanical system of Patent Document 1, when the user-intended action and the autonomous action do not match, the weighting coefficient for the autonomous action is made larger than the weighting coefficient for the user-intended action, and a weighted linear sum of the user-intended action and the autonomous action is calculated to determine the action to be performed by the robot. As a result, the hassle and burden on the user when operating a machine using BMI is reduced.
[0005] JP 2010-237859 A
[0006] However, in the mechanical system of Patent Document 1, when the user's intended operation and the autonomous operation do not match, the user's intention is hardly reflected, which tends to reduce the discretion of the control, and the controlled object may perform an operation different from the user's intention.
[0007] The present invention has been made in view of the above-mentioned problems, and its purpose is to provide a control system that makes it difficult for the controlled object to behave in a manner different from the user's intention.
[0008] According to one aspect of the present invention, a control system controls an object to be operated. The control system includes a detection unit, a first determination unit, a second determination unit, and a weight determination unit. The detection unit detects a detection object related to autonomous control of the object to be operated. The first determination unit determines a next action of the object to be operated by voluntary control based on brain waves of an operator operating the object to be operated. The second determination unit determines a next action of the object to be operated by autonomous control based on a detection result by the detection unit. The weight determination unit determines a first weight for a next action of the object to be operated by voluntary control and a second weight for a next action of the object to be operated by autonomous control based on a detection result by the detection unit. The weight determination unit changes the first weight and the second weight according to a detection result by the detection unit.
[0009] In one embodiment, the control system further includes an intention detection unit that detects the intention of the operator based on electroencephalograms of the operator, and the weight determination unit determines the first weight and the second weight based on a result of detection by the detection unit and the intention of the operator.
[0010] In one embodiment, the control system further includes an instruction unit that instructs the control object to take a next action based on the next action determined by the first determination unit, the next action determined by the second determination unit, the first weight, and the second weight.
[0011] In one embodiment, the control system further includes a storage unit. The storage unit stores change conditions for the first weight and the second weight. The change conditions are adjusted based on an evaluation function including, as parameters, control performance and operational feel acquired during training of the control system. The control performance indicates a value reflecting a deviation between an action intended by an operator operating the operation object during training and an actual action of the operation object. The operational feel indicates a value reflecting the ease of operation of the operation object evaluated by the operator.
[0012] In one embodiment, the instruction section instructs the operation subject to take a next action based on the next action determined by the first determination section, at an early stage of the action of the operation subject.
[0013] In one embodiment, the instruction section instructs the operation subject to take a next action based on the next action determined by the second determination section, at a final stage of the action of the operation subject.
[0014] In one embodiment, the autonomous control controls the operation target more precisely than the voluntary control.
[0015] In one embodiment, the detection unit detects an object, and the weight determination unit dynamically changes the first weight and the second weight in accordance with a distance between the operation target and the object.
[0016] In one embodiment, the control system further includes an instruction unit that instructs the operation target to take a next action based on the next action determined by the first determination unit and the next action determined by the second determination unit. The second determination unit determines the next action of the operation target through autonomous control based on the detection result by the detection unit, the first weight, and the second weight.
[0017] In one embodiment, the control system further includes a storage unit. The storage unit stores change conditions for the first weight and the second weight. The change conditions are adjusted based on an evaluation function including, as parameters, control performance and operational feel acquired during training of the control system. The control performance indicates a value reflecting a deviation between a behavior predicted by an operator who will operate the operation object during training and the behavior of the operation object determined by the second determination unit. The operational feel indicates a value reflecting the ease of operation of the operation object evaluated by the operator.
[0018] In one embodiment, the behavior of the operation target is an utterance by the operation target. The detection unit detects an utterance by a speech partner who is the other party to the utterance of the operation target. The second determination unit determines at least one candidate for the content of the next utterance of the operation target based on the detection result by the detection unit, the first weight, and the second weight. The candidate for the next utterance content determined by the second determination unit can incorporate content of utterance by voluntary control. The first determination unit determines the content of the next utterance based on the electroencephalogram. The instruction unit instructs the operation target on the content of the utterance by the operation target.
[0019] According to the control system of the present invention, the behavior of the controlled object is less likely to differ from the user's intention.
[0020] 1 is a diagram illustrating a control system according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating a configuration of a robot control system, which is an example of a control system according to an embodiment of the present invention. FIG. 3 is a diagram illustrating a portion of a robot arm included in the robot control system. FIG. 4 is a diagram illustrating a configuration of a drive mechanism of a robot arm included in the robot control system. FIG. 5 is a block diagram illustrating a configuration of a drive unit included in the robot control system and a configuration of a drive mechanism of the robot arm. (a) is a plan view illustrating a gripper of a robot arm included in the robot control system. (b) is a block diagram illustrating a configuration of a control unit and a robot arm included in the robot control system. (b) is a block diagram illustrating a configuration of an intention detection unit included in the robot control system. (c) is a block diagram illustrating hardware of a control unit included in the robot control system. (a) is a side view illustrating a robot arm included in the robot control system. (b) is a plan view illustrating a robot arm included in the robot control system. (c) is a flowchart illustrating an example of processing executed by the robot control system. (d) is a block diagram illustrating a configuration of a communication system, which is an example of a control system according to an embodiment of the present invention. FIG. 4 is a diagram illustrating a two-dimensional avatar controlled by the communication system. FIG. 5 is a diagram illustrating a voluntary action determination unit included in the communication system. FIG. 6 is a diagram illustrating an autonomous action determination unit included in the communication system. FIG. 7 is a diagram illustrating an avatar system included in the communication system. FIG. 8 is a block diagram illustrating a configuration of an intention detection unit included in the communication system. 1 is a diagram showing a computer program and data stored in a second storage unit of a control unit included in a communication system. (a) to (c) are diagrams showing screens displayed on a display unit when a two-dimensional avatar is made to speak using a template. (a) to (c) are diagrams showing screens displayed on a display unit when a candidate is selected from candidates for the next utterance content of the two-dimensional avatar. (a) and (b) are diagrams showing screens displayed on a display unit when a user inputs arbitrary characters. (a) and (b) are diagrams showing screens displayed on a display unit when a user inputs arbitrary characters.1 is a diagram showing a screen displayed on a display unit when deciding whether to confirm the next utterance content of a two-dimensional avatar. FIG. 2 is a flowchart showing processing executed by an intention detection unit and a control unit included in a communication system.
[0021] Hereinafter, an embodiment of the control system of the present invention will be described with reference to the drawings (FIGS. 1 to 24). However, the present invention is not limited to the following embodiment, and can be implemented in various forms without departing from the spirit of the present invention. Note that where explanations are redundant, they may be omitted as appropriate. In addition, in the drawings, the same or equivalent parts are designated by the same reference numerals, and explanations will not be repeated.
[0022] First, an embodiment of the present invention will be described with reference to FIG. 1 . FIG. 1 is a diagram illustrating a control system 100 according to this embodiment. The control system 100 controls the operation of an object to be operated 2. Specifically, the control system 100 has a function of autonomously controlling the object to be operated 2. The control system 100 further has a function of voluntarily controlling the object to be operated 2 based on the brain waves of a user P. Specifically, the control system 100 may include a BMI (Brain Machine Interface). Here, the user P is an operator who operates the object to be operated 2. As shown in FIG. 1 , the control system 100 according to this embodiment includes the object to be operated 2, a detection unit 3, a control unit 4, an intention detection unit 5, and an brain wave detection device 6. Hereinafter, autonomous control may be referred to as "autonomous control." Similarly, voluntarily control may be referred to as "voluntary control."
[0023] The operation target 2 is not particularly limited as long as it is capable of voluntary control and autonomous control. The operation target 2 may be, for example, a robot arm 2A described later with reference to Figures 2 to 10, or a two-dimensional avatar 2B described later with reference to Figures 11 to 24. Alternatively, the operation target 2 may be a humanoid robot, an avatar robot, an electric wheelchair, a traveling robot, a drone, or a mouse (user interface device).
[0024] The brain wave detection device 6 detects brain waves from the brain B of the user P. For example, the brain wave detection device 6 detects brain waves every 0.1 seconds or 0.2 seconds. In this embodiment, the brain wave detection device 6 is placed inside the skull of the user P to detect intracranial electroencephalography (iEEG). Specifically, the brain wave detection device 6 has electrodes 61 and a wireless communication unit 62.
[0025] The electrode 61 may be, for example, a brain surface electrode type. In this case, the electrode 61 has a grid-shaped dish electrode. The grid-shaped dish electrode is placed on the surface of the brain B to detect cortical electroencephalograms. Alternatively, the electrode 61 may be an insertion needle electrode type. In this case, the electrode 61 has an array of tiny needle electrodes. The needle electrodes are inserted into the brain B.
[0026] The wireless communication unit 62 performs wireless communication with the intention detection unit 5. The wireless communication unit 62 transmits a wireless signal to the intention detection unit 5. The wireless signal indicates the brain waves detected by the electrodes 61. Specifically, an electrical signal indicating the brain waves is output from the electrodes 61 to the wireless communication unit 62. The wireless communication unit 62 converts the electrical signal into a wireless signal and transmits it to the intention detection unit 5. Note that the wireless communication standard is not particularly limited. For example, the wireless communication unit 62 and the intention detection unit 5 may perform wireless communication in accordance with a short-range wireless communication standard such as Bluetooth (registered trademark).
[0027] The brain wave detection device 6 may further include a signal amplifier. The signal amplifier amplifies the electrical signal output from the electrode 61 and transmits it to the wireless communication unit 62. Specifically, the signal amplifier amplifies the weak current output from the electrode 61 and inputs it to the wireless communication unit 62.
[0028] The detection unit 3 detects a detection target related to the autonomous control of the operation target 2. For example, the detection unit 3 may have an imaging device 31A as described later with reference to FIGS. 2 to 10. Alternatively, the detection unit 3 may have an imaging device 31B and a sound collector 32B as described later with reference to FIGS. 11 to 24. The intention detection unit 5 detects the intention of the user P based on the brain waves of the user P. The control unit 4 controls the operation of the operation target 2 based on the result of detection by the detection unit 3 and the brain waves of the user P. In this embodiment, the control unit 4 controls the operation of the operation target 2 based on the result of detection by the detection unit 3 and the intention of the user P detected by the intention detection unit 5. More specifically, the control unit 4 includes an environment recognition unit 41 and a harmony control unit 42.
[0029] The environment recognition unit 41 recognizes the environment around the operation target 2 based on the detection result by the detection unit 3. For example, the environment recognition unit 41 may determine the type of object ST or the position of the center of gravity of the object ST as the environment around the operation target 2, as will be described later with reference to Figures 2 to 10. Alternatively, the environment recognition unit 41 may determine the content (meaning) of an utterance made by a conversation partner or the emotion of the conversation partner as the environment around the operation target 2, as will be described later with reference to Figures 11 to 24.
[0030] The harmonic control unit 42 controls the operation target 2 by harmonizing voluntary control and autonomous control. Specifically, the harmonic control unit 42 determines the next action of the operation target 2 under voluntary control based on the brain waves of the user P. The harmonic control unit 42 also determines the next action of the operation target 2 under autonomous control based on the detection result by the detection unit 3. Then, the harmonic control unit 42 harmonizes the next action of the operation target 2 under voluntary control with the next action of the operation target 2 under autonomous control to determine the next action of the operation target 2.
[0031] In this embodiment, the harmony control unit 42 determines the next action of the operation target 2 through voluntary control, based on the intention of the user P detected by the intention detection unit 5. The harmony control unit 42 also determines the next action of the operation target 2 through autonomous control, based on the environment around the operation target 2 recognized by the environment recognition unit 41. More specifically, the harmony control unit 42 includes a voluntary action determination unit 43, an autonomous action determination unit 44, a weight determination unit 45, and an instruction unit 46.
[0032] The voluntary action determination unit 43 determines the next action of the operation target 2 under voluntary control based on the brain waves of the user P. In this embodiment, the voluntary action determination unit 43 determines the next action of the operation target 2 under voluntary control based on the intention of the user P detected by the intention detection unit 5. Hereinafter, the next action of the operation target 2 under voluntary control may be referred to as a "voluntary action." The voluntary action determination unit 43 is an example of a "first determination unit."
[0033] The autonomous operation determination unit 44 determines the next action of the operation target 2 under autonomous control based on the detection result by the detection unit 3. In this embodiment, the autonomous operation determination unit 44 determines the next action of the operation target 2 under autonomous control based on the environment surrounding the operation target 2 recognized by the environment recognition unit 41. Hereinafter, the next action of the operation target 2 under autonomous control may be referred to as "autonomous operation." The autonomous operation determination unit 44 is an example of a "second determination unit."
[0034] The weight determination unit 45 determines a weight for the voluntary movement and a weight for the autonomous movement based on the detection result by the detection unit 3. In this embodiment, the weight determination unit 45 determines a weight for the voluntary movement and a weight for the autonomous movement based on the environment around the operation target 2 recognized by the environment recognition unit 41.
[0035] Hereinafter, the weight for voluntary movement may be referred to as the "first weight." Furthermore, the weight for autonomous movement may be referred to as the "second weight." When the value of the first weight is greater than the value of the second weight, the weight of voluntary control becomes greater relative to autonomous control. In other words, the weight of voluntary movement becomes greater relative to autonomous movement. Conversely, when the value of the second weight is greater than the value of the first weight, the weight of autonomous control becomes greater relative to voluntary control. In other words, the weight of autonomous movement becomes greater relative to voluntary movement.
[0036] More specifically, the weight determination unit 45 changes the first weight and the second weight according to the detection result by the detection unit 3. In this embodiment, the weight determination unit 45 changes the first weight and the second weight based on the environment around the operation target 2 recognized by the environment recognition unit 41. Therefore, the values of the first weight and the second weight change according to changes in the environment around the operation target 2.
[0037] The instruction unit 46 instructs the operation target 2 to perform the next action. Specifically, the instruction unit 46 outputs a command (signal) to cause the operation target 2 to execute the next action. The action instructed by the instruction unit 46 to the operation target 2 indicates an action in which the autonomous action and the voluntary action are harmonized by the first weight and the second weight.
[0038] According to this embodiment, the first weight and the second weight can be changed in response to changes in the environment around the operation target 2. In other words, the proportion of voluntary movement and the proportion of autonomous movement can be changed in response to changes in the environment around the operation target 2. As a result, the movement of the operation target 2 can be controlled by harmonizing voluntary control and autonomous control.
[0039] Furthermore, according to this embodiment, when the voluntary movement and the autonomous movement differ, the instruction unit 46 does not instruct the operation target 2 to perform an autonomous movement, but rather instructs the operation target 2 to perform a movement (behavior) that harmonizes the voluntary movement and the autonomous movement. Therefore, even when the voluntary movement and the autonomous movement differ, the voluntary movement can be reflected in the behavior instructed to the operation target 2. This makes it less likely that the movement of the operation target 2 will differ from the movement intended by the user P. For example, in a case where the operation target 2 is a robot arm 2A described below with reference to FIGS. 2 to 10 , when the voluntary movement indicates a lateral movement of the wrist 25 of the robot arm 2A and the autonomous movement indicates a forward movement of the wrist 25 of the robot arm 2A, the instruction unit 46 does not instruct the robot arm 2A to move the wrist 25 forward, but rather instructs the robot arm 2A to perform a behavior that harmonizes the voluntary movement and the autonomous movement using the first weight and the second weight. For example, the instruction unit 46 instructs the robot arm 2A to move the wrist 25 diagonally forward. Therefore, the voluntary movement (lateral movement of the wrist 25) is reflected in the next action of the robot arm 2A.
[0040] Next, the control system 100 of this embodiment will be further described with reference to Fig. 1. In this embodiment, the weight determination unit 45 determines the first weight and the second weight based on the detection result by the detection unit 3 and the electroencephalogram of the user P. More specifically, the weight determination unit 45 determines the first weight and the second weight based on the environment around the operation target 2 recognized by the environment recognition unit 41 and the intention of the user P detected by the intention detection unit 5.
[0041] Specifically, the weight determination unit 45 changes the first weight and the second weight according to the environment around the operation target 2 recognized by the environment recognition unit 41 and the intention of the user P detected by the intention detection unit 5. Therefore, the values of the first weight and the second weight change according to changes in the environment around the operation target 2. Furthermore, the values of the first weight and the second weight change according to changes in the intention of the user P. Therefore, the intention of the user P can be more effectively reflected in the values of the first weight and the second weight. As a result, it becomes even less likely that the operation target 2 will perform an action different from the action intended by the user P.
[0042] [Robot Control System 100A] A robot control system 100A, which is an example of the control system 100 of this embodiment, will be described below with reference to Figures 1 to 10. However, matters that differ from those described with reference to Figure 1 will be described, and descriptions of matters that are the same as those described with reference to Figure 1 will be omitted as appropriate.
[0043] Fig. 2 is a block diagram showing the configuration of a robot control system 100A, which is an example of the control system 100 of this embodiment. Fig. 3 is a diagram showing a portion of a robot arm 2A included in the robot control system 100A. The robot control system 100A is an example of a "control system 100." The operation target 2 of the robot control system 100A is the robot arm 2A.
[0044] As shown in Fig. 2, the robot control system 100A includes a robot arm 2A, a detection unit 3A, a control unit 4A, an intention detection unit 5A, and an electroencephalogram (EEG) detection device 6. The robot control system 100A has a function of autonomously controlling the robot arm 2A. The robot control system 100A further has a function of voluntarily controlling the robot arm 2A based on the electroencephalogram of a user P. Here, the user P is an operator who operates the robot arm 2A. In robot control, autonomous control can control the operation target 2 more precisely than voluntary control.
[0045] Note that precise control of the object 2 includes controlling the object 2 with high accuracy and causing the object 2 to perform complex movements. "Complex movements" include linking multiple movements. For example, precise control is required to cause the robot arm 2A to grasp the object ST. Therefore, the movement of the robot arm 2A to grasp the object ST is an autonomously controlled movement, but is difficult to execute through voluntary control. Specifically, when causing the robot arm 2A to grasp the object ST, the gripper 26 (see FIG. 3 ) of the robot arm 2A must be precisely moved to a position where it can grasp the object ST. However, it is difficult to precisely control the position of the gripper 26 through voluntary control (brain waves). Furthermore, the movement of the gripper 26 to grasp the object ST is a complex movement, and is difficult to control through voluntary control (brain waves).
[0046] The detection unit 3A detects a detection target related to the autonomous control of the robot arm 2A. Specifically, the detection unit 3A has an imaging device 31A. The imaging device 31A captures an image of the detection target and transmits imaging data indicating the captured image to the control unit 4A. The control unit 4A determines the next action of the robot arm 2A under autonomous control based on the image captured by the imaging device 31A. The image may be, for example, a two-dimensional image. Note that the imaging device 31A and the control unit 4A may communicate with each other. For example, the imaging device 31A and the control unit 4A may communicate with each other via a local area network (LAN) or a wireless LAN. Alternatively, the imaging device 31A and the control unit 4A may communicate with each other via a public network such as the Internet.
[0047] More specifically, the imaging device 31A captures an image of the area around the robot arm 2A to detect the detection target. The imaging device 31A may capture a moving image of the area around the robot arm 2A, or may capture a still image of the area around the robot arm 2A. In the robot control system 100A described with reference to FIGS. 2 to 10, the imaging device 31A captures an image in front of the robot arm 2A (see FIG. 3). The imaging device 31A is controlled by the control unit 4A.
[0048] Detection targets of the imaging device 31A include, for example, the object ST and the robot arm 2A (see FIG. 3). That is, the imaging device 31A may detect both the object ST and the robot arm 2A. In the robot control system 100A described with reference to FIGS. 2 to 10, the control unit 4A controls the operation of the robot arm 2A to cause the robot arm 2A to grasp the object ST.
[0049] The control unit 4A acquires information indicating the relationship between the current state of the robot arm 2A and the position of the object ST from the image (imaging data) captured by the imaging device 31A. The control unit 4A then determines the next action of the robot arm 2A through autonomous control based on the information acquired from the imaging data. For example, the control unit 4A may identify the position of the wrist 25 (see FIG. 3) of the robot arm 2A and the position of the object ST based on the imaging data, and calculate the distance between the wrist 25 (see FIG. 3) of the robot arm 2A and the object ST. In this case, the control unit 4A determines the next action (autonomous operation) of the robot arm 2A through autonomous control based on the distance between the wrist 25 of the robot arm 2A and the object ST.
[0050] More specifically, the imaging device 31A may capture a moving image indicating information about the distance to the subject, or alternatively, the imaging device 31A may capture a still image indicating information about the distance to the subject.
[0051] For example, the imaging device 31A may be a stereo camera. In this case, the imaging device 31A has two cameras arranged at a distance from each other, and calculates the distance from the imaging device 31A to the subject based on the principle of triangulation. The imaging device 31A then generates an image indicating the distance to the subject and transmits it to the control unit 4A.
[0052] Alternatively, the imaging device 31A may be a TOF (Time Of Flight) camera. In this case, the imaging device 31A has a light source that irradiates an object with infrared light and a camera, and captures an image of the infrared light reflected from the object with the camera. The imaging device 31A calculates the distance from the imaging device 31A to the object based on the time it takes for the infrared light emitted from the light source to return to the camera. The imaging device 31A then generates an image indicating the distance to the object and transmits it to the control unit 4A.
[0053] The image showing the distance to the subject may be expressed, for example, by color. Specifically, the image capture device 31A may generate an image in which the closer the object is from the image capture device 31A, the bluer the object is, and the farther the object is, the reddish-black the object is.
[0054] The intention detection unit 5A detects the intention of the user P based on the brain waves of the user P. More specifically, the intention detection unit 5A detects the next action of the robot arm 2A intended by the user P based on the brain waves detected by the brain wave detection device 6. Specifically, the intention detection unit 5A acquires the brain waves of the user P from a wireless signal transmitted from the wireless communication unit 62 of the brain wave detection device 6. Then, the intention detection unit 5A analyzes the acquired brain waves to detect the next action of the robot arm 2A intended by the user P. Hereinafter, the next action of the robot arm 2A intended by the user P may be referred to as an "intended action."
[0055] The control unit 4A controls the robot arm 2A based on the output of the imaging device 31A and the output of the intention detection unit 5A. Here, the output of the imaging device 31A indicates the detection result by the imaging device 31A. That is, the output of the imaging device 31A indicates the image (imaging data) captured by the imaging device 31A. The output of the intention detection unit 5A indicates the intended action. Note that the control unit 4A and the intention detection unit 5A may communicate with each other. For example, the control unit 4A and the intention detection unit 5A may communicate with each other via a LAN or a wireless LAN. Alternatively, the control unit 4A and the intention detection unit 5A may communicate with each other via a public network such as the Internet.
[0056] The robot arm 2A has a drive mechanism 27 and a drive unit 28. The drive unit 28 drives the drive mechanism 27. The drive of the drive mechanism 27 causes the robot arm 2A to operate. The control unit 4A controls the drive unit 28 to operate the robot arm 2A. Specifically, the control unit 4A includes an environment recognition unit 41A and a harmony control unit 42A. The harmony control unit 42A includes a voluntary movement determination unit 43A, an autonomous movement determination unit 44A, a weight determination unit 45A, and an instruction unit 46A.
[0057] The voluntary movement determination unit 43A determines the next action (voluntary movement) of the robot arm 2A under voluntary control based on the brain waves of the user P. In detail, the voluntary movement determination unit 43A determines the next action (voluntary movement) of the robot arm 2A under voluntary control based on the intended movement detected by the intention detection unit 5A.
[0058] Specifically, the intention detection unit 5A estimates a velocity vector Vpp of the wrist 25 (see FIG. 3 ) of the robot arm 2A as the intended movement. Hereinafter, the velocity vector Vpp estimated by the intention detection unit 5A may be referred to as the "intended velocity Vpp." The voluntary movement determination unit 43A includes a first velocity determination unit 431. The first velocity determination unit 431 determines a velocity vector Vp of the wrist 25 (see FIG. 3 ) of the robot arm 2A based on the intended velocity Vpp. Hereinafter, the velocity vector Vp determined by the voluntary movement determination unit 43A (first velocity determination unit 431) may be referred to as the "velocity Vp due to voluntary control." For example, the first velocity determination unit 431 may correct the intended velocity Vpp to determine the velocity Vp due to voluntary control.
[0059] The autonomous operation determination unit 44A determines the next action (autonomous operation) of the robot arm 2A under autonomous control based on the detection result by the imaging device 31A. That is, the autonomous operation determination unit 44A determines the autonomous operation based on the image (imaging data) captured by the imaging device 31A. More specifically, the autonomous operation determination unit 44A determines the autonomous operation based on the environment around the robot arm 2A recognized by the environment recognition unit 41A.
[0060] Specifically, the environment recognition unit 41A identifies the robot arm 2A and the object ST included in the image captured by the imaging device 31A, and identifies the current state of the robot arm 2A and the position of the object ST. The autonomous operation determination unit 44A determines the autonomous operation based on the current state of the robot arm 2A and the position of the object ST.
[0061] In the robot control system 100A described with reference to Figures 2 to 10, the environment recognition unit 41A identifies the three-dimensional position of the wrist 25 (see Figure 3) of the robot arm 2A and the three-dimensional position of the object ST from images captured by the imaging device 31A. The environment recognition unit 41A then calculates the distance between the wrist 25 (see Figure 3) and the object ST. The autonomous operation determination unit 44A determines an autonomous operation based on the distance between the wrist 25 (see Figure 3) and the object ST. The autonomous operation includes an operation of moving the wrist 25 (see Figure 3) closer to the object ST.
[0062] More specifically, the autonomous operation determination unit 44A includes a second velocity determination unit 441. The second velocity determination unit 441 determines a velocity vector Va of the wrist 25 (see FIG. 3) of the robot arm 2A based on the distance between the wrist 25 (see FIG. 3) of the robot arm 2A and the object ST. Hereinafter, the velocity vector Va determined by the autonomous operation determination unit 44A (second velocity determination unit 441) may be referred to as the "velocity Va by autonomous control."
[0063] In the robot control system 100A described with reference to FIGS. 2 to 10, the autonomous operation further includes an operation of grasping an object ST with the gripper 26 (see FIG. 3) of the robot arm 2A. Specifically, the intention detection unit 5A further detects an intention to grasp as an intended operation. The intention to grasp indicates an intention to grasp the object ST. When the intention detection unit 5A detects an intention to grasp, the autonomous operation determination unit 44A determines an operation to cause the gripper 26 (see FIG. 3) to grasp the object ST as the autonomous operation.
[0064] Specifically, the environment recognition unit 41A identifies the three-dimensional shape of the object ST from the image captured by the imaging device 31A. Furthermore, the environment recognition unit 41A determines the position of the center of gravity of the object ST based on the three-dimensional shape of the object ST. The autonomous operation determination unit 44A determines the autonomous operation of the gripping unit 26 (see FIG. 3 ) of the robot arm 2A based on the three-dimensional shape and the position of the center of gravity of the object ST. More specifically, the autonomous operation determination unit 44A determines the position at which the gripping unit 26 will grip the object ST based on the position of the center of gravity of the object ST. Then, the autonomous operation determination unit 44A determines the operation of the gripping unit 26 when gripping the object ST based on the three-dimensional shape of the object ST.
[0065] 2 to 10, the environment recognition unit 41A identifies the type of object ST based on the three-dimensional shape of the object ST. The autonomous operation determination unit 44A adjusts the pressure with which the gripping unit 26 grips the object ST based on the type of object ST.
[0066] The weight determination unit 45A determines a weight (first weight) for the voluntary movement and a weight (second weight) for the autonomous movement based on the detection results of the imaging device 31A. That is, the weight determination unit 45A determines a weight (first weight) for the voluntary movement and a weight (second weight) for the autonomous movement based on the image captured by the imaging device 31A. More specifically, the weight determination unit 45A determines the first weight and the second weight based on the distance between the wrist 25 (see FIG. 3) of the robot arm 2A and the object ST. In the robot control system 100A described with reference to FIGS. 2 to 10 , the first weight indicates a weight for the velocity Vp under voluntary control. The second weight indicates a weight for the velocity Va under autonomous control. As already described, the distance between the wrist 25 (see FIG. 3) of the robot arm 2A and the object ST is calculated by the environment recognition unit 41A.
[0067] The weight determination unit 45A changes the values of the first weight and the second weight in accordance with the detection result by the imaging device 31 A. Specifically, the weight determination unit 45A dynamically changes the values of the first weight and the second weight in accordance with the distance between the wrist 25 (see FIG. 3) of the robot arm 2A and the object ST.
[0068] In the robot control system 100A described with reference to Figures 2 to 10, the weight determiner 45A determines the first weight and the second weight based on the distance between the wrist 25 (see Figure 3) of the robot arm 2A and the object ST, and the intended velocity Vpp. As already described, the intended velocity Vpp indicates the velocity vector of the wrist 25 (see Figure 3) estimated based on the electroencephalogram of the user P. By reflecting the intended velocity Vpp in the first weight and the second weight, it becomes even more unlikely that the movement of the robot arm 2A will differ from the movement intended by the user P.
[0069] Specifically, the weight determiner 45A may determine the values of the first weight and the second weight based on the distance between the wrist 25 (see FIG. 3 ) of the robot arm 2A and the object ST, and then correct the values of the first weight and the second weight based on the intended velocity Vpp. For example, the weight determiner 45A may correct the values of the first weight and the second weight by multiplying each of the values of the first weight and the second weight by a coefficient. Here, the coefficient may represent a value that changes depending on the value of the intended velocity Vpp, or may represent a constant value. When a coefficient that changes depending on the value of the intended velocity Vpp is used, the weight determiner 45A may increase the value of the first weight and decrease the value of the second weight as the intended velocity Vpp increases.
[0070] The instruction unit 46A instructs the robot arm 2A to perform the next action. Specifically, the instruction unit 46A controls the drive unit 28 of the robot arm 2A to drive the drive mechanism 27. As a result, the robot arm 2A operates. More specifically, the instruction unit 46A controls the next action of the robot arm 2A based on the voluntary action determined by the voluntary action determination unit 43A, the autonomous action determined by the autonomous action determination unit 44A, the first weight, and the second weight. As a result, voluntary control and autonomous control of the robot arm 2A are harmonized.
[0071] More specifically, the instructing unit 46A changes the proportion of the voluntary movement using the first weight, changes the proportion of the autonomous movement using the second weight, and calculates the linear sum of the voluntary movement and the autonomous movement after the proportions have been changed. Then, the instructing unit 46A determines the next action to be performed by the robot arm 2A based on the calculated linear sum.
[0072] In the robot control system 100A described with reference to FIGS. 2 to 10 , the instruction unit 46A includes a posture determination unit 461. The instruction unit 46A applies a first weight to the velocity Vp obtained by the voluntary control to change the velocity Vp obtained by the voluntary control, and applies a second weight to the velocity Va obtained by the autonomous control to change the velocity Va obtained by the autonomous control. The instruction unit 46A then calculates the linear sum of the velocity Vp and the velocity Va obtained after the changes based on the first and second weights. The posture determination unit 461 determines the next posture of the robot arm 2A based on the calculated linear sum and controls the drive unit 28 based on the determined posture. As a result, the drive mechanism 27 operates, and the posture of the robot arm 2A becomes the determined posture.
[0073] Next, the robot control system 100A will be described with reference to Figures 2 to 5. Figure 4 is a diagram showing the configuration of a drive mechanism 27 for a robot arm 2A included in the robot control system 100A. Figure 5 is a block diagram showing the configuration of a drive unit 28 included in the robot control system 100A and the configuration of the drive mechanism 27 for the robot arm 2A.
[0074] As shown in Fig. 3, the imaging device 31A may be supported by the robot arm 2A. In the robot control system 100A described with reference to Figs. 2 to 10, the imaging device 31A is fixed to the upper part of the robot arm 2A. The line of sight VA of the imaging device 31A may be set obliquely downward. The angle θ between the line of sight VA of the imaging device 31A and the horizontal plane is, for example, 50°.
[0075] As shown in Fig. 3, the robot arm 2A has a shoulder 21, an upper arm 22, an elbow 23, a forearm 24, a wrist 25, and a grip 26. As shown in Fig. 4, the drive mechanism 27 includes a first joint drive mechanism 27a to a seventh joint drive mechanism 27g. In other words, the robot arm 2A has seven degrees of freedom. For example, each of the first joint drive mechanism 27a to the seventh joint drive mechanism 27g may include an electric motor capable of forward and reverse rotation. For example, the electric motor capable of forward and reverse rotation may be a stepping motor.
[0076] Specifically, the shoulder 21 has a first joint drive mechanism 27a, a second joint drive mechanism 27b, and a third joint drive mechanism 27c. The elbow 23 has a fourth joint drive mechanism 27d. The forearm 24 has a fifth joint drive mechanism 27e. The wrist 25 has a sixth joint drive mechanism 27f and a seventh joint drive mechanism 27g. Figures 3 and 4 show the robot arm 2A in a posture in which the upper arm 22 extends vertically downward from the shoulder 21, the elbow 23 is bent at a right angle, and the forearm 24 and wrist 25 extend horizontally.
[0077] As shown in Fig. 4, the first joint drive mechanism 27a is oscillatable in a rotational direction around an axis extending in the Y-axis direction. Here, the Y-axis direction is included in the horizontal direction. The first joint drive mechanism 27a is connected to the second joint drive mechanism 27b. The drive force of the first joint drive mechanism 27a is transmitted to the second joint drive mechanism 27b. Therefore, the second joint drive mechanism 27b is oscillated by the first joint drive mechanism 27a around the axis of the first joint drive mechanism 27a.
[0078] The second joint drive mechanism 27b is disposed on the -Y side relative to the first joint drive mechanism 27a. The second joint drive mechanism 27b is freely oscillating in a rotational direction around an axis extending in a direction perpendicular to the axis of the first joint drive mechanism 27a. In the posture shown in FIG. 4, the axis of the second joint drive mechanism 27b extends in the X-axis direction. In other words, the axis of the second joint drive mechanism 27b extends horizontally. The second joint drive mechanism 27b is coupled to the third joint drive mechanism 27c. The drive force of the second joint drive mechanism 27b is transmitted to the third joint drive mechanism 27c. Therefore, the third joint drive mechanism 27c is oscillated by the second joint drive mechanism 27b around the axis of the second joint drive mechanism 27b.
[0079] In the posture shown in FIG. 4, the third joint drive mechanism 27c is disposed on the −Z side relative to the second joint drive mechanism 27b. The third joint drive mechanism 27c is freely oscillating in a rotational direction about an axis extending perpendicular to the axis of the second joint drive mechanism 27b. In the posture shown in FIG. 4, the axis of the third joint drive mechanism 27c extends in the Z-axis direction. In other words, the axis of the third joint drive mechanism 27c extends in the vertical direction. The third joint drive mechanism 27c is coupled to the fourth joint drive mechanism 27d. The drive force of the third joint drive mechanism 27c is transmitted to the fourth joint drive mechanism 27d. Therefore, the fourth joint drive mechanism 27d is oscillated by the third joint drive mechanism 27c about the axis of the third joint drive mechanism 27c.
[0080] In the posture shown in FIG. 4, the fourth joint drive mechanism 27d is disposed on the −Z side relative to the third joint drive mechanism 27c. The fourth joint drive mechanism 27d is freely oscillating in a rotational direction about an axis extending perpendicular to the axis of the third joint drive mechanism 27c. In the posture shown in FIG. 4, the axis of the fourth joint drive mechanism 27d extends in the Y-axis direction. In other words, the axis of the fourth joint drive mechanism 27d extends horizontally. The fourth joint drive mechanism 27d is coupled to the fifth joint drive mechanism 27e. The drive force of the fourth joint drive mechanism 27d is transmitted to the fifth joint drive mechanism 27e. Therefore, the fifth joint drive mechanism 27e is oscillated by the fourth joint drive mechanism 27d about the axis of the fourth joint drive mechanism 27d.
[0081] In the posture shown in FIG. 4 , the fifth joint drive mechanism 27e is disposed on the +X side relative to the fourth joint drive mechanism 27d. The fifth joint drive mechanism 27e is freely oscillating in a rotational direction around an axis extending perpendicular to the axis of the fourth joint drive mechanism 27d. In the posture shown in FIG. 4 , the axis of the fifth joint drive mechanism 27e extends in the X-axis direction. In other words, the axis of the fifth joint drive mechanism 27e extends horizontally. The fifth joint drive mechanism 27e is coupled to the sixth joint drive mechanism 27f. The drive force of the fifth joint drive mechanism 27e is transmitted to the sixth joint drive mechanism 27f. Therefore, the sixth joint drive mechanism 27f is oscillated by the fifth joint drive mechanism 27e around the axis of the fifth joint drive mechanism 27e.
[0082] In the posture shown in FIG. 4 , the sixth joint drive mechanism 27f is disposed on the +X side relative to the fifth joint drive mechanism 27e. The sixth joint drive mechanism 27f is freely oscillating in a rotational direction around an axis extending in the same direction as the axis of the fifth joint drive mechanism 27e. In the posture shown in FIG. 4 , the axis of the sixth joint drive mechanism 27f extends in the X-axis direction. In other words, the axis of the sixth joint drive mechanism 27f extends horizontally. The sixth joint drive mechanism 27f is coupled to the seventh joint drive mechanism 27g. The drive force of the sixth joint drive mechanism 27f is transmitted to the seventh joint drive mechanism 27g. Therefore, the seventh joint drive mechanism 27g is oscillated by the sixth joint drive mechanism 27f around the axis of the sixth joint drive mechanism 27f.
[0083] In the posture shown in FIG. 4 , the seventh joint drive mechanism 27g is disposed on the +X side relative to the sixth joint drive mechanism 27f. The seventh joint drive mechanism 27g is freely oscillating in a rotational direction around an axis extending in a direction perpendicular to the axis of the sixth joint drive mechanism 27f. In the posture shown in FIG. 4 , the axis of the seventh joint drive mechanism 27g extends in the Z-axis direction. In other words, the axis of the seventh joint drive mechanism 27g extends in the vertical direction. The seventh joint drive mechanism 27g is coupled to the gripper 26. The drive force of the seventh joint drive mechanism 27g is transmitted to the gripper 26. Therefore, the gripper 26 is oscillated by the seventh joint drive mechanism 27g around the axis of the seventh joint drive mechanism 27g.
[0084] 5, the drive unit 28 includes a first drive unit 28a to a seventh drive unit 28g. The first drive unit 28a drives the first joint drive mechanism 27a. Similarly, the second drive unit 28b to the seventh drive unit 28g drive the second joint drive mechanism 27b to the seventh joint drive mechanism 27g, respectively. The first drive unit 28a to the seventh drive unit 28g include, for example, motor drivers.
[0085] The harmonic control unit 42A controls the first drive unit 28a to drive the first joint drive mechanism 27a. Similarly, the harmonic control unit 42A controls the second drive unit 28b to the seventh drive unit 28g to drive the second joint drive mechanism 27b to the seventh joint drive mechanism 27g, respectively.
[0086] Specifically, the harmonic control unit 42A controls the swing angle θc1 (rotation angle) of the first joint drive mechanism 27a. More specifically, the harmonic control unit 42A controls the swing angle θc1 (rotation angle) of the second joint drive mechanism 27b around the axis of the first joint drive mechanism 27a.
[0087] Similarly, the harmonic control unit 42A controls the swing angles θc2 to θc7 of the second joint drive mechanism 27b to seventh joint drive mechanism 27g, respectively. Specifically, the harmonic control unit 42A controls the swing angle θc2 (rotation angle) of the third joint drive mechanism 27c around the axis of the second joint drive mechanism 27b. The harmonic control unit 42A controls the swing angle θc3 (rotation angle) of the fourth joint drive mechanism 27d around the axis of the third joint drive mechanism 27c. The harmonic control unit 42A controls the swing angle θc4 (rotation angle) of the fifth joint drive mechanism 27e around the axis of the fourth joint drive mechanism 27d. The harmonic control unit 42A controls the swing angle θc5 (rotation angle) of the sixth joint drive mechanism 27f around the axis of the fifth joint drive mechanism 27e. The harmonic control unit 42A controls the swing angle θc6 (rotation angle) of the seventh joint drive mechanism 27g around the axis of the sixth joint drive mechanism 27f. The harmonic control unit 42A controls the swing angle θc7 (rotation angle) of the gripper 26 around the axis of the seventh joint drive mechanism 27g.
[0088] Next, the gripper 26 included in the robot arm 2A will be described with reference to Figures 6(a) and 6(b). Figure 6(a) is a plan view showing the gripper 26 of the robot arm 2A included in the robot control system 100A. Figure 6(b) is a block diagram showing the configuration of the control unit 4A and robot arm 2A included in the robot control system 100A.
[0089] As shown in FIG. 6( a ), the gripping portion 26 may have a first gripping member 26 a , a second gripping member 26 b , a first gear 26 c , a second gear 26 d , and a support member 26 e .
[0090] The support member 26e rotatably supports the first gear 26c and the second gear 26d. Specifically, the first gear 26c and the second gear 26d are supported by the support member 26e with their teeth meshing with each other. Therefore, the first gear 26c and the second gear 26d rotate in opposite directions.
[0091] The first gripping member 26a is pivotally supported by a support member 26e and protrudes from the support member 26e. The driving force of the first gear 26c is transmitted to the base end of the first gripping member 26a. Similarly, the second gripping member 26b is pivotally supported by the support member 26e and protrudes from the support member 26e on the same side as the first gripping member 26a. The driving force of the second gear 26d is transmitted to the base end of the second gripping member 26b.
[0092] The first gripping member 26a and the second gripping member 26b face each other. As the first gear 26c and the second gear 26d swing, the first gripping member 26a and the second gripping member 26b open and close. Here, the opening of the first gripping member 26a and the second gripping member 26b indicates that the first gripping member 26a and the second gripping member 26b rotate in directions away from each other around their respective base ends. Furthermore, the closing of the first gripping member 26a and the second gripping member 26b indicates that the first gripping member 26a and the second gripping member 26b rotate in directions toward each other around their respective base ends.
[0093] More specifically, as shown in FIG. 6B , the gripper 26 has a gripper drive mechanism 27h. The drive unit 28 includes an eighth drive unit 28h. The eighth drive unit 28h drives the gripper drive mechanism 27h. When the gripper drive mechanism 27h is driven, the first gear 26c and the second gear 26d oscillate. For example, the gripper drive mechanism 27h includes an electric motor that can rotate forward and backward. For example, the electric motor that can rotate forward and backward may be a stepping motor. The eighth drive unit 28h includes, for example, a motor driver. The support member 26e may be box-shaped. In this case, the gripper drive mechanism 27h may be housed inside the support member 26e.
[0094] The harmonic control unit 42A controls the eighth drive unit 28h to operate the gripper 26. For example, when causing the gripper 26 to grip the object ST, the harmonic control unit 42A controls the eighth drive unit 28h to close the first gripping member 26a and the second gripping member 26b. As a result, the object ST sandwiched between the first gripping member 26a and the second gripping member 26b is gripped by the first gripping member 26a and the second gripping member 26b. Furthermore, when releasing the gripper 26 from gripping the object ST, the harmonic control unit 42A controls the eighth drive unit 28h to open the first gripping member 26a and the second gripping member 26b.
[0095] In the robot control system 100A described with reference to FIGS. 2 to 10 , as described with reference to FIG. 2 , the environment recognition unit 41A identifies the three-dimensional shape of the object ST, and the autonomous operation determination unit 44A determines the operation of the gripping unit 26 when gripping the object ST based on the three-dimensional shape of the object ST. Therefore, the harmonic control unit 42A can control the swing angle (rotation angle) of the first gripping member 26 a and the second gripping member 26 b in accordance with the three-dimensional shape of the object ST. In other words, the harmonic control unit 42A can control the opening angle between the first gripping member 26 a and the second gripping member 26 b in accordance with the three-dimensional shape of the object ST. As a result, the object ST can be gripped with a more appropriate pressure.
[0096] Furthermore, in the robot control system 100A described with reference to Figures 2 to 10, the environment recognition unit 41A identifies the type of object ST, and the autonomous operation determination unit 44A determines the operation of the gripping unit 26 when gripping the object ST based on the three-dimensional shape of the object ST and the type of object ST. Therefore, the harmonic control unit 42A can control the swing angle (rotation angle) of the first gripping member 26a and the second gripping member 26b in accordance with the three-dimensional shape and type of object ST. In other words, the harmonic control unit 42A can control the opening angle between the first gripping member 26a and the second gripping member 26b in accordance with the three-dimensional shape and type of object ST. As a result, the object ST can be gripped with more appropriate pressure.
[0097] Next, the intention detection unit 5A included in the robot control system 100A will be described with reference to Fig. 7. Fig. 7 is a block diagram showing the configuration of the intention detection unit 5A included in the robot control system 100A. As shown in Fig. 7, the intention detection unit 5A has a first storage unit 51, a first processing unit 52, and a wireless communication unit 53. The intention detection unit 5A may be, for example, a personal computer.
[0098] The wireless communication unit 53 receives a wireless signal indicating the brain waves of the user P from the brain wave detection device 6. Specifically, the wireless communication unit 53 performs wireless communication with the wireless communication unit 62 of the brain wave detection device 6 described with reference to FIG. 1 . More specifically, the wireless communication unit 53 performs wireless communication in accordance with the same wireless communication standard as the wireless communication unit 62 of the brain wave detection device 6. The wireless communication unit 53 converts the wireless signal indicating the brain waves into an electrical signal indicating the brain waves and inputs it to the first processing unit 52.
[0099] The first storage unit 51 stores various computer programs and various data. The first storage unit 51 includes, for example, a semiconductor memory. The semiconductor memory includes, for example, a random access memory (RAM) and a read-only memory (ROM). The semiconductor memory may include a flash memory, an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a video RAM (VRAM). The first storage unit 51 may include a hard disk drive (HDD) or a solid state drive (SSD), or may include removable media.
[0100] The first processing unit 52 performs various processes such as numerical calculations and information processing by executing computer programs stored in the first storage unit 51. The first processing unit 52 may include, for example, a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The first processing unit 52 may further include a GPU (Graphics Processing Unit) or an NPU (Neural Network Processing Unit). Alternatively, the first processing unit 52 may include a quantum computer.
[0101] For example, the first processing unit 52 acquires the intention (intended action) of the user P from electroencephalograms by multivariate analysis. The multivariate analysis includes analysis using a machine learning model. In the robot control system 100A described with reference to FIGS. 2 to 10 , the first storage unit 51 stores a first machine learning model D1. The first machine learning model D1 is a computer program that analyzes electroencephalograms to estimate the intention of the user P. Specifically, the first machine learning model D1 is trained to analyze electroencephalograms to estimate the intended action.
[0102] More specifically, the first machine learning model D1 is trained to output a velocity vector (intended velocity Vpp) of the wrist 25 of the robot arm 2A through regression analysis. The intended velocity Vpp (velocity vector) includes an X-axis component Vppx, a Y-axis component Vppy, and a Z-axis component Vppz. The first machine learning model D1 is also trained to detect a grasping intention through regression analysis.
[0103] For example, the first machine learning model D1 may perform analysis using support vector regression. The first machine learning model D1 may perform preprocessing before the analysis using support vector regression. The preprocessing includes a process of extracting features from the electroencephalogram. For example, the preprocessing may be partial least squares regression (PLS regression) or independent component analysis (ICA).
[0104] When preprocessing is performed, the intended velocity Vpp can be calculated by the following formula (1). In the following formula (1), "iEEG" indicates an intracranial electroencephalogram. "W1" indicates a weight for preprocessing. "W2" indicates a weight for support vector regression. Vpp = iEEG × W1 × W2 = (Vppx, Vppy, Vppz) (1)
[0105] The first processing unit 52 converts the electrical signals indicative of brain waves into data indicative of brain waves and inputs the data to the first machine learning model D1. As a result, the first machine learning model D1 analyzes the brain waves to estimate the intention of the user P. Specifically, the first machine learning model D1 analyzes the brain waves to estimate the intended action. The first processing unit 52 outputs data indicative of the intended action estimated by the first machine learning model D1 to the control unit 4A. In the robot control system 100A described with reference to FIGS. 2 to 10 , the first processing unit 52 (intention detection unit 5A) outputs a signal indicative of the intended velocity Vpp to the control unit 4A as data indicative of the intended action. The first processing unit 52 (intention detection unit 5A) also outputs a signal indicative of a grasping intention to the control unit 4A as data indicative of the intended action.
[0106] Specifically, the first processing unit 52 performs band-pass filtering on the electrical signals representing the brain waves and inputs the band-pass filtered brain wave data to the first machine learning model D1. Specifically, the band-pass filtering extracts signals in a frequency band related to movement from the electrical signals representing the brain waves. The frequency band related to movement is, for example, a frequency band between 60 Hz and 200 Hz.
[0107] Note that the machine learning algorithm for constructing the first machine learning model D1 is not limited to support vector regression. The machine learning algorithm for constructing the first machine learning model D1 may be, for example, a decision tree, a nearest neighbor method, a naive Bayes classifier, or a neural network. Therefore, the first machine learning model D1 includes a decision tree, a nearest neighbor method, a naive Bayes classifier, a support vector machine, or a neural network. Backpropagation may also be used for machine learning.
[0108] For example, the neural network includes an input layer, one or more hidden layers, and an output layer. Specifically, the neural network includes at least one of a deep neural network (DNN), a recurrent neural network (RNN), or a convolutional neural network (CNN) to perform deep learning. For example, the deep neural network includes an input layer, multiple hidden layers, and an output layer.
[0109] Next, the control unit 4A included in the robot control system 100A will be described with reference to Fig. 8. Fig. 8 is a block diagram showing the hardware of the control unit 4A included in the robot control system 100A. As shown in Fig. 8, the control unit 4A has a second storage unit 4a, a second processing unit 4b, and an input unit 4c. The control unit 4A may be, for example, a personal computer.
[0110] The second storage unit 4a stores various computer programs and various data. The second storage unit 4a includes, for example, a semiconductor memory. The semiconductor memory includes, for example, a RAM and a ROM. The semiconductor memory may include a flash memory, an EPROM, or an EEPROM, or may include a VRAM. The second storage unit 4a may further include an HDD or an SSD, or may include removable media.
[0111] Specifically, as shown in Fig. 8, the second storage unit 4a stores a robot operating system (ROS) D11. The robot operating system D11 is middleware (computer program) for operating the robot. The second processing unit 4b executes the robot operating system D11 to control the robot arm 2A.
[0112] The second processing unit 4b executes computer programs stored in the second storage unit 4a to perform various processes such as numerical calculations and information processing. The second processing unit 4b may include, for example, a CPU or an MPU. The second processing unit 4b may further include a GPU or an NPU. Alternatively, the second processing unit 4b may include a quantum computer.
[0113] Specifically, the second processing unit 4b functions as the environment recognition unit 41A, the voluntary action determination unit 43A, the autonomous action determination unit 44A, the weight determination unit 45A, and the instruction unit 46A described with reference to FIG. 2 by executing the computer program stored in the second storage unit 4a.
[0114] For example, when functioning as the voluntary movement determiner 43A, the second processing unit 4b determines a voluntary movement based on an intended movement. Specifically, when functioning as the voluntary movement determiner 43A (first velocity determiner 431), the second processing unit 4b determines a velocity vector Vp (velocity Vp due to voluntary control) of the wrist 25 of the robot arm 2A based on the intended velocity Vpp. The velocity Vp due to voluntary control includes a component Vpx in the X-axis direction, a component Vpy in the Y-axis direction, and a component Vpz in the Z-axis direction.
[0115] When functioning as the environment recognition unit 41A, the second processing unit 4b identifies the current position of the wrist 25 of the robot arm 2A and the position of the object ST based on the image captured by the imaging device 31A, and calculates the distance between the wrist 25 and the object ST. When functioning as the autonomous operation determination unit 44A (second speed determination unit 441), the second processing unit 4b determines an autonomous operation based on the distance between the wrist 25 and the object ST.
[0116] 2 to 10, in the robot control system 100A, the second processing unit 4b, when functioning as the autonomous operation determination unit 44A (second velocity determination unit 441), calculates a velocity Va (velocity vector Va of the wrist 25) based on the distance between the wrist 25 and the object ST. The velocity Va based on the autonomous control includes a component Vax in the X-axis direction, a component Vay in the Y-axis direction, and a component Vaz in the Z-axis direction.
[0117] Furthermore, when functioning as the environment recognition unit 41A, the second processing unit 4b identifies the three-dimensional shape of the object ST based on the image captured by the imaging device 31A. Then, the second processing unit 4b identifies the center of gravity position and the type of the object ST based on the three-dimensional shape of the object ST.
[0118] Specifically, as shown in FIG. 8 , the second memory unit 4a stores the second machine learning model D12. Specifically, the second machine learning model D12 is incorporated into the robot operating system D11. The second machine learning model D12 is trained to analyze images captured by the imaging device 31A to identify the current position of the wrist 25 of the robot arm 2A and the position of the object ST, and to calculate the distance between the wrist 25 of the robot arm 2A and the object ST. Furthermore, the second machine learning model D12 is trained to analyze images captured by the imaging device 31A to estimate the three-dimensional shape of the object ST, the center of gravity of the object ST, and the type of the object ST. The second processing unit 4b inputs data representing the images captured by the imaging device 31A to the second machine learning model D12. As a result, information representing the distance between the wrist 25 of the robot arm 2A and the object ST, the three-dimensional shape of the object ST, the center of gravity of the object ST, and the type of the object ST is output from the second machine learning model D12.
[0119] The machine learning algorithm for constructing the second machine learning model D12 is not particularly limited and may be, for example, a decision tree, a nearest neighbor algorithm, a naive Bayes classifier, a support vector machine, or a neural network. Therefore, the second machine learning model D12 includes a decision tree, a nearest neighbor algorithm, a naive Bayes classifier, a support vector machine, or a neural network. Backpropagation may also be used for machine learning.
[0120] When functioning as the weight determiner 45A, the second processing unit 4b determines the first weight and the second weight based on the detection results of the imaging device 31A. In the robot control system 100A described with reference to FIGS. 2 to 10, the second processing unit 4b determines the first weight and the second weight based on the distance between the wrist 25 (see FIG. 3) and the object ST and the intended velocity Vpp. More specifically, as shown in FIG. 8, the second storage unit 4a further stores a weight change condition D13. Specifically, the weight change condition D13 is incorporated into the robot operating system D11. The second processing unit 4b determines the values of the first weight and the second weight by referring to the weight change condition D13.
[0121] The weight change condition D13 indicates a change condition for each of the first weight and the second weight. Specifically, the weight change condition D13 indicates a condition for changing each of the first weight and the second weight in accordance with the detection result by the imaging device 31A. In the robot control system 100A described with reference to FIGS. 2 to 10 , the weight change condition D13 indicates a condition for changing each of the first weight and the second weight in accordance with the detection result by the imaging device 31A and the intended action.
[0122] The input unit 4c is a user interface device operated by the worker. The input unit 4c typically includes a keyboard and a mouse. The input unit 4c may also include a touch sensor. The worker can adjust the weight change condition D13 by operating the input unit 4c. That is, the worker can adjust the change conditions for each value of the first weight and the second weight by operating the input unit 4c.
[0123] 2 to 10, the weight change condition D13 indicates a condition for changing the values of the first weight and the second weight in accordance with the distance between the wrist 25 of the robot arm 2A and the object ST and the intended velocity Vpp. In other words, the weight change condition D13 indicates the relationship between the distance between the wrist 25 and the object ST, the intended velocity Vpp, and the values of the first weight and the second weight.
[0124] Specifically, the weight change condition D13 indicates the relationship between the distance between the wrist 25 and the object ST, the intended velocity Vpp, and the weight Wp (first weight) for the voluntary movement. The second processing unit 4b acquires the intended velocity Vpp from the intention detection unit 5A and calculates the distance between the wrist 25 and the object ST, and then determines the value of the weight Wp (first weight) by referring to the weight change condition D13. The second processing unit 4b then calculates the weight (second weight) for the autonomous movement based on the weight Wp (first weight). More specifically, the second processing unit 4b calculates the second weight based on the following equation (2): second weight=1-Wp (2)
[0125] In the robot control system 100A described with reference to Figures 2 to 10, the first weight (Wp) is a weight for the velocity Vp under voluntary control, and the second weight (1-Wp) is a weight for the velocity Va under autonomous control. As the value of the weight Wp increases, the weight of the velocity Vp under voluntary control increases relative to the velocity Va under autonomous control. Conversely, as the value of the weight Wp decreases, the weight of the velocity Va under autonomous control increases relative to the velocity Vp under voluntary control.
[0126] The weight change condition D13 may specify the first weight (Wp) so that the robot arm 2A behaves under voluntary control in the initial stage of the behavior of the robot arm 2A. For example, the weight change condition D13 may specify the first weight (Wp) so that the robot arm 2A behaves under voluntary control when the distance between the object ST and the wrist 25 of the robot arm 2A is 1 meter or more. Specifically, the weight change condition D13 may specify the value of the first weight (Wp) to be "1" or approximately "1" when the distance between the object ST and the wrist 25 of the robot arm 2A is 1 meter or more. As a result, in the initial stage of the behavior of the robot arm 2A, the second processing unit 4b controls the robot arm 2A by an intended action, and the behavior of the robot arm 2A becomes the behavior intended by the user P. For example, when the distance between the object ST and the wrist 25 of the robot arm 2A is 1 meter or more, the behavior of the robot arm 2A becomes the behavior intended by the user P.
[0127] Furthermore, the weight change condition D13 may specify the first weight (Wp) so that the robot arm 2A acts by autonomous control in the final stage of the behavior of the robot arm 2A. For example, the weight change condition D13 may specify the first weight (Wp) so that the robot arm 2A acts by autonomous control when the distance between the object ST and the wrist 25 of the robot arm 2A is 30 cm or less. Specifically, the weight change condition D13 may specify the value of the first weight (Wp) to be "0" or approximately "0" when the distance between the object ST and the wrist 25 of the robot arm 2A is 30 cm or less. As a result, in the final stage of the behavior of the robot arm 2A, the second processing unit 4b controls the robot arm 2A by autonomous operation, thereby enabling more precise control of the robot arm 2A (the operation target 2) than with voluntary control.
[0128] The relationship between the distance between the wrist 25 and the object ST and the weight Wp may be continuous or stepwise. That is, the value of the weight Wp may change continuously or stepwise in response to changes in the distance between the wrist 25 and the object ST. For example, the weight change condition D13 may be a condition for continuously or stepwise decreasing the value of the first weight (Wp) during the course of the behavior of the robot arm 2A. Specifically, the weight change condition D13 may be a condition for continuously or stepwise decreasing the value of the first weight (Wp) as the distance between the object ST and the wrist 25 of the robot arm 2A decreases, when the distance between the object ST and the wrist 25 of the robot arm 2A is shorter than 1 m and greater than 30 cm.
[0129] In the robot control system 100A described with reference to FIGS. 2 to 10 , the weight change condition D13 may indicate a condition for assigning a coefficient to the first weight (Wp) determined based on the distance between the object ST and the wrist 25 of the robot arm 2A when the intended velocity Vpp is equal to or greater than a predetermined velocity. Here, the coefficient indicates a value greater than "1." The coefficient may be a constant value or a value that changes depending on the intended velocity Vpp. The coefficient may change continuously or in stages depending on the intended velocity Vpp. By increasing the value of the first weight when the intended velocity Vpp is equal to or greater than a predetermined velocity, the movement of the robot arm 2A is less likely to deviate from the movement intended by the user P.
[0130] When functioning as the instruction unit 46A, the second processing unit 4b changes the proportion of the intended movement using the first weight, changes the proportion of the autonomous movement using the second weight, and calculates the linear sum of the intended movement and the autonomous movement after the proportions have been changed. Then, the second processing unit 4b determines the next action to be performed by the robot arm 2A based on the calculated linear sum.
[0131] In the robot control system 100A described with reference to FIGS. 2 to 10, when the second processing unit 4b functions as the instruction unit 46A, it changes the velocity Vp obtained by voluntary control using a first weight (Wp) and changes the velocity Va obtained by autonomous control using a second weight (1-Wp). The second processing unit 4b then calculates the linear sum of the changed velocity Vp and velocity Va to determine a velocity vector Vcmd of the wrist 25 (see FIG. 3) of the robot arm 2A. The velocity vector Vcmd is a command value. Hereinafter, the velocity vector Vcmd may be referred to as a "command velocity Vcmd."
[0132] For example, the command velocity Vcmd can be calculated based on the following equation (3): The command velocity Vcmd includes an X-axis component Vcmd_x, a Y-axis component Vcmd_y, and a Z-axis component Vcmd_z: Vcmd=Vp×Wp+Va×(1−Wp) (3)
[0133] When functioning as the instruction unit 46A, the second processing unit 4b controls the robot arm 2A based on the command velocity Vcmd. Specifically, the second processing unit 4b executes the robot operating system D11 and controls the drive mechanism 27 (the first joint drive mechanism 27a to the seventh joint drive mechanism 27g) described with reference to Figures 3 to 5 so that the wrist unit 25 moves at the command velocity Vcmd.
[0134] Specifically, the second processing unit 4b (robot operating system D11) integrates the X-axis component Vcmd_x, the Y-axis component Vcmd_y, and the Z-axis component Vcmd_z included in the command velocity Vcmd to determine the next three-dimensional position (Xo, Yo, Zo) of the wrist 25 of the robot arm 2A, and controls the drive mechanism 27 described with reference to FIGS. 3 to 5 so that the wrist 25 of the robot arm 2A moves to the next three-dimensional position (Xo, Yo, Zo). For example, the next three-dimensional positions (Xo, Yo, Zo) can be calculated using the following equations (4) to (6). Note that the three-dimensional coordinates may be set with the shoulder 21 of the robot arm 2A as the origin (0, 0, 0). Hereinafter, the next three-dimensional position may be referred to as the "target position."
[0135] The robot operating system D11 incorporates a control formula that calculates the target angles θ1 to θ7 of the first joint drive mechanism 27a to the seventh joint drive mechanism 27g based on inverse kinematics. The second processing unit 4b (robot operating system D11) calculates the target angles θ1 to θ7 of the first joint drive mechanism 27a to the seventh joint drive mechanism 27g from the target positions (Xo, Yo, Zo). The second processing unit 4b (robot operating system D11) then controls the first joint drive mechanism 27a to the seventh joint drive mechanism 27g so that the swing angles θc1 to θc7 of the first joint drive mechanism 27a to the seventh joint drive mechanism 27g become the target angles θ1 to θ7.
[0136] For example, the robot operating system D11 may incorporate the following equations (7) to (19) as control equations. Specifically, when Yo is equal to or less than 0, the second processing unit 4b (robot operating system D11) calculates the target angles θ1 to θ7 from equations (7) to (14). When Yo is greater than 0, the second processing unit 4b (robot operating system D11) calculates the target angles θ1 to θ7 from equations (7) to (9) and equations (15) to (19).
[0137] Next, "α", "β", "L1", "L2", and "r" included in the above formulas (7) to (19) will be described with reference to Figures 9(a) and 9(b). Figure 9(a) is a side view showing the robot arm 2A included in the robot control system 100A, and Figure 9(b) is a plan view showing the robot arm 2A included in the robot control system 100A.
[0138] As shown in FIG. 9( a), "α" indicates the angle between the Z axis and the upper arm 22. "β" indicates the angle between the upper arm 22 and the forearm 24 when viewed from the side. "L1" indicates the length of the upper arm 22. "L2" indicates the length of the forearm 24. As shown in FIGS. 9( a) and 9(b), "r" indicates the distance from the shoulder 21 (origin) to the target position (Xo, Yo, Zo).
[0139] Next, the processing executed by the robot control system 100A will be described with reference to Figures 2 to 10. Figure 10 is a flowchart showing an example of the processing executed by the robot control system 100A. In detail, Figure 10 shows an example of the processing for causing the gripper 26 of the robot arm 2A to grip an object ST. As shown in Figure 10, the processing executed by the robot control system 100A may include steps S1 to S10.
[0140] 10 starts, for example, in response to detection of the brain waves of the user P by the brain wave detection device 6. When the process shown in Fig. 10 starts, first, the intention detection unit 5A (first processing unit 52) estimates an intended action from the brain waves of the user P (step S1).
[0141] When the intended movement is estimated, the intention detection unit 5A (first processing unit 52) determines whether the intended movement is a grasping intention (step S2). If it is determined that the intended movement is not a grasping intention (No in step S2), the intended movement indicates an intended velocity Vpp. In this case, the intention detection unit 5A (first processing unit 52) transmits a signal indicating the intended velocity Vpp to the control unit 4A. If it is determined that the intended movement is a grasping intention (Yes in step S2), the intention detection unit 5A (first processing unit 52) transmits a signal indicating a grasping intention to the control unit 4A.
[0142] When the second processing unit 4b (voluntary movement determination unit 43A) acquires information indicating the intended velocity Vpp from the intention detection unit 5A (No in step S2), it determines a voluntary movement (step S3). Specifically, the second processing unit 4b (voluntary movement determination unit 43A) calculates a velocity vector Vp due to voluntary control from the intended velocity Vpp.
[0143] Furthermore, when the second processing unit 4b (weight determination unit 45A) acquires information indicating the intended velocity Vpp from the intention detection unit 5A (No in step S2), it determines the first weight (Wp) and calculates the second weight (1-Wp) (step S4).
[0144] Specifically, the second processing unit 4b (environment recognition unit 41A) calculates the distance between the object ST and the wrist 25 of the robot arm 2A based on the image captured by the imaging device 31A. Then, the second processing unit 4b (weight determination unit 45A) determines the first weight (Wp) and calculates the second weight (1-Wp) based on the distance between the object ST and the wrist 25 of the robot arm 2A and the intended velocity Vpp.
[0145] After determining the first weight (Wp) and calculating the second weight (1-Wp), the second processing unit 4b determines an autonomous operation based on the distance between the object ST and the wrist 25 of the robot arm 2A (step S5). Specifically, the second processing unit 4b (autonomous operation determination unit 44A) calculates the autonomously controlled velocity Va. The second processing unit 4b (instruction unit 46A) then calculates a command velocity Vcmd based on the voluntary control velocity Vp, the autonomously controlled velocity Va, the first weight (Wp), and the second weight (1-Wp) (step S6).
[0146] After calculating the command velocity Vcmd, the second processing unit 4b (instruction unit 46A) integrates the command velocity Vcmd to calculate the target position (Xo, Yo, Zo) (step S7).The second processing unit 4b (instruction unit 46A) then calculates the target angles θ1 to θ7 of the first joint-driving mechanism 27a to the seventh joint-driving mechanism 27g based on the target positions (Xo, Yo, Zo) (step S8).
[0147] After calculating the target angles θ1 to θ7, the second processing unit 4b (instruction unit 46A) drives the first joint drive mechanism 27a to the seventh joint drive mechanism 27g so that the swing angles θc1 to θc7 of the first joint drive mechanism 27a to the seventh joint drive mechanism 27g become the target angles θ1 to θ7. As a result, the wrist 25 of the robot arm 2A moves to the target position (Xo, Yo, Zo) (step S9). Once the wrist 25 of the robot arm 2A has moved to the target position (Xo, Yo, Zo), the process returns to step S1. The intention detection unit 5A (first processing unit 52) estimates the intended movement every 0.1 or 0.2 seconds.
[0148] For example, when the distance between the object ST and the wrist 25 of the robot arm 2A is 1 m or more, the second processing unit 4b (weight determination unit 45A) determines the value of the first weight (Wp) to be "1" or approximately "1." As a result, the command velocity Vcmd becomes equal to or approximately equal to the velocity Vp by voluntary control. Therefore, the wrist 25 moves as intended by the user P.
[0149] When the distance between the object ST and the wrist 25 of the robot arm 2A becomes less than 1 m, the second processing unit 4b (weight determination unit 45A) reduces the value of the first weight (Wp) as the distance between the object ST and the wrist 25 of the robot arm 2A decreases. As a result, as the wrist 25 of the robot arm 2A approaches the object ST, the proportion to which the velocity Vp based on the voluntary control is reflected in the command velocity Vcmd decreases, and the proportion to which the velocity Va based on the autonomous control is reflected in the command velocity Vcmd increases.
[0150] When the distance between the object ST and the wrist 25 of the robot arm 2A becomes 30 cm or less, the second processing unit 4b (weight determination unit 45A) determines the value of the first weight (Wp) to be "0" or approximately "0." As a result, the command velocity Vcmd becomes equal to or approximately equal to the velocity Va under autonomous control. Therefore, the wrist 25 moves under autonomous control.
[0151] If the second processing unit 4b determines that the intended action is a grasping intention (Yes in step S2), the second processing unit 4b controls the gripping unit 26 based on the image captured by the imaging device 31A to cause the gripping unit 26 to grasp the object ST (step S10). As a result, the process shown in FIG. 10 ends.
[0152] Specifically, the second processing unit 4b (environment recognition unit 41A) identifies the three-dimensional shape, center of gravity position, and type of the object ST based on the image captured by the imaging device 31A. Then, the second processing unit 4b (autonomous operation determination unit 44A) determines an autonomous operation based on the three-dimensional shape, center of gravity position, and type of the object ST of the object ST. The second processing unit 4b (instruction unit 46A) controls the gripping drive mechanism 27h based on the determined autonomous operation to operate the first gripping member 26a and the second gripping member 26b. In other words, the second processing unit 4b operates the gripping unit 26 by autonomous control. As a result, the object ST is gripped by the first gripping member 26a and the second gripping member 26b.
[0153] According to this embodiment, when the robot arm 2A grasps the object ST, the grasper 26 is operated by autonomous control. Therefore, the object ST can be grasped by the robot arm 2A with greater precision than when the grasper 26 is operated by voluntary control.
[0154] Next, the weight change condition D13 will be described with reference to FIGS. 2 to 9 . In the robot control system 100A, the weight change condition D13 is adjusted based on an evaluation function. The evaluation function includes, as parameters, control performance and operability acquired during training of the robot control system 100A (control system 100). The control performance indicates a value reflecting the amount of deviation between the behavior intended by the operator operating the robot arm 2A (operation object 2) during training of the robot control system 100A (control system 100) and the actual behavior of the robot arm 2A (operation object 2). The operability indicates a value reflecting the ease of operation of the robot arm 2A (operation object 2) evaluated by the operator.
[0155] For example, the weight change condition D13 may be adjusted using the following equation (20) as the evaluation function: Evaluation function=a·f (operation feeling)−b·g (error) (20)
[0156] In equation (20), "f (operation feel)" is a function that includes the operation feel as a parameter. "g (error)" is a function that includes the deviation (error) between the action intended by the operator operating the robot arm 2A (operation object 2) and the actual action of the robot arm 2A (operation object 2) as a parameter. For example, "f (operation feel)" may be a function whose value increases as the value of the operation feel increases. "g (error)" may be a function whose value increases as the deviation (error) increases. The coefficients "a" and "b" may be arbitrarily determined by the operator.
[0157] The coefficients "a" and "b" may be determined based on the following formulas (21) and (22). The "t" included in the formulas (21) and (22) may be determined arbitrarily by the operator. a=t (21) b=1-t (22)
[0158] Specifically, training of the robot control system 100A (control system 100) represents training of voluntary control. Training of voluntary control is performed after training of the first machine learning model D1. More specifically, during training of the robot control system 100A (control system 100), the operator causes the brain wave detection device 6 to detect the operator's brain waves, causes the intention detection unit 5A (intention detection unit 5) to detect the operator's intention, and causes the control unit 4A (control unit 4) to operate the robot arm 2A (operation target 2) based on the operator's intention. In other words, during training of the robot control system 100A (control system 100), the control unit 4A (control unit 4) controls the robot arm 2A (operation target 2) by voluntary control. That is, the control unit 4A (control unit 4) operates the robot arm 2A (operation target 2) based on the operator's intention.
[0159] During training for the robot control system 100A (control system 100), the worker determines the difference (error) between the motion of the robot arm 2A (operation object 2) intended by the worker and the actual motion of the robot arm 2A (operation object 2). Then, the worker operates the input unit 4c to adjust the weight change condition D13 so as to reduce the difference (error) between the motion of the robot arm 2A (operation object 2) intended by the worker and the actual motion of the robot arm 2A (operation object 2). In other words, the worker adjusts the change condition for the weight Wp so as to reduce the error.
[0160] Furthermore, during training for the robot control system 100A (control system 100), the worker evaluates the ease of operation (operational feel) of the robot arm 2A (operation object 2) using a scale such as a five-point scale. Alternatively, the worker may evaluate the ease of operation (operational feel) of the robot arm 2A (operation object 2) using a visual continuous scale (visual analog scale). The worker then operates the input unit 4c to adjust the weight change condition D13 so as to obtain a higher evaluation. In other words, the worker adjusts the change condition for the weight Wp so as to obtain a higher evaluation.
[0161] Specifically, the operator adjusts the weight change condition D13 (the change condition for the weight Wp) by operating the input unit 4c so that the value of the evaluation function becomes as large as possible, with reference to equation (20). In other words, the operator adjusts the weight change condition D13 (the change condition for the weight Wp) so that the value of the evaluation function becomes a locally optimal solution.
[0162] The robot control system 100A, which is an example of the control system 100 according to the present embodiment, has been described above with reference to FIGS. 2 to 10 . According to this embodiment, voluntary control and autonomous control can be harmoniously combined. As a result, the robot arm 2A (operation target 2) is less likely to move in a manner that differs from the user P's intention. For example, when the robot arm 2A is to grasp an object ST, the farther the wrist 25 is from the object ST, the larger the value of the weight Wp. Therefore, voluntary control becomes dominant, and the robot arm 2A moves as intended by the user P. The closer the wrist 25 is to the object ST, the smaller the value of the weight Wp, and the proportion of voluntary control gradually decreases. Even at this stage, the proportion of voluntary control is not zero. Therefore, if the movement of the robot arm 2A (operation target 2) deviates from the user P's intention, the user P can correct the movement of the robot arm 2A (operation target 2) using their brain waves. In particular, according to this embodiment, the user P's intention can be reflected in the weight Wp. Therefore, the movement of the robot arm 2A (operation target 2) can be more reliably corrected using the user P's brain waves. When the gripper 26 moves close to the object ST (for example, within 30 cm of the object ST), autonomous control becomes dominant, allowing the robot arm 2A to be controlled with higher precision and grip the object ST.
[0163] Furthermore, according to this embodiment, the conditions for changing the weight Wp can be adjusted based on the control performance and the operational feel so that the evaluation function becomes as large as possible. As a result, it is possible to harmonize the robot arm 2A (operation target 2) performing an action that is closer to the intention of the user P with ease of operation by the user P. This improves the convenience of the robot control system 100A (control system 100).
[0164] 2 to 10, the first processing unit 52 acquires the intention of the user P from electroencephalograms by multivariate analysis, but the first processing unit 52 may also acquire the intention of the user P by image recognition by converting the electroencephalogram signal into an image. Specifically, the first storage unit 51 may store a machine learning model for image recognition that is trained to estimate the intention of the user P from an electroencephalogram image, and the first processing unit 52 may acquire the intention of the user P by inputting the electroencephalogram image into the machine learning model for image recognition.
[0165] [Communication System 100B] Next, a communication system 100B, which is an example of the control system 100 of this embodiment, will be described with reference to Figures 11 to 24. However, matters different from those described with reference to Figures 1 to 10 will be described, and descriptions of matters that are the same as those described with reference to Figures 1 to 10 will be omitted as appropriate.
[0166] Fig. 11 is a block diagram showing the configuration of a communication system 100B, which is an example of the control system 100 of this embodiment. Fig. 12 is a diagram showing a two-dimensional avatar 2B controlled by the communication system 100B. The operation target 2 of the communication system 100B is the two-dimensional avatar 2B. The communication system 100B controls the behavior of the two-dimensional avatar 2B (operation target 2). The behavior of the two-dimensional avatar 2B (operation target 2) controlled by the communication system 100B includes speech by the two-dimensional avatar 2B and body language by the two-dimensional avatar 2B.
[0167] As shown in FIG. 11, the communication system 100B includes a two-dimensional avatar 2B, a detection unit 3B, a control unit 4B, an intention detection unit 5B, and an electroencephalogram detection device 6.
[0168] The communication system 100B has a function of controlling the speech content of the two-dimensional avatar 2B based on the brain waves of the user P. In other words, the communication system 100B performs voluntary control. Specifically, the communication system 100B may include a BMI. Here, the user P is an operator who operates the two-dimensional avatar 2B.
[0169] The communication system 100B also has a function of autonomously generating candidates for the content of speech of the two-dimensional avatar 2B. In other words, the communication system 100B performs autonomous control. The communication system 100B, which will be described with reference to Figures 11 to 24, further has a function of autonomously determining the body language of the two-dimensional avatar 2B.
[0170] Detector 3B detects a detection target related to the autonomous control of two-dimensional avatar 2 B. Specifically, detector 3B has an imaging device 31B and a sound collector 32B.
[0171] The imaging device 31B captures an image of the face of the person with whom the two-dimensional avatar 2B is speaking. The imaging device 31B may include, for example, a video camera or a webcam. In the communication system 100B described with reference to FIGS. 11 to 24, the facial expression of the person with whom the two-dimensional avatar 2B is speaking is included in the detection targets related to the autonomous control of the operation target 2 (two-dimensional avatar 2B).
[0172] The imaging device 31B transmits the captured moving image (imaging data) to the control unit 4B. Specifically, the imaging device 31B transmits imaging data (moving image) showing the face of the person with whom the conversation is with the control unit 4B.
[0173] The sound collector 32B collects sound and generates sound data. The sound collector 32B has, for example, a microphone. Specifically, the sound collector 32B collects sound emitted by the speaking partner of the two-dimensional avatar 2B. In other words, the sound collector 32B detects the speech of the speaking partner. In the communication system 100B described with reference to FIGS. 11 to 24, the detection targets related to the autonomous control of the operation target 2 (two-dimensional avatar 2B) include the speech of the speaking partner.
[0174] The sound collector 32B transmits the voice data to the control unit 4B. The voice data indicates the voice collected by the sound collector 32B. Specifically, the voice data indicates the speech of the speaking partner.
[0175] The detection unit 3B (imaging device 31B and sound collector 32B) and the control unit 4B may communicate with each other. For example, the detection unit 3B (imaging device 31B and sound collector 32B) and the control unit 4B may communicate with each other via a LAN or wireless LAN. Alternatively, the detection unit 3B (imaging device 31B and sound collector 32B) and the control unit 4B may communicate with each other via a public network such as the Internet.
[0176] The control unit 4B controls the speech of the two-dimensional avatar 2B based on the output of the imaging device 31B, the output of the sound collector 32B, and the output of the intention detection unit 5B. Specifically, the control unit 4B determines the content of the next speech to be made by the two-dimensional avatar 2B. The next speech of the two-dimensional avatar 2B is an example of the "next action of the operation target 2."
[0177] Specifically, the control unit 4B autonomously generates a sentence ASE indicating a candidate for the next utterance content of the two-dimensional avatar 2B (autonomous control). That is, the control unit 4B generates the sentence ASE through autonomous control. The sentence ASE includes a gap. That is, the control unit 4B generates a sentence ASE with a gap through autonomous control. The control unit 4B determines the remaining part of the next utterance content of the two-dimensional avatar 2B (the utterance content of the gap part of the sentence ASE) based on input from the intention detection unit 5B. In other words, the control unit 4B determines the remaining part of the next utterance content of the two-dimensional avatar 2B based on the brain waves of the user P. As a result, a sentence CSE indicating the next utterance content of the two-dimensional avatar 2B is completed. The control unit 4B causes the two-dimensional avatar 2B to speak based on the completed sentence CSE.
[0178] When multiple candidates (sentence ASE) for the next utterance content of the two-dimensional avatar 2B are determined based on the utterance content of the other party, the control unit 4B determines one of the multiple candidates based on the input from the intention detection unit 5B. In other words, the control unit 4B determines one of the multiple candidates based on the brain waves of the user P.
[0179] The intention detection unit 5B communicates wirelessly with the brain wave detection device 6 and acquires the brain waves of the user P from the brain wave detection device 6. Then, the intention detection unit 5B causes the control unit 4B to determine the remaining part of the next utterance content of the two-dimensional avatar 2B (the utterance content of the blank part of the sentence ASE) based on the brain waves of the user P. As a result, part of the next utterance content of the two-dimensional avatar 2B is controlled arbitrarily. Therefore, the next utterance content of the two-dimensional avatar 2B can include the content intended by the user P.
[0180] When multiple candidates for the next utterance content of the two-dimensional avatar 2B are determined based on the utterance content of the speaking partner, the intention detection unit 5B causes the control unit 4B to select one of the multiple candidates based on the electroencephalogram of the user P. As a result, the candidate for the next utterance content of the two-dimensional avatar 2B is arbitrarily determined.
[0181] The control unit 4B will be further described. As shown in FIG. 11 , the control unit 4B includes an environment recognition unit 41B and a harmony control unit 42B. The environment recognition unit 41B recognizes the environment surrounding the two-dimensional avatar 2B based on the detection results by the detection unit 3B. More specifically, the environment recognition unit 41B recognizes the meaning of the speech content of the interlocutor and the emotion of the interlocutor as the environment surrounding the two-dimensional avatar 2B. Specifically, the environment recognition unit 41B includes a voice processing unit 411, a meaning determination unit 412, and a first emotion determination unit 413.
[0182] The voice processing unit 411 extracts the contents of the speech of the other party based on the voice data received from the sound collector 32 B. The meaning determination unit 412 identifies the meaning of the contents of the speech extracted by the voice processing unit 411 .
[0183] Furthermore, the meaning determination unit 412 determines whether or not the reply to the speech content of the speech partner includes content that should be input arbitrarily, based on the meaning of the speech content of the speech partner.
[0184] The first emotion determination unit 413 analyzes the imaging data (moving images showing the face of the speech partner) received from the imaging device 31B and determines the emotion of the speech partner from the facial expression of the speech partner. For example, the first emotion determination unit 413 may determine whether the speech partner is calm, troubled, angry, etc. from the facial expression of the speech partner.
[0185] The harmony control unit 42B autonomously determines candidates for the next utterance content of the two-dimensional avatar 2B (sentence ASE) based on the output (imaging data) of the imaging device 31B and the output (audio data) of the sound collector 32B. The harmony control unit 42B also voluntarily determines the utterance content of the gap in the sentence ASE (part of the next utterance content of the two-dimensional avatar 2B) based on the brain waves of the user P. The harmony control unit 42B then harmonizes the candidate utterance content (sentence ASE with a gap) determined by the autonomous control with the part of the utterance content (utterance content of the gap) determined by the voluntary control to complete the sentence CSE and control the utterance of the two-dimensional avatar 2B.
[0186] In a communication system 100B described with reference to Figures 11 to 24, an autonomous operation determination unit 44B autonomously determines candidates for the next utterance content (sentence ASE) of the two-dimensional avatar 2B based on the output of a detection unit 3B (autonomous control). Specifically, the autonomous operation determination unit 44B autonomously determines candidates for the next utterance content of the two-dimensional avatar 2B based on the output of an environment recognition unit 41B. In other words, the autonomous operation determination unit 44B generates candidates for the next utterance content of the two-dimensional avatar 2B through autonomous control based on the meaning of the utterance content of the speaking partner and the emotions of the speaking partner.
[0187] The voluntary action determination unit 43B determines part of the next utterance content of the two-dimensional avatar 2B based on the brain waves of the user P (voluntary control). Specifically, the voluntary action determination unit 43B determines the utterance content of the blank part of the sentence ASE based on input from the intention detection unit 5B. As a result, part of the next utterance content of the two-dimensional avatar 2B is determined voluntarily. Furthermore, the voluntary action determination unit 43B determines (selects) one of multiple utterance content candidates based on the brain waves of the user P (voluntary control). Specifically, the voluntary action determination unit 43B determines (selects) one of multiple utterance content candidates based on input from the intention detection unit 5B.
[0188] The weight determination unit 45B determines a weight for voluntary control and a weight for autonomous control based on the output of the environment recognition unit 41B. That is, the weight determination unit 45B determines a weight for voluntary control and a weight for autonomous control based on the meaning of the speech content of the speech partner recognized by the environment recognition unit 41B and the emotions of the speech partner. Hereinafter, the weight for voluntary control may be referred to as a "first weight." Similarly, the weight for autonomous control may be referred to as a "second weight."
[0189] The first weight determines the proportion of the next utterance content of the two-dimensional avatar 2B that will be generated by voluntary control. The second weight determines the proportion of the next utterance content of the two-dimensional avatar 2B that will be generated by autonomous control. Specifically, the first weight determines the proportion of the number of characters in the next utterance content of the two-dimensional avatar 2B that will be generated by voluntary control. For example, the first weight determines the number of blank spaces included in the sentence ASE. The second weight determines the proportion of the number of characters in the next utterance content of the two-dimensional avatar 2B that will be generated by autonomous control. For example, the second weight determines the number of characters in the portion of the sentence ASE other than the blank spaces.
[0190] The weight determination unit 45B dynamically changes the values of the first weight and the second weight according to the meaning of the content of the utterance of the conversation partner. Specifically, the weight determination unit 45B may decrease the first weight and increase the second weight as the proportion of the estimable portion of the reply to the utterance of the conversation partner increases.
[0191] Hereinafter, utterance content determined by voluntary control may be referred to as a "part of utterance by voluntary control." Similarly, utterance content determined by autonomous control may be referred to as a "part of utterance by autonomous control."
[0192] In the communication system 100B described with reference to FIGS. 11 to 24, the weight determination unit 45B further dynamically changes the values of the first weight and the second weight while creating the utterance portion under voluntary control (the speech content of the gap portion of the sentence ASE) based on the output of the intention detection unit 5B. Specifically, the weight determination unit 45B dynamically changes the values of the first weight and the second weight when a portion of the utterance portion under voluntary control (the speech content of the gap portion of the sentence ASE) is determined based on the output of the intention detection unit 5B. Specifically, when the speech content of the gap portion of the sentence ASE is determined, the weight determination unit 45B decreases the first weight and increases the second weight. As a result, a new sentence ASE with a gap is generated. In the new sentence ASE, the amount of the utterance portion under autonomous control increases and the number of gaps decreases.
[0193] The instruction unit 46B instructs the two-dimensional avatar 2B to take the next action. Specifically, the instruction unit 46B harmonizes the speech portion generated by voluntary control with the speech portion generated by autonomous control, and instructs the two-dimensional avatar 2B on the content of the next utterance. More specifically, the instruction unit 46B generates a sentence CSE indicating the content of the next utterance of the two-dimensional avatar 2B by inserting characters indicating the speech portion generated by voluntary control into the blank spaces of the sentence ASE generated by autonomous control. The instruction unit 46B then causes the two-dimensional avatar 2B to speak based on the completed sentence CSE. As a result, the action instructed by the instruction unit 46B to the two-dimensional avatar 2B is an action in which the autonomous action and the voluntary action are harmonized by the first weight and the second weight. In other words, the voluntary control and autonomous control of the next utterance of the two-dimensional avatar 2B are harmonized.
[0194] When the fixed phrase TSE is determined as the next utterance content of the two-dimensional avatar 2B, the instructing unit 46B instructs the two-dimensional avatar 2B on the next utterance content based on the fixed phrase TSE. In other words, the instructing unit 46B causes the two-dimensional avatar 2B to utter the fixed phrase TSE.
[0195] Next, the voluntary action determination unit 43B will be described with reference to Fig. 13. Fig. 13 is a diagram showing the voluntary action determination unit 43B included in the communication system 100B. As shown in Fig. 13, the voluntary action determination unit 43B includes a character input unit 432 and a sentence selection unit 434.
[0196] The character input unit 432 determines the characters to be input into the blank spaces of the sentence ASE based on the output of the intention detection unit 5 B. In other words, the character input unit 432 determines the characters to be input into the blank spaces of the sentence ASE based on the electroencephalogram of the user P.
[0197] When a plurality of sentence ASEs (a plurality of candidates for utterance content) are generated by autonomous control, the sentence selection unit 434 selects (determines) one of the plurality of sentence ASEs based on the output of the intention detection unit 5 B. The sentence selection unit 434 also selects (determines) one of the plurality of fixed phrase TSEs (a plurality of candidates for utterance content) based on the output of the intention detection unit 5 B.
[0198] Next, the autonomous operation determination unit 44B will be described with reference to Fig. 14. Fig. 14 is a diagram showing the autonomous operation determination unit 44B included in the communication system 100B. As shown in Fig. 14, the autonomous operation determination unit 44B includes a sentence prediction unit 442, a second emotion determination unit 444, and a body language determination unit 446.
[0199] The sentence prediction unit 442 determines the portion of speech to be uttered by autonomous control based on the output of the environment recognition unit 41 B. Specifically, the sentence prediction unit 442 autonomously determines candidates for the next speech content of the two-dimensional avatar 2 B based on the meaning of the speech content of the speech partner and the facial expression (emotion) of the speech partner.
[0200] In detail, the sentence prediction unit 442 determines the next utterance content (the part of the utterance by autonomous control) of the two-dimensional avatar 2B by autonomous control based on the meaning of the utterance content of the speaking partner determined by the meaning determination unit 412 shown in FIG. 11 and the emotion of the speaking partner determined by the first emotion determination unit 413 shown in FIG. 11.
[0201] More specifically, the sentence prediction unit 442 determines at least one sentence ASE indicating a candidate for the next utterance content of the two-dimensional avatar 2B based on the meaning of the utterance content of the speech partner, the emotion of the speech partner, and the first weight and second weight determined by the weight determination unit 45B. As already described, the sentence ASE indicating the utterance content generated by autonomous control includes gaps. In other words, the sentence prediction unit 442 generates a sentence ASE with gaps. The gaps are portions into which utterance content generated by voluntary control is incorporated. Specifically, the sentence prediction unit 442 determines the proportion of gap portions (portions of speech generated by voluntary control) and the proportion of portions where characters are generated autonomously (portions of speech generated by autonomous control) based on the first weight and the second weight, and generates a sentence ASE with gaps based on these proportions.
[0202] For example, when the value of the first weight is greater than the value of the second weight, the number of characters in the blank spaces of the entire sentence ASE will be greater than the number of characters generated autonomously. Conversely, when the value of the first weight is smaller than the value of the second weight, the number of characters in the blank spaces of the entire sentence ASE will be less than the number of characters generated autonomously. In other words, the greater the value of the first weight, the greater the number of characters in the blank spaces of the entire sentence ASE. Conversely, the smaller the value of the first weight, the fewer the number of characters in the blank spaces of the entire sentence ASE.
[0203] The sentence prediction unit 442 leaves blank any portion that cannot be predicted as a reply to the speaking partner. Furthermore, the sentence prediction unit 442 may assign a value indicating likelihood to each autonomously generated character. Here, likelihood indicates the degree to which the character is suitable as a reply to the speaking partner. The sentence prediction unit 442 may leave blank any portion that cannot be predicted as a reply to the speaking partner, in order from the character having the lowest likelihood value based on the first weight and the second weight.
[0204] The second emotion determination unit 444 analyzes the content of the next utterance of the two-dimensional avatar 2B to determine the emotion of the user P. Specifically, the second emotion determination unit 444 analyzes the sentence CSE generated by the instruction unit 46B or the fixed phrase TSE determined by optional control to determine the emotion of the user P.
[0205] The body language determination unit 446 determines the body language of the two-dimensional avatar 2B. Specifically, the body language determination unit 446 determines the body language of the two-dimensional avatar 2B depending on the emotion of the user P determined by the second emotion determination unit 444. For example, the body language determination unit 446 may determine whether the user P is calmly responding to the other person, is troubled by the content of the other person's utterance, or is angry by the content of the other person's utterance. The body language determination unit 446 determines different body language depending on whether the user P is calmly responding to the other person, is troubled by the content of the other person's utterance, or is angry by the content of the other person's utterance.
[0206] The instruction unit 46B controls the body language of the two-dimensional avatar 2B based on the body language determined by the body language determination unit 446.
[0207] Next, the avatar system 7 included in the communication system 100B will be described with reference to Fig. 12 and Fig. 15. Fig. 15 is a diagram showing the avatar system 7 included in the communication system 100B. As shown in Fig. 15, the communication system 100B further includes the avatar system 7. The avatar system 7 is a system that displays a two-dimensional avatar 2B and causes the two-dimensional avatar 2B to speak. The avatar system 7 includes a communication unit 71, an avatar display unit 72, an audio output unit 73, a storage unit 74, and a system processing unit 75.
[0208] The communication unit 71 is controlled by the system processing unit 75 and communicates with the control unit 4B. Specifically, the communication unit 71 receives data indicating what the two-dimensional avatar 2B is to speak from the control unit 4B. More specifically, the communication unit 71 receives data indicating a sentence CSE generated by the harmony control unit 42B or data indicating a fixed phrase TSE determined by the harmony control unit 42B. The sentence CSE and the fixed phrase TSE indicate what the two-dimensional avatar 2B will next speak. The communication unit 71 also receives data indicating body language from the control unit 4B. The communication unit 71 may communicate with the control unit 4B via, for example, a LAN or a wireless LAN. Alternatively, the communication unit 71 may communicate with the control unit 4B via a public network such as the Internet. The communication unit 71 may include, for example, a LAN board or a LAN module.
[0209] The avatar display unit 72 is controlled by the system processing unit 75 to display a two-dimensional avatar 2B (see FIG. 12 ). The avatar display unit 72 includes, for example, a liquid crystal display or an organic electroluminescence (EL) display. The system processing unit 75 generates an image of the two-dimensional avatar 2B and causes the avatar display unit 72 to display the two-dimensional avatar 2B.
[0210] The audio output unit 73 outputs audio under the control of the system processing unit 75. Specifically, the system processing unit 75 causes the audio output unit 73 to output the next utterance of the two-dimensional avatar 2B based on data received from the control unit 4B. More specifically, the system processing unit 75 generates audio data from data indicating a text CSE or a fixed phrase TSE, and causes the audio output unit 73 to output audio. As a result, the two-dimensional avatar 2B speaks. The audio output unit 73 includes, for example, a speaker.
[0211] When the two-dimensional avatar 2B next speaks, the system processing unit 75 causes the two-dimensional avatar 2B to use the body language determined by the body language determination unit 446 described with reference to Fig. 14. Specifically, the system processing unit 75 generates an image of the two-dimensional avatar 2B using the body language determined by the body language determination unit 446 described with reference to Fig. 14.
[0212] The storage unit 74 stores various computer programs and various data. The storage unit 74 includes, for example, a semiconductor memory. The semiconductor memory includes, for example, a RAM and a ROM. The semiconductor memory may include a flash memory, an EPROM, or an EEPROM, or may include a VRAM. The storage unit 74 may include an HDD or an SSD, or may include removable media.
[0213] The system processing unit 75 controls the operation of each unit of the avatar system 7 by executing a computer program stored in the storage unit 74. The system processing unit 75 may include, for example, a CPU or an MPU. The system processing unit 75 may further include a GPU.
[0214] Next, the intention detection unit 5B included in the communication system 100B will be described with reference to Fig. 16. Fig. 16 is a block diagram showing the configuration of the intention detection unit 5B included in the communication system 100B. As shown in Fig. 16, the intention detection unit 5B has a first storage unit 51, a first processing unit 52, and a wireless communication unit 53, similar to the intention detection unit 5A described with reference to Fig. 7.
[0215] In a communication system 100B described with reference to Figures 11 to 24, a first storage unit 51 stores a first machine learning model D2. The first machine learning model D2 is a computer program that analyzes electroencephalograms to estimate the intention of a user P. Specifically, the first machine learning model D2 is trained to analyze electroencephalograms and estimate an input to a control unit 4B. A first processing unit 52 outputs data indicating the input content estimated by the first machine learning model D2 to the control unit 4B.
[0216] In addition, the first processing unit 52, similar to the first processing unit 52 described with reference to Figure 7, performs bandpass filtering on the electrical signal indicating the brain waves, and inputs the brain wave data after bandpass filtering into the first machine learning model D2.
[0217] The first machine learning model D2 may perform analysis using support vector regression. However, the machine learning algorithm for constructing the first machine learning model D2 is not limited to support vector regression. The machine learning algorithm for constructing the first machine learning model D2 may be, for example, a decision tree, a nearest neighbor method, a naive Bayes classifier, or a neural network. Backpropagation may be used for the machine learning.
[0218] Next, the control unit 4B included in the communication system 100B will be described with reference to Figures 17 and 18. Figure 17 is a block diagram showing the hardware of the control unit 4B included in the communication system 100B. As shown in Figure 17, the control unit 4B has a second storage unit 81, a display unit 82, a communication unit 84, and a second processing unit 85. Figure 18 is a diagram showing computer programs and data stored in the second storage unit 81 of the control unit 4B included in the communication system 100B. The control unit 4B may be, for example, a personal computer.
[0219] The second storage unit 81 stores various computer programs and various data. The second storage unit 81 includes, for example, a semiconductor memory. The semiconductor memory includes, for example, a RAM and a ROM. The semiconductor memory may include a flash memory, an EPROM, or an EEPROM, or may include a VRAM. The second storage unit 81 may further include an HDD or an SSD, or may include removable media.
[0220] The display unit 82 is controlled by the second processing unit 85 to display various screens GA. The display unit 82 includes, for example, a liquid crystal display or an organic EL display. The various screens GA include a first screen GA1 to a sixth screen GA6, which will be described with reference to FIGS. 19 to 23.
[0221] The communication unit 84 is controlled by the second processing unit 85 and communicates with the communication unit 71 (see FIG. 15 ) of the avatar system 7. The communication unit 84 transmits text data indicating the next utterance content of the two-dimensional avatar 2B to the avatar system 7. The communication unit 84 also transmits data indicating the body language to be used by the two-dimensional avatar 2B when the two-dimensional avatar 2B next speaks to the avatar system 7. The communication unit 84 may include, for example, a LAN board or a LAN module.
[0222] The second processing unit 85 performs various processes such as numerical calculations and information processing by executing computer programs stored in the second storage unit 81. The second processing unit 85 may include, for example, a CPU or an MPU. The second processing unit 85 may further include a GPU or an NPU. Alternatively, the second processing unit 85 may include a quantum computer.
[0223] Specifically, the second processing unit 85 executes the computer programs stored in the second storage unit 81, thereby functioning as the voice processing unit 411, meaning determination unit 412, first emotion determination unit 413, weight determination unit 45B, instruction unit 46B, character input unit 432, sentence selection unit 434, sentence prediction unit 442, second emotion determination unit 444, and body language determination unit 446 described with reference to Figures 11, 13, and 14. In more detail, as shown in Figure 18, the second storage unit 81 stores second machine learning models D21 to D26.
[0224] The second machine learning model D21 is trained to analyze voice data and extract the content of the speech. For example, the second machine learning model D21 may be trained to convert the voice data into a multidimensional vector. When functioning as the voice processing unit 411, the second processing unit 85 inputs the voice data (voice of the speaking partner) received from the detection unit 3B (sound collector 32B) to the second machine learning model D21. As a result, data indicating the content of the speaking partner's speech is output from the second machine learning model D21 to the second processing unit 85.
[0225] The third machine learning model D22 is trained to determine the meaning of the speech content of the speech partner. For example, the third machine learning model D22 may be trained to determine meaning from a multidimensional vector. Furthermore, the third machine learning model D22 is trained to determine whether a fixed phrase TSE can be used to reply to the speech content of the speech partner based on the meaning of the speech content of the speech partner. For example, when training the third machine learning model D22, data indicating the meaning of the speech content of the expected speech partner and whether a fixed phrase TSE can be used to reply may be used. In other words, the training data for the third machine learning model D22 may include data combining the meaning of the speech content of the expected speech partner and data indicating whether a fixed phrase TSE can be used to reply.
[0226] When functioning as the meaning determination unit 412, the second processing unit 85 inputs data indicating the utterance content extracted by the second machine learning model D21 to the third machine learning model D22. As a result, data indicating the meaning of the utterance content of the other party and a determination result indicating whether or not a fixed phrase TSE can be used to reply to the utterance content of the other party are output from the third machine learning model D22 to the second processing unit 85.
[0227] The fourth machine learning model D23 is trained to analyze imaging data and determine the emotion of the speech partner from the facial expression of the speech partner. When functioning as the first emotion determination unit 413, the second processing unit 85 inputs imaging data (moving images showing the speech partner's face) received from the detection unit 3B (imaging device 31B) to the fourth machine learning model D23. As a result, data indicating the emotion of the speech partner is output from the fourth machine learning model D23 to the second processing unit 85.
[0228] The fifth machine learning model D24 is trained to analyze the meaning of the utterance content of the speech partner and the emotion of the speech partner to determine the values of the first weight and the second weight. For example, training data used when training the fifth machine learning model D24 may be created based on the meaning of the utterance content of the expected speech partner, the expected emotion of the speech partner, the number of characters in the utterance content expected as a reply to the speech partner, and the number of characters to be arbitrarily input in the utterance content of the reply.
[0229] When functioning as the weight determination unit 45B, the second processing unit 85 inputs data indicating the meaning of the utterance content determined by the third machine learning model D22 and data indicating the emotion of the speaking partner determined by the fourth machine learning model D23 to the fifth machine learning model D24. As a result, data indicating the values of the first weight and the second weight are output from the fifth machine learning model D24 to the second processing unit 85.
[0230] Furthermore, the fifth machine learning model D24 is trained to decrease the first weight and increase the second weight when the data input from the intention detection unit 5B indicates that the speech content of a part of the gap in the sentence ASE is to be determined. The second processing unit 85 inputs the data input from the intention detection unit 5B to the fifth machine learning model D24 while creating the speech part (the speech content of the gap in the sentence ASE) through voluntary control. As a result, the values of the first weight and the second weight change dynamically.
[0231] The fifth machine learning model D24 may be trained to determine the value of the first weight by analyzing the meaning of the speech content of the speaking partner and the emotions of the speaking partner. In this case, the second processing unit 85 may determine the value of the second weight based on, for example, the following equation (23): second weight=1−first weight (23)
[0232] The sixth machine learning model D25 is trained to predict a reply to a speech partner based on the meaning of the speech content of the speech partner and the emotions of the speech partner. When training the sixth machine learning model D25, training data is used that combines the meaning of the speech content of the expected speech partner, the expected emotions of the speech partner, the expected next utterance content of the two-dimensional avatar 2B (the content of the reply to the speech partner), and the values of the first weight and the second weight determined according to the content of the reply.
[0233] When functioning as the sentence prediction unit 442, the second processing unit 85 inputs data indicating the meaning of the utterance content determined by the third machine learning model D22, data indicating the emotion of the speaking partner determined by the fourth machine learning model D23, and the values of the first weight and the second weight determined by the fifth machine learning model D24 to the sixth machine learning model D25. As a result, candidates for the next utterance content of the two-dimensional avatar 2B (data indicating the sentence ASE) are output from the sixth machine learning model D25 to the second processing unit 85.
[0234] The seventh machine learning model D26 is trained to determine the emotion of the user P from the next utterance content of the two-dimensional avatar 2B. When functioning as the second emotion determination unit 444, the second processing unit 85 inputs data indicating the next utterance content of the two-dimensional avatar 2B to the seventh machine learning model D26. As a result, data indicating the emotion of the user P is output from the seventh machine learning model D26 to the second processing unit 85.
[0235] The machine learning algorithm for constructing the second machine learning model D21 to the seventh machine learning model D26 is not particularly limited and may be, for example, a decision tree, a nearest neighbor method, a naive Bayes classifier, a support vector machine, or a neural network. Backpropagation may be used for machine learning. The machine learning algorithm for constructing the second machine learning model D21 may be a Transformer. The Transformer is a deep learning model.
[0236] Next, the control unit 4B will be further described with reference to Figures 17 and 18. As shown in Figure 18, the second storage unit 81 further stores fixed phrase data D27 and body language data D28.
[0237] The fixed phrase data D27 indicates various fixed phrases TSE. Specifically, the fixed phrase data D27 may indicate "Good morning," "Hello," "Good evening," "Welcome," "What are you looking for?", etc. When the third machine learning model D22 determines that a reply to the content of the utterance of the other party with a fixed phrase TSE is possible, the second processing unit 85 refers to the fixed phrase data D27 and causes the display unit 82 to display a list of fixed phrases TSE.
[0238] The body language data D28 indicates various types of body language. The various types of body language include body language corresponding to various emotions. For example, the body language data D28 may indicate body language suitable for calmly responding to the other person, body language suitable for being troubled by the content of the other person's utterance, and body language suitable for being angry by the content of the other person's utterance. When functioning as the body language determination unit 446, the second processing unit 85 determines body language expressing the user P's emotions from the body language data D28 based on the output from the seventh machine learning model D26.
[0239] Next, the operation of the communication system 100B when the user P controls the two-dimensional avatar 2B will be described with reference to Figures 17 to 19. Figures 19(a) to 19(c) are diagrams showing screens GA that are displayed on the display unit 82 when the user P is making the two-dimensional avatar 2B speak using a fixed phrase TSE. In detail, Figures 19(a) and 19(b) show screens GA that are displayed on the display unit 82 when the user P is making the two-dimensional avatar 2B speak using a fixed phrase TSE. Figure 19(c) shows screen GA that is displayed on the display unit 82 when the user P is deciding whether to confirm the next utterance content of the two-dimensional avatar 2B.
[0240] As shown in Fig. 19(a), when the two-dimensional avatar 2B is made to speak using a fixed phrase TSE, the second processing unit 85 refers to the fixed phrase data D27 and causes the display unit 82 to display a screen GA (first screen GA1) showing a list of fixed phrases TSE. Fig. 19(a) illustrates the first screen GA1 on which five fixed phrases TSE1 to TSE5 are arranged vertically. Note that "x" represents a letter.
[0241] The second processing unit 85 further displays a first cursor CA1 on the first screen GA1. The first cursor CA1 is a cursor for selecting a sentence. The first cursor CA1 may enclose one of the plurality of standard phrases TSE. FIG. 19(a) illustrates the first cursor CA1 selecting the standard phrase TSE1 from the five standard phrases TSE1 to TSE5. In this case, the first cursor CA1 may enclose the standard phrase TSE1.
[0242] 19(a) and 19(b), the second processing unit 85 moves the first cursor CA1 in the direction in which the fixed phrases TSE are arranged. For example, the second processing unit 85 moves the first cursor CA1 vertically. As a result, the fixed phrases TSE selected by the first cursor CA1 change over time.
[0243] While the first screen GA1 is being displayed, the intention detection unit 5B (first processing unit 52) and the control unit 4B (second processing unit 85) select one of the plurality of standard phrases TSE based on the brain waves of the user P. Specifically, the user P thinks about the first action at the timing when the standard phrase TSE of the content that the user wants the two-dimensional avatar 2B to speak is selected by the first cursor CA1.
[0244] The first action indicates an action of the user P. The first action that the user P thinks of is not particularly limited as long as it is an action that can be detected from electroencephalograms. For example, the first action may be an action of the user P raising their hand or an action of the user P moving their finger. The first action may be determined arbitrarily. Data that associates the first action with the stopping of the first cursor CA1 is stored in advance in the first storage unit 51 (see FIG. 16 ) of the intention detection unit 5B.
[0245] When the electroencephalogram of the user P indicates the first movement, the intention detection unit 5B (first processing unit 52) generates a command to stop the movement of the first cursor CA1 and transmits the command to the control unit 4B. As a result, the second processing unit 85 stops the first cursor CA1. As shown in FIG. 19( c), after stopping the first cursor CA1, the second processing unit 85 displays the second screen GA2 on the display unit 82.
[0246] The second screen GA2 displays the fixed phrase TSE selected based on the electroencephalogram of the user P. The second screen GA2 further displays a first button SB1, a second button SB2, and a second cursor CA2. Fig. 19(c) illustrates a case where a fixed phrase TSE3 is selected based on the electroencephalogram of the user P.
[0247] The first button SB1 is a soft button for confirming the next utterance content of the two-dimensional avatar 2B. The second button SB2 is a soft button for canceling the selection of the template TSE. In other words, the second button SB2 is a cancel button. The second cursor CA2 may surround one of the first button SB1 and the second button SB2.
[0248] The second processing unit 85 moves the second cursor CA2 in the direction in which the first button SB1 and the second button SB2 are aligned, so that the soft button selected by the second cursor CA2 changes over time between the first button SB1 and the second button SB2.
[0249] When determining the content of the template TSE, the intention detection unit 5B (first processing unit 52) and the control unit 4B (second processing unit 85) press the first button SB1 based on the user P's brain waves while the second screen GA2 is being displayed.
[0250] Specifically, when user P determines the next utterance content of the two-dimensional avatar 2B based on the content of the standard phrase TSE displayed on the second screen GA2, he or she thinks about the second action at the time when the first button SB1 is selected by the second cursor CA2.
[0251] The second action indicates an action of the user P. The second action that the user P thinks of is not particularly limited as long as it is an action that can be detected from electroencephalograms. The second action can be determined arbitrarily. The second action may be the same as the first action, or may be a different action from the first action.
[0252] The first storage unit 51 (see FIG. 16 ) of the intention detection unit 5B pre-stores data associating the second action with the stopping of the second cursor CA2. When the user P's electroencephalogram indicates the second action, the intention detection unit 5B (first processing unit 52) generates a command to stop the movement of the second cursor CA2 and transmits the command to the control unit 4B. As a result, the second processing unit 85 presses the first button SB1 to confirm the content of the template TSE. In other words, the second processing unit 85 presses the soft button selected by the second cursor CA2.
[0253] When canceling the selection of the standard phrase TSE, the intention detection unit 5B (first processing unit 52) and the control unit 4B (second processing unit 85) press the second button SB2 based on the user P's brain waves while the second screen GA2 is being displayed.
[0254] Specifically, when the user P wants to change the next utterance content of the two-dimensional avatar 2B to a content different from the content of the fixed phrase TSE displayed on the second screen GA2, the user P thinks about the second action while the second button SB2 is being selected by the second cursor CA2. As a result, the second processing unit 85 presses the second button SB2 to cancel the selection of the fixed phrase TSE. When the second button SB2 is pressed, the second processing unit 85 causes the display unit 82 to display the first screen GA1 again.
[0255] Next, with reference to Figures 17, 18, and 20 to 23, the operation of the communication system 100B when determining the next utterance content of the two-dimensional avatar 2B through voluntary control and autonomous control will be described. Figures 20(a) to 20(c) are diagrams showing screens GA displayed on the display unit 82 when selecting one of candidates (plurality of sentences ASE) for the next utterance content of the two-dimensional avatar 2B. Figures 21(a), 21(b), 22(a), and 22(b) are diagrams showing screens GA displayed on the display unit 82 when the user P inputs any characters. Figure 23 is a diagram showing screen GA displayed on the display unit 82 when deciding whether to determine the next utterance content of the two-dimensional avatar 2B.
[0256] As shown in FIG. 20(a), when the second processing unit 85 autonomously generates candidates (multiple sentence ASEs) for the next utterance content of the two-dimensional avatar 2B, it causes the display unit 82 to display a screen GA (third screen GA3) showing a list of the candidate utterance content (sentence ASEs). FIG. 20(a) illustrates the third screen GA3 in which five candidates (sentence ASE11 to sentence ASE15) are arranged vertically. As already explained, the sentence ASEs contain blank spaces. In FIGS. 20(a) to 20(c), the portions indicated by "◯" are blank spaces. Note that "x" represents a letter.
[0257] The second processing unit 85 further displays a first cursor CA1 on the third screen GA3. The first cursor CA1 on the third screen GA3 selects a candidate for the speech content (sentence ASE). The first cursor CA1 may surround one candidate (sentence ASE) among multiple candidates (sentence ASE).
[0258] 20(a) and 20(b), the second processing unit 85 moves the first cursor CA1 in the direction in which the candidates for the utterance content (sentence ASE) are arranged. As a result, the candidates (sentence ASE) selected by the first cursor CA1 change over time.
[0259] Similar to the case of selecting a fixed phrase TSE based on the electroencephalogram of the user P, the intention detection unit 5B (first processing unit 52) and the control unit 4B (second processing unit 85) select one candidate (sentence ASE) from among a plurality of candidates (sentence ASE) based on the electroencephalogram of the user P while the third screen GA3 is being displayed. As a result, the second processing unit 85 causes the display unit 82 to display the fourth screen GA4, as shown in FIG. 20(c).
[0260] The fourth screen GA4 displays candidate utterance contents (sentence ASE) selected based on the electroencephalogram of the user P while the third screen GA3 is displayed. The fourth screen GA4 further displays a first button SB1, a second button SB2, and a second cursor CA2, similar to the second screen GA2 described with reference to Fig. 19(c). Fig. 20(c) illustrates an example in which a sentence ASE13 is selected based on the electroencephalogram of the user P.
[0261] As in the case of the fixed phrase TSE, when the first button SB1 is pressed based on the user P's brain waves while the fourth screen GA4 is displayed, the candidate utterance content (sentence ASE) is confirmed. Also, when the second button SB2 is pressed based on the user P's brain waves while the fourth screen GA4 is displayed, the selection of the candidate utterance content (sentence ASE) is canceled. When the second button SB2 is pressed, the second processing unit 85 causes the display unit 82 to display the third screen GA3 again.
[0262] When the candidate speech content (sentence ASE) is confirmed, the second processing unit 85 displays the fifth screen GA5 on the display unit 82, as shown in Fig. 21(a). The fifth screen GA5 displays the Japanese syllabary JS and the third cursor CA3.
[0263] The third cursor CA3 is a cursor that selects one of the multiple rows included in the 50-sound table JS. The third cursor CA3 may surround one of the multiple rows included in the 50-sound table JS. Figure 21 (a) illustrates the third cursor CA3 that has selected the "A" row in the 50-sound table JS. In this case, the third cursor CA3 may surround the "A" row.
[0264] 21(a) and 21(b), the second processing unit 85 moves the third cursor CA3 in the direction in which the rows of the Japanese syllabary JS are arranged. For example, the second processing unit 85 moves the third cursor CA3 horizontally. As a result, the row selected by the third cursor CA3 changes over time.
[0265] While the third cursor CA3 is displayed, the intention detection unit 5B (first processing unit 52) and the control unit 4B (second processing unit 85) select one of the multiple rows included in the Japanese syllabary JS based on the brain waves of the user P. For example, when the user P selects the "na" row, the user P thinks about the third action at the time the "na" row is selected by the third cursor CA3.
[0266] The third action indicates an action of the user P. The third action that the user P thinks of is not particularly limited as long as it is an action that can be detected from electroencephalograms. The third action can be determined arbitrarily. The third action may be the same as the first action or the second action, or may be a different action from the first action and the second action.
[0267] The first storage unit 51 (see FIG. 16 ) of the intention detection unit 5B pre-stores data associating the third action with the stopping of the third cursor CA3. When the user P's electroencephalogram indicates the third action, the intention detection unit 5B (first processing unit 52) generates a command to stop the movement of the third cursor CA3 and transmits the command to the control unit 4B. As a result, the second processing unit 85 stops the third cursor CA3.
[0268] As shown in FIG. 22(a), the second processing unit 85 stops the third cursor CA3, and then displays a fourth cursor CA4 on the fifth screen GA5 in place of the third cursor CA3.
[0269] The fourth cursor CA4 is a cursor that selects each character included in the 50-sound table JS one by one. The fourth cursor CA4 may circle one of the characters included in the 50-sound table JS. FIG. 22( a) illustrates the fourth cursor CA4 selecting the character "na." In this case, the fourth cursor CA4 may circle the character "na."
[0270] 22(a) and 22(b), the second processing unit 85 moves the fourth cursor CA4 in the direction in which multiple rows (columns) included in the Japanese syllabary JS are arranged. For example, the second processing unit 85 moves the fourth cursor CA4 vertically. More specifically, the second processing unit 85 moves the fourth cursor CA4 so as to sequentially select each character included in the row selected based on the user P's electroencephalogram. For example, if the "na" row is selected based on the user P's electroencephalogram, each character included in the "na" row is sequentially selected by the fourth cursor CA4.
[0271] While the fourth cursor CA4 is displayed, the intention detection unit 5B (first processing unit 52) and the control unit 4B (second processing unit 85) select one of the characters included in the Japanese syllabary JS based on the brain waves of the user P. For example, when the user P selects the character "ne" included in the "na" row, the user P thinks of the fourth action in his or her mind at the time the character "ne" is selected by the fourth cursor CA4.
[0272] The fourth action indicates an action of the user P. The fourth action that the user P thinks of is not particularly limited as long as it is an action that can be detected from electroencephalograms. The fourth action can be determined arbitrarily. The fourth action may be the same as the first action, the second action, or the third action, or may be a different action from any of the first action, the second action, and the third action.
[0273] The first storage unit 51 (see FIG. 16 ) of the intention detection unit 5B pre-stores data associating the fourth movement with the stopping of the fourth cursor CA4. When the user P's electroencephalogram indicates the fourth movement, the intention detection unit 5B (first processing unit 52) generates a command to stop the movement of the fourth cursor CA4 and transmits the command to the control unit 4B. As a result, the second processing unit 85 stops the fourth cursor CA4.
[0274] The second processing unit 85 inserts the character selected by the fourth cursor CA4 when the fourth cursor CA4 is stopped into the blank space of the candidate utterance content (sentence ASE) confirmed on the fourth screen GA4 (see FIG. 20(c)). After that, the intention detection unit 5B and the control unit 4B repeat the processes described with reference to FIGS. 21(a) to 22(b) until all blank spaces included in the candidate utterance content (sentence ASE) are filled with characters.
[0275] When all blank spaces included in the candidate utterance content (sentence ASE) have been filled with characters, the second processing unit 85 displays a sixth screen GA6 on the display unit 82, as shown in Fig. 23. The sixth screen GA6 displays the completed sentence CSE. The sixth screen GA6 further displays a first button SB1, a second button SB2, and a second cursor CA2, similar to the second screen GA2 described with reference to Fig. 19(c).
[0276] As in the case of the fixed phrase TSE, when the first button SB1 is pressed based on the user P's brain waves while the sixth screen GA6 is being displayed, the next utterance content (sentence CSE) of the two-dimensional avatar 2B is confirmed. Also, when the second button SB2 is pressed based on the user P's brain waves while the sixth screen GA6 is being displayed, the next utterance content (sentence CSE) of the two-dimensional avatar 2B is canceled. When the second button SB2 is pressed, the second processing unit 85 causes the display unit 82 to redisplay the third screen GA3 (see FIG. 20(a)). Alternatively, when the second button SB2 is pressed, the second processing unit 85 may cause the display unit 82 to redisplay the fifth screen GA5 (see FIG. 21(a)).
[0277] Next, the communication system 100B will be described with reference to Figs. 11 to 24. Fig. 24 is a flowchart showing the processing executed by the intention detection unit 5B and the control unit 4B included in the communication system 100B. The processing shown in Fig. 24 starts when the speech partner speaks. In other words, the processing shown in Fig. 24 starts when the sound collector 32B collects the voice (utterance) of the speech partner.
[0278] When the sound collector 32B detects sound, the second processing unit 85 of the control unit 4B acquires the content of the speech of the other party via the sound processing unit 411 (step S41). Specifically, the second processing unit 85 inputs the speech data received from the sound collector 32B to the second machine learning model D21. As a result, data indicating the content of the speech of the other party is output from the second machine learning model D21 to the second processing unit 85.
[0279] When the second processing unit 85 acquires the utterance content of the other party, the meaning determination unit 412 acquires the meaning of the utterance content. Then, the second processing unit 85 determines whether a reply using a fixed phrase TSE is possible based on the meaning of the utterance content (step S42). Specifically, the second processing unit 85 inputs data indicating the utterance content extracted by the second machine learning model D21 to the third machine learning model D22. As a result, a result of estimating the meaning of the utterance content of the other party is output from the third machine learning model D22 to the second processing unit 85. Furthermore, a determination result indicating whether a reply using a fixed phrase TSE is possible to the utterance content of the other party is output from the third machine learning model D22 to the second processing unit 85.
[0280] If a reply using the standard phrase TSE is possible (Yes in step S42), the second processing unit 85 displays the first screen GA1 on the display unit 82 by referring to the standard phrase data D27, as described with reference to Figure 19 (a) (step S43).
[0281] When the first screen GA1 is displayed, one of the plurality of fixed phrases TSE is selected by optional control (step S44). In detail, as described with reference to Figures 19(a) to 19(c), the intention detection unit 5B (first processing unit 52) and the control unit 4B (second processing unit 85) determine one of the plurality of fixed phrases TSE based on the electroencephalogram of the user P.
[0282] Once the fixed phrase TSE is confirmed, the second processing unit 85 determines the emotion of the user P from the confirmed fixed phrase TSE using the second emotion determination unit 444 (step S45). Specifically, the second processing unit 85 inputs data indicating the next utterance content of the two-dimensional avatar 2B to the seventh machine learning model D26. As a result, data indicating the emotion of the user P is output from the seventh machine learning model D26 to the second processing unit 85. If a reply using the fixed phrase TSE is possible (Yes in step S42), the second processing unit 85 inputs the confirmed fixed phrase TSE to the seventh machine learning model D26 through optional control, and determines the emotion of the user P.
[0283] After determining the emotion of the user P, the second processing unit 85 refers to the body language data D28 and determines the body language that expresses the emotion of the user P (step S46).
[0284] After determining the next utterance content and body language of the two-dimensional avatar 2B, the second processing unit 85 transmits data indicating the next utterance content of the two-dimensional avatar 2B and data indicating the body language to the avatar system 7 via the communication unit 84 (step S47). As a result, the processing shown in FIG. 24 ends.
[0285] On the other hand, if a reply using a fixed phrase TSE is not possible (No in step S42), the second processing unit 85 determines the emotion of the other party based on the facial expression of the other party captured in the video received from the imaging device 31B using the first emotion determination unit 413 (step S51). Specifically, the second processing unit 85 inputs the imaging data (video showing the face of the other party) received from the imaging device 31B to the fourth machine learning model D23. As a result, data indicating the emotion of the other party is output from the fourth machine learning model D23 to the second processing unit 85.
[0286] After determining the emotion of the speech partner, the second processing unit 85 calculates the values of the first weight and the second weight (step S52). Specifically, the second processing unit 85 inputs data indicating the meaning of the speech content estimated in step S42 and data indicating the emotion of the speech partner determined in step S51 to the fifth machine learning model D24. As a result, data indicating the values of the first weight and the second weight are output from the fifth machine learning model D24 to the second processing unit 85.
[0287] After calculating the values of the first weight and the second weight, the second processing unit 85 predicts a reply to the other party based on the meaning of the utterance content of the other party and the emotion of the other party, and displays a list of candidates for the next utterance content of the two-dimensional avatar 2B on the display unit 82 (step S53). That is, the second processing unit 85 displays the third screen GA3 on the display unit 82, as described with reference to FIG. 20(a).
[0288] Specifically, the second processing unit 85 inputs data indicating the meaning of the utterance content estimated in step S42, data indicating the emotion of the speech partner determined in step S51, and the values of the first weight and the second weight calculated in step S52 to the sixth machine learning model D25. As a result, candidates for the next utterance content of the two-dimensional avatar 2B (sentence ASE with a blank) are output from the sixth machine learning model D25 to the second processing unit 85. The second processing unit 85 causes the display unit 82 to display a list of the candidates for the utterance content output from the sixth machine learning model D25.
[0289] When the third screen GA3 is displayed, one of the multiple candidates (sentence ASE) is selected by optional control (step S54). In detail, as described with reference to Figures 20(a) to 20(c), the intention detection unit 5B (first processing unit 52) and the control unit 4B (second processing unit 85) determine one of the multiple candidates based on the brain waves of the user P.
[0290] Once the candidate for the next utterance content of the two-dimensional avatar 2B is determined, characters are input into the blank spaces included in the candidate utterance content by optional control (step S55). In detail, as described with reference to Figures 21(a), 21(b), 22(a), 22(b), and 23, the intention detection unit 5B (first processing unit 52) and the control unit 4B (second processing unit 85) select characters from the Japanese syllabary JS based on the electroencephalogram of the user P and insert them into the blank spaces. As a result, the next utterance content of the two-dimensional avatar 2B is determined.
[0291] Once the next utterance content of the two-dimensional avatar 2B is determined, the second processing unit 85 determines the emotion of the user P from the next utterance content (text CSE) of the two-dimensional avatar 2B using the second emotion determination unit 444 (step S45).
[0292] After determining the emotion of the user P, the second processing unit 85 refers to the body language data D28 and determines the body language that expresses the emotion of the user P (step S46).
[0293] After determining the next utterance content and body language of the two-dimensional avatar 2B, the second processing unit 85 transmits data indicating the next utterance content of the two-dimensional avatar 2B and data indicating the body language to the avatar system 7 via the communication unit 84 (step S47). As a result, the processing shown in FIG. 24 ends.
[0294] Next, the fifth machine learning model D24 (see FIG. 18 ) will be further described. The fifth machine learning model D24 is an example of a “change condition.” That is, the first weight and the second weight are changed by the fifth machine learning model D24.
[0295] In the communication system 100B, the fifth machine learning model D24 (the change conditions for the first weight and the second weight) is adjusted based on the evaluation function. Adjusting the fifth machine learning model D24 indicates retraining the fifth machine learning model D24. When the fifth machine learning model D24 is retrained, training data adjusted based on the evaluation function is input to the fifth machine learning model D24.
[0296] As already described, the training data for the fifth machine learning model D24 is created based on the meaning of the utterance content of the expected interlocutor, the expected emotions of the interlocutor, the number of characters in the utterance content expected as a reply to the interlocutor, and the number of characters to be optionally input in the utterance content of the reply. The number of characters to be optionally input in the utterance content of the reply corresponds to the number of "blank spaces" included in the sentence ASE indicating candidates for the next utterance content of the two-dimensional avatar 2B. In the communication system 100B, when the fifth machine learning model D24 is retrained, the "number of blank spaces" to be input as training data to the fifth machine learning model D24 is adjusted based on an evaluation function. The evaluation function includes, as parameters, the control performance and operability acquired during training of the communication system 100B.
[0297] The control performance indicates a value reflecting the amount of deviation between the number of blank spaces predicted by the worker operating the two-dimensional avatar 2B during training of the communication system 100B and the number of blank spaces included in the candidate utterance content determined by the sentence prediction unit 442 (second processing unit 85). In other words, the control performance indicates a value reflecting the amount of deviation between the number of blank spaces included in the next utterance content predicted by the worker and the number of blank spaces included in the candidate utterance content displayed on the third screen GA3. For example, the control performance may indicate the difference between the number of blank spaces predicted by the worker and the actual number of blank spaces.
[0298] The operability indicates a value reflecting the ease of operation of the speech content evaluated by the worker. The operability may be evaluated using a scale such as a five-point scale. Alternatively, the operability may be evaluated using a visual continuous scale (visual analog scale). For example, the worker may rate the operability low when the actual number of blank spaces differs from the predicted number. Furthermore, the worker may rate the operability high when the actual number of blank spaces differs from the predicted number but the worker feels that the actual number of blank spaces makes it easier to complete the intended sentence.
[0299] For example, the number of blank spaces included in the training data may be adjusted using the following equation (24) as the evaluation function: Evaluation function=a·f (operation feel)−b·g (error) (24)
[0300] In equation (24), "f (operation feel)" is a function that includes the operation feel as a parameter. "g (error)" is a function that includes the deviation (error) between the number of voids predicted by the operator and the actual number of voids as a parameter. For example, "f (operation feel)" may be a function whose value increases as the value of the operation feel increases. "g (error)" may be a function whose value increases as the deviation (error) increases. The coefficients "a" and "b" may be arbitrarily determined by the operator.
[0301] The coefficients "a" and "b" may be determined based on the following formulas (25) and (26). The "t" included in formulas (25) and (26) may be determined arbitrarily by the operator. a=t (25) b=1-t (26)
[0302] Specifically, the worker refers to equation (24) and adjusts the data included in the training data (the number of blank spaces) so that the value of the evaluation function becomes as large as possible. As a result, the change conditions for the first and second weights (fifth machine learning model D24) are adjusted so that the value of the evaluation function becomes as large as possible. In other words, the worker adjusts the data included in the training data (the number of blank spaces) so that the value of the evaluation function becomes a locally optimal solution. As a result, the change conditions for the first and second weights (fifth machine learning model D24) are adjusted so that the value of the evaluation function becomes a locally optimal solution.
[0303] The communication system 100B, which is an example of the control system 100 of this embodiment, has been described above with reference to FIGS. 11 to 24. According to this embodiment, voluntary control and autonomous control can be harmonized. As a result, the next utterance content of the two-dimensional avatar 2B (the action of the operation target 2) is less likely to differ from the intention of the user P. Furthermore, by harmonizing voluntary control and autonomous control, it becomes easier to create a sentence CSE using voluntary control. In other words, the burden on the user P can be reduced. In particular, according to this embodiment, the intention of the user P can be reflected in each value of the first weight and the second weight. As a result, the burden on the user P can be further reduced.
[0304] Furthermore, according to this embodiment, the conditions for changing the first and second weights (fifth machine learning model D24) can be adjusted based on the control performance and the operational feel so as to maximize the evaluation function. In other words, the proportion of speech portions based on voluntary control and the proportion of speech portions based on autonomous control are adjusted so as to maximize the evaluation function. Therefore, it is possible to create a sentence that is closer to the user P's intention as a candidate for the next speech content of the two-dimensional avatar 2B (operation target 2) while harmonizing the ease with which the user P can operate the speech content. This improves the convenience of the communication system 100B.
[0305] 11 to 24, at least one candidate for the next utterance content of the two-dimensional avatar 2B is determined based on the utterance content of the speech partner, the emotion of the speech partner, the first weight, and the second weight. However, the sentence prediction unit 442 may determine at least one candidate for the next utterance content of the two-dimensional avatar 2B based on the utterance content of the speech partner, the first weight, and the second weight. In other words, the sentence prediction unit 442 may determine the content of a reply to the speech partner by using only the utterance content of the speech partner out of the utterance content of the speech partner and the emotion of the speech partner.
[0306] The embodiments of the present invention have been described above with reference to the drawings (FIGS. 1 to 24). However, the present invention is not limited to the above embodiments and can be implemented in various forms without departing from the spirit of the present invention. Furthermore, the components disclosed in the above embodiments can be modified as appropriate. For example, some of the components shown in one embodiment may be added to the components of another embodiment, or some of the components shown in one embodiment may be deleted from the embodiment.
[0307] The drawings mainly show each component in a schematic manner to facilitate understanding of the invention, and the thickness, length, number, spacing, etc. of each component shown in the drawings may differ from the actual ones due to the convenience of creating the drawings. Furthermore, the configuration of each component shown in the above embodiment is merely an example and is not particularly limited, and it goes without saying that various modifications are possible within a range that does not substantially deviate from the effects of the present invention.
[0308] The present invention is useful for a system for controlling an operation object that can be autonomously controlled or voluntarily controlled.
[0309] 2: Operation target 2A: Robot arm 2B: Two-dimensional avatar 3: Detection unit 3A: Detection unit 3B: Detection unit 4a: Second memory unit 5: Intention detection unit 5A: Intention detection unit 5B: Intention detection unit 43: Voluntary movement determination unit 43A: Voluntary movement determination unit 43B: Voluntary movement determination unit 44: Autonomous movement determination unit 44A: Autonomous movement determination unit 44B: Autonomous movement determination unit 45: Weight determination unit 45A: Weight determination unit 45B: Weight determination unit 46: Instruction unit 46A: Instruction unit 46B: Instruction unit 81: Second memory unit 100: Control system 100A: Robot control system 100B: Communication system D13: Weight change condition D24: Fifth machine learning model ST: Object
Claims
1. A control system for controlling an object to be operated, comprising: a detection unit that detects a detection target related to autonomous control of the object to be operated; a first decision unit that decides a next action of the object to be operated by voluntary control based on brain waves of an operator operating the object to be operated; a second decision unit that decides a next action of the object to be operated by autonomous control based on a result of detection by the detection unit; and a weight decision unit that decides a first weight for the next action of the object to be operated by voluntary control and a second weight for the next action of the object to be operated by autonomous control based on a result of detection by the detection unit, wherein the weight decision unit changes the first weight and the second weight in accordance with the result of detection by the detection unit.
2. A control system as described in claim 1, further comprising an intention detection unit that detects the intention of the operator based on the brain waves of the operator, and the weight determination unit determines the first weight and the second weight based on the detection result by the detection unit and the intention of the operator.
3. A control system as described in claim 1 or claim 2, further comprising an instruction unit that instructs the object to take a next action based on the next action determined by the first determination unit, the next action determined by the second determination unit, the first weight, and the second weight.
4. A control system as described in claim 3, further comprising a memory unit for storing change conditions for the first weight and the second weight, the change conditions being adjusted based on an evaluation function including, as parameters, control performance and operability obtained during training of the control system, the control performance indicating a value reflecting the amount of deviation between the intended behavior of an operator operating the operation object during the training and the actual behavior of the operation object, and the operability indicating a value reflecting the ease of operation of the operation object evaluated by the operator.
5. A control system as described in claim 3, wherein the instruction unit instructs the next action of the object to be operated based on the next action determined by the first determination unit at an initial stage of the action of the object to be operated.
6. A control system as described in claim 3, wherein the instruction unit instructs the next action of the object to be operated based on the next action determined by the second determination unit at a final stage of the action of the object to be operated.
7. The control system according to claim 6, wherein the autonomous control controls the object of operation more precisely than the voluntary control.
8. The control system according to claim 3, wherein the detection unit detects an object, and the weight determination unit dynamically changes the first weight and the second weight according to a distance between the target and the object.
9. A control system as described in claim 1 or claim 2, further comprising an instruction unit that instructs the operation target to take a next action of the operation target based on the next action determined by the first decision unit and the next action determined by the second decision unit, wherein the second decision unit determines the next action of the operation target by autonomous control based on the detection result by the detection unit, the first weight, and the second weight.
10. A control system as described in claim 9, further comprising a memory unit for storing change conditions for the first weight and the second weight, the change conditions being adjusted based on an evaluation function including, as parameters, control performance and operability obtained during training of the control system, the control performance indicating a value reflecting the amount of deviation between the behavior predicted by an operator who will operate the operation object during the training and the behavior of the operation object determined by the second determination unit, and the operability indicating a value reflecting the ease of operation of the operation object evaluated by the operator.
11. The control system of claim 9, wherein the behavior of the object to be operated is an utterance by the object to be operated, the detection unit detects an utterance by a speech partner to which the object to be operated is to be spoken, the second determination unit determines at least one candidate for the content of the next utterance of the object to be operated based on a detection result by the detection unit, the first weight, and the second weight, the candidate for the content of the next utterance determined by the second determination unit is capable of incorporating utterance content by voluntary control, the first determination unit determines the content of the next utterance based on the electroencephalogram, and the instruction unit instructs the object to the content of the utterance by the object to be operated.
Citation Information
Patent Citations
Machine and mechanical system
JP2010237859A
Information processing device and program
JP2022059140A
