Control model generating device, robot control device, control system, control model generating method and program

Abstract

A control model generation unit (2) according to the present disclosure comprising: a data storage unit (23) that stores personal behavior data which is data pertaining to an action of a person who has performed a cooperative action; and a learning unit (22) that generates, using the personal behavior data stored in the data storage unit (23), a control model for a humanoid, which is for causing the humanoid to cooperatively act with a user (5) and in which the person's personality is reflected. The personal behavior data includes data which is acquired when the user (5) is performing a cooperative action with a cooperative action subject (6) with whom the user (5) is familiar.

Description

[Technical field]

[0001] The present disclosure relates to a control model generating device, a robot control device, a control system, a control model generating method, and a program. [Background technology]

[0002] In recent years, the development of devices that can interact with humans, such as service robots and smart speakers, has progressed. In addition, in virtual spaces on computers, such as the metaverse, it is now possible for virtual characters to interact with users. An example of a device or virtual character that can interact with humans is a humanoid. Humanoids are required to act and speak in a way that suits each individual so as not to cause stress when living together with humans. However, generally, the same device (or virtual character) behaves and speaks in the same way regardless of the user, that is, the person with whom it interacts, and even the same device (or virtual character) can cause stress depending on the user.

[0003] Patent document 1 discloses a robot control device that controls the behavior of a robot so that the expected reaction to the robot's behavior matches the user's actual reaction to that behavior, enabling the robot to build an affinity with the user. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2013-027937 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology described in Patent Document 1, when the expected reaction of the robot's behavior does not match the user's actual reaction to the behavior, the robot's behavior is controlled so that the expected reaction matches the user's actual reaction. In other words, after the robot actually operates, if the behavior is not appropriate, the robot's behavior is controlled after the fact. This results in an inappropriate behavior, which causes the user to feel stressed to some extent. In addition, a user may wish to reflect at least one of the individual characteristics of a particular person in the behavior and speaking style of the humanoid.

[0006] The present disclosure has been made in consideration of the above, and aims to provide a control model generation device that is capable of reducing stress on a user caused by at least one of the behavior and speaking style of a humanoid. [Means for solving the problem]

[0007] In order to solve the above problems and achieve the object, the control model generation device according to the present disclosure includes a data storage unit that stores personal behavior data, which is data related to the behavior of a person who has performed a cooperative action, and a learning unit that uses the personal behavior data stored in the data storage unit to generate a control model of a humanoid for performing a cooperative action with a user, the control model reflecting the personality of the person. The personal behavior data includes data acquired when the user performs a cooperative action with a cooperative action target with whom the user is familiar. Effect of the Invention

[0008] The control model generation device according to the present disclosure has an effect of reducing stress on a user caused by at least one of the behavior and speech of a humanoid. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of a configuration of a collaborative operation system according to a first embodiment; [Diagram 2]FIG. 1 is a diagram showing an example of correction information according to the first embodiment; [Diagram 3] Schematic diagram showing an example of a neural network [Figure 4] A flowchart showing an example of a processing procedure in a control model generating unit according to the first embodiment. [Diagram 5] 1 is a flowchart showing an example of a processing procedure in a robot control unit according to the first embodiment. [Figure 6] FIG. 1 is a schematic diagram showing an example of acquiring personal behavior data in a first example; [Figure 7] FIG. 13 is a schematic diagram showing an example of a cooperative operation between a user and a robot in the first example. [Figure 8] FIG. 11 is a schematic diagram showing an example of acquiring personal behavior data in the second example; [Figure 9] FIG. 11 is a schematic diagram showing an example of a cooperative operation between a user and a robot in the second example. [Figure 10] FIG. 1 is a diagram showing an example of the configuration of a computer system that realizes a control system according to a first embodiment. [Figure 11] FIG. 13 is a diagram showing a configuration example of a cooperative operation system according to a second embodiment. [Figure 12] A flowchart showing an example of a processing procedure in a control model generating unit according to the second embodiment. [Figure 13] FIG. 13 is a diagram showing a configuration example of a cooperative operation system according to a third embodiment. [Figure 14] A flowchart showing an example of a control model update processing procedure in a control model generation unit according to a third embodiment. [Figure 15] FIG. 13 is a diagram showing a configuration example of a cooperative operation system according to a fourth embodiment. [Figure 16] FIG. 13 is a diagram showing a configuration example of a cooperative operation system according to a fifth embodiment. [Figure 17] FIG. 23 is a diagram showing a configuration example of a cooperative operation system according to a sixth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A control model generating device, a robot control device, a control system, a control model generating method, and a program according to embodiments will be described in detail below with reference to the accompanying drawings.

[0011] Embodiment 1 1 is a diagram showing an example of the configuration of a collaborative operation system according to the embodiment 1. The collaborative operation system 100 of the present embodiment includes a control system 1, a robot 7, a detection device 4, and a situation detection device 8.

[0012] The collaborative operation system 100 of this embodiment accumulates personal behavior data of the user 5 including collaborative operation data acquired when the robot 7 and the collaborative operation target 6 performed a collaborative operation before the collaborative operation between the robot 7 and the user 5 is performed, and generates a control model for controlling the robot 7 using the accumulated personal behavior data. The collaborative operation data is personal behavior data acquired when performing a collaborative operation. The acquired personal behavior data may be only the collaborative operation data, or may include other than the collaborative operation data.

[0013] The cooperative action includes, but is not limited to, at least one of, for example, conversation, work, games, and sports. The cooperative action target 6 is another person who performs the cooperative action with the user 5 and is familiar with the user 5 in the cooperative action. A person who is familiar with the user 5 in the cooperative action is, for example, a person who is not meeting the user for the first time and knows each other's personality or behavioral habits to some extent, and therefore, the user 5 feels that they are compatible in the cooperative action and is less likely to feel stressed when the user 5 performs the cooperative action together. The cooperative action target 6 may be determined, for example, by the user 5, or by an operator of the cooperative action system 100 other than the user 5. When the cooperative action target 6 is determined by an operator of the cooperative action system 100, for example, a person who is clearly performing a cooperative action with the user 5 for a long period of time is determined as the cooperative action target 6. The cooperative action target 6 may be one person or more than one person. When there are more than one cooperative action target 6, individual behavior data including cooperative action data when the user 5 performs a cooperative action with each of the cooperative action targets 6 is acquired. For example, when the collaborative action participants 6 include a first collaborative action participant and a second collaborative action participant, the personal behavior data of user 5 includes collaborative action data acquired during collaborative action between the first collaborative action participant and user 5, and collaborative action data acquired during collaborative action between the second collaborative action participant and user 5.

[0014] The personal behavior data is data on the behavior of a person who has performed a cooperative action. In this embodiment, the personal behavior data of the user 5 is acquired, and therefore the personal behavior data is data reflecting the individuality of the user 5. The personal behavior data of the user 5 includes data acquired when the user 5 is performing a cooperative action with the cooperative action target 6 with which the user 5 is familiar, and therefore it can be said that the personal behavior data of the user 5 is data indirectly reflecting the individuality of the cooperative action target 6. In addition, the control model is generated using personal data including data of the user 5 acquired when the user 5 is performing a cooperative action with the cooperative action target 6 with which the user 5 is familiar, and therefore the individuality of the user 5 is reflected in the control model. Here, the individuality of the user 5 reflected in the control model is not simply the individuality of the user 5, but the individuality of the user 5 when performing a cooperative action with the cooperative action target 6 with which the user 5 is familiar, and therefore not only the individuality of the user 5 but also the individuality of the cooperative action target 6 with which the user 5 is familiar is indirectly reflected in the control model. The indirectly reflected control here refers to control that reflects the behavior or manner of speech of the collaborative action target person 6 when interacting with the user 5, and can be said to be control that includes elements that are characteristic of the behavior or manner of speech that constitute the individuality of the collaborative action target person 6. This allows the robot 7 to perform actions that do not cause stress to the user 5 when performing collaborative actions.

[0015] The individual behavior data is data including the behavioral habits of the user 5. Here, the behavior is assumed to be an object to be used for controlling the robot 7, and may include not only the movement but also at least one of the speech style of the user 5 and the position and posture of the user 5. That is, the behavior includes, for example, at least one of the movement style, speech style, and at least one of the position and posture of the user 5. The movement style may include not only a continuous movement but also at least one of the position and posture at a certain moment. The behavioral habits include at least one of the movement habits and speech habits. The movement habits are, for example, at least one of the movement styles such as the trajectory of movement and the speed of movement, gestures, etc., and the speech habits are, for example, at least one of the speaking speed, catchphrases, speaking style, intonation, dialect, etc., but are not limited to these. The movements, actions, speaking styles, habits, etc. of the user 5, the cooperative operation target person 6, and the robot 7 may be called behaviors.

[0016] The robot 7 is an example of a humanoid that operates in cooperation with the user 5. The humanoid may also be referred to as a controlled object. More specifically, in this embodiment, the robot 7 is an example of a machine that operates in cooperation with the user 5. The robot 7 may be humanoid, a machine that does not have any moving parts and only communicates with the user 5, an industrial machine equipped with a manipulator or the like, or something other than these, and there are no particular restrictions on the shape and function.

[0017] The control system 1 controls a robot 7. The control system 1 includes a control model generation unit 2 that generates a control model of the robot 7, and a robot control unit 3 that controls the robot 7 using the control model generated by the control model generation unit 2 and situation data detected by a situation detection device 8. The control model generation unit 2, which is the control model generation device, and the robot control unit 3, which is the robot control device, may be integrated together, or each may be provided separately.

[0018] The detection device 4 detects the behavior of the user 5 when the user 5 is performing a cooperative action with the cooperative action target person 6 as personal behavior data, and transmits the personal behavior data to the control model generation unit 2. The detection device 4 is a device that detects at least one of, for example, position, speed, acceleration, posture, voice, pulse, blood pressure, body temperature, emotion, and the like, and there may be a plurality of detection devices 4.

[0019] In addition, the detection device 4 may detect psychological or internal information, such as biometric information or emotions, of the user 5 when the user 5 is performing a cooperative action with the cooperative action target person 6, and include the detected information in the personal behavior data and transmit it to the control model generation unit 2.

[0020] The detection device 4 may be a wearable terminal that the user 5 can wear, or a portable terminal that the user 5 can carry. The detection device 4 may be a device installed so as to be able to detect the motion of the user 5, or may be a device that detects motion in a virtual space such as the metaverse. The detection device 4 may be a combination of these, or may be something else. When the detection device 4 is a wearable terminal or a portable terminal, the detection device 4 may be a terminal capable of detecting at least one of its own position, speed, and acceleration, a terminal capable of detecting its own rotation, a terminal equipped with a microphone capable of collecting and recording sound, or a combination of these. A GPS (Global Positioning System) receiver, an RFID (Radio Frequency IDentification) tag, or other devices may be used to detect the position of the wearable terminal. When a wearable terminal or a portable terminal is used as the detection device 4, it is possible to acquire personal behavior data of the user 5 on a daily basis, not limited to the position of the user 5. When recording the voice, sounds uttered by persons other than the user 5 may also be recorded, but the detection device 4, the control model generation unit 2, or a device not shown in Fig. 1 performs voice recognition processing to extract the voice of the user 5. The voice recognition processing may be any processing, but may also be, for example, a processing in which the voice of the user 5 is acquired in advance and the voice uttered by the user 5 is identified using the previously acquired voice.

[0021] When a device installed so as to be able to detect the movement of the user 5 is used as the detection device 4, for example, the device may be a photographing device such as a camera that photographs the place where the cooperative movement is performed, or may be a device such as a microphone that can collect and record sound at the place where the cooperative movement is performed, or may be a combination of these. When the detection device 4 is a photographing device, the control model generation unit 2 or a device not shown in FIG. 1 recognizes the user 5 in the video photographed by the photographing device and detects the position, speed, acceleration, movement trajectory, etc. of the user 5. The method of recognizing the user 5 can be a general image recognition method, such as using an image of the user 5 photographed in advance. In addition, the detection device 4 may recognize the user 5 and detect the position, speed, acceleration, movement trajectory, etc. of the user 5. A general method can also be used for detecting the position, speed, acceleration, movement trajectory, etc. of the user 5.

[0022] When the detection device 4 is a device that detects actions in a virtual space such as the metaverse, the device may be, for example, a computer system that manages the virtual space, a terminal device that the user 5 uses to make his or her avatar move in the virtual space, or a device that records images in the virtual space. When a device that records images in the virtual space is used as the detection device 4, the control model generation unit 2 or a device not shown in FIG. 1 recognizes the user 5 in the image captured by the imaging device, and detects the position, speed, acceleration, movement trajectory, etc. of the user 5.

[0023] Furthermore, when the cooperative action includes an action using an object, such as moving or processing the object, the detection device 4 may include a device that detects the force that the user 5 applies to the object.

[0024] The control model generation unit 2 includes a basic model storage unit 21, a learning unit 22, a data storage unit 23, a data acquisition unit 24, and a correction information storage unit 25. The basic model storage unit 21 stores a predetermined basic control model that serves as a reference for the control model for controlling the robot 7. The basic control model is a general control model that does not depend on the user 5, i.e., does not reflect the individuality of the user 5, and is a model that defines the basic operations of the robot 7.

[0025] The basic control model may be stored in the data storage unit 23 in advance by a vendor of the control system 1 or a vendor of the robot 7, or may be transmitted from another device, received by a communication unit not shown in FIG. 1, and stored in the data storage unit 23. For example, the user 5 may operate the control system 1, and the control system 1 may receive a basic control model from an external server or the like that provides a basic control model corresponding to the robot 7. The basic control model may be provided for each type of robot 7, or may be provided according to the type of cooperative motion performed by the robot 7. For example, a basic control model corresponding to the type of cooperative motion performed by the robot 7 may be stored in the basic model storage unit 21, and the user 5 may select a basic control model corresponding to the type of cooperative motion performed by the robot 7 together with the user 5. Alternatively, an external server or the like may provide a basic control model corresponding to the type of cooperative motion performed by the robot 7, and the user 5 may select and download a basic control model corresponding to the type of cooperative motion performed by the robot 7 together with the user 5, so that the basic control model is stored in the basic model storage unit 21.

[0026] The basic control model and the control model include, for example, one or more control parameters for controlling the robot 7. The control parameters include at least one of parameters for controlling the movement of the robot 7, such as the trajectory, speed, and acceleration of the robot 7, parameters for controlling the movement of each part of the robot 7, such as the hands and joints of the robot 7, and parameters for controlling the way the robot 7 speaks, such as the speaking speed of the robot 7 and the pitch (frequency) of the voice emitted by the robot 7.

[0027] The data acquisition unit 24 acquires the personal behavior data of the user 5 from the detection device 4 and stores the acquired personal behavior data in the data storage unit 23. As described above, in order to obtain the personal behavior data, processing such as image processing on the video acquired by the detection device 4 and voice recognition processing on the voice acquired by the detection device 4 may be performed. In this case, other devices (not shown) perform these processes, and the data acquisition unit 24 acquires the personal behavior data from the other devices. The data acquisition unit 24 may also perform extraction processing such as image processing on the video acquired by the detection device 4 and voice recognition processing on the voice acquired by the detection device 4. In this case, the data acquisition unit 24 may perform extraction processing on the data acquired from the detection device 4 and store the processed data in the data storage unit 23 as personal behavior data, or the data itself acquired from the detection device 4 may be stored in the data storage unit 23 as personal behavior data, and the learning unit 22 may perform extraction processing in the process of generating a control model described later. Note that, here, the data acquisition unit 24 acquires the personal behavior data by receiving the personal behavior data, but this is not limited thereto, and the personal behavior data may be recorded on a recording medium or the like. In this case, the data acquisition unit 24 acquires the personal behavior data by reading the personal data from the recording medium.

[0028] The correction information storage unit 25 stores correction information indicating correction contents for the basic control model according to the individual behavior data. The correction information is, for example, information in which one or more feature amounts indicated by the individual behavior data are associated with the correction contents. The feature amount indicates the personality of the person from whom the individual behavior data is acquired. The personality includes, for example, at least one of behavior, catchphrases, intonation, dialect, and habit of movement. The correction contents are determined so that the robot 7 performs an action that does not cause the user 5 to feel stress when performing a cooperative action with the robot 7 according to the personality of the user 5. The learning unit 22 uses the individual behavior data stored in the data storage unit 23, i.e., the accumulated individual behavior data, to generate a control model of the robot 7 for performing a cooperative action with the user 5, which reflects the personality of the user 5. The learning unit 22 generates a control model using, for example, the basic control model stored in the basic model storage unit 21, the individual behavior data accumulated in the data storage unit 23, and the correction information stored in the correction information storage unit 25.

[0029] FIG. 2 is a diagram showing an example of the correction information of the present embodiment. In the example shown in FIG. 2, the personality of the user 5 is classified into types based on N (N is an integer equal to or greater than 1) feature amounts, and the correction information includes correction contents of parameters (control parameters) in the control model for each type. The feature amount may be an angle indicating the direction of movement of the user 5 during cooperative movement from a reference direction, or a numerical value indicating the amount of movement from a reference point during cooperative movement of the user 5, or information obtained by frequency-converting time-series data of the position of the user 5 during cooperative movement, or the speaking speed of the user 5, or information obtained by frequency-converting the voice of the user 5. In addition, the feature amount may be whether or not the user 5 has performed a specific predetermined movement, or the number of times the user 5 has performed a specific predetermined movement per unit time, or whether or not the user 5 has uttered a specific word, or the number of times a specific word has been uttered by the user 5 per unit time, or the like. In addition, as a feature amount indicating the habits or personality of the user 5, for example, a behavior that tends to differ from one individual to another and in which a specific regularity is recognized may be extracted. The feature amount may be the individual behavior data itself, or the collaborative action data in the individual behavior data itself. The feature amount is not limited to the above example, and may be anything that indicates at least one of the features of the user 5's movement and speech, that is, the habits of the user 5. Note that, although FIG. 2 shows an example in which N is 3 or more, the present invention is not limited to this, and N may be 1 or more.

[0030] In Fig. 2, an example in which the correction information is defined in a table format has been described, but the format of the correction information is not limited to the example shown in Fig. 2. When the correction information shown in Fig. 2 is used, the learning unit 22 extracts features from the personal behavior data accumulated in the data storage unit 23, identifies a type corresponding to the extracted features using the correction information, extracts correction content corresponding to the identified type from the correction information, and corrects the basic control model based on the extracted correction content, thereby generating a control model.

[0031] The correction information may be determined manually in advance. For example, the correction information may be determined by a vendor or an administrator of the robot 7 or the control system 1, or may be determined by learning through machine learning (pre-learning).

[0032] In the former case, the vendor or administrator determines the correction information by inferring how the behavior of the robot 7 based on the basic control model should be modified for each type of user 5 in accordance with the content of the collaborative behavior so that the user 5 does not feel stressed.

[0033] In the latter case, for example, before the robot 7 starts to operate, the robot 7 is made to perform a cooperative operation with an arbitrary person, and for each cooperative operation, a feature amount extracted from the personal behavior data of the person who performed the cooperative operation and the correction content of each control parameter in the control model of the robot 7 (correction content from the basic control model) are acquired as a set of data sets. In addition, in each cooperative operation, an evaluation is performed to indicate whether or not the person who performed the cooperative operation with the robot 7 felt stress. The operation of the robot 7 and the person who performs the cooperative operation are appropriately changed to acquire a plurality of data sets with different conditions and corresponding evaluation results. Note that the person who performs the cooperative operation at this time may be any person, and may or may not include the user 5. In addition, the person who performs the cooperative operation at this time may not be the robot 7 itself that performs the cooperative operation with the user 5, but may be another robot of the same type as the robot 7, or may be a different type of robot that can perform the same operation as the robot 7.

[0034] When a plurality of data sets with different conditions and corresponding evaluation results that are corrective data are acquired, a trained model is generated by supervised learning using the correction contents of each control parameter in the data set in which the evaluation result that the user did not feel stress was obtained by pre-learning as corrective data. The pre-learning may be performed by the learning unit 22, a pre-learning unit (not shown) of the control model generating unit 2, or a learning device other than the control system 1. When pre-learning is performed, the correction information is a trained model for inferring the correction contents of the control parameters from the feature amount extracted from the personal behavior data, and the learning unit 22 can infer the correction contents of the control parameters suitable for the user 5 by inputting the feature amount extracted from the personal behavior data of the user 5 to the trained model. The learning unit 22 generates a control model by reflecting the inferred correction contents in the basic control model.

[0035] Any algorithm may be used for the supervised learning, but for example, a neural network model may be used. A neural network is composed of an input layer consisting of multiple neurons, an intermediate layer (hidden layer) consisting of multiple neurons, and an output layer consisting of multiple neurons. The intermediate layer may be one layer, or two or more layers.

[0036] Fig. 3 is a schematic diagram showing an example of a neural network. For example, in a three-layered neural network as shown in Fig. 3, when multiple inputs are input to the input layer (X1-X3), the values ​​are multiplied by weight W1 (w11-w16) and input to the intermediate layer (Y1-Y2), and the result is further multiplied by weight W2 (w21-w26) and output from the output layer (Z1-Z3). This output result changes depending on the value of weight W1 and the value of weight W2.

[0037] In this embodiment, the weights W1 and W2 are adjusted so that the output from the output layer when the feature extracted from the individual behavior data is input approaches the correction content of the control parameter, which is the correct data, to learn the relationship between the feature and the correct data. Note that the machine learning algorithm is not limited to a neural network, and may be other algorithms such as a support vector machine. Furthermore, the machine learning used to generate the trained model is not limited to supervised learning, and may be reinforcement learning, etc.

[0038] In addition, when pre-learning is performed, information in a table format may be used as the correction information. For example, after the above-mentioned trained model is generated, a plurality of input data are generated by changing the values ​​of each feature amount, and the control parameters obtained by inputting each input data into the trained model are inferred. Then, input data in which the correction contents of the control parameters obtained by inference are all the same or the difference is within a certain range may be defined as one type, and control parameters for each type may be determined, and correction information in a table format as exemplified in FIG. 2 may be generated.

[0039] Furthermore, even if supervised learning, reinforcement learning, or the like is not used, the motion of the robot 7 and the person who performs the cooperative motion may be appropriately changed to obtain a plurality of data sets under different conditions, and the correction information may be determined using the obtained data sets and the corresponding evaluation results. For example, the values ​​of the type classification thresholds X1, X2, etc. and the control parameters may be manually determined using the correction contents of each control parameter in a data set in which the evaluation result that no stress was felt is obtained and the feature amounts of the individual behavior data.

[0040] In addition, although an example using the correction information has been described here, depending on the content of the cooperative action, overall information indicating the total value of the index obtained by the action of the robot 7 and the index obtained by the action of the user 5 in the cooperative action may be defined instead of the correction information. For example, when the cooperative action is an action in which the robot 7 and the user 5 cooperate to pull an object with a certain force, if the pulling force of the user 5 is weak, the cooperative action will fail unless the robot 7 increases the pulling force, which will cause stress to the user 5. Also, if the pulling force of the user 5 is strong, the cooperative action will fail unless the robot 7 decreases the pulling force, which will cause stress to the user 5. In the case of such a cooperative action, the pulling force of the user 5 is acquired as individual behavior data, and the total force is defined as the overall information. Then, the learning unit 22 calculates the pulling force of the robot 7 by subtracting the pulling force of the user 5 from the total force, and calculates the correction amount of the control parameter according to the calculated force. The above-mentioned overall information is an example, and the overall information is not limited to the above-mentioned example.

[0041] Returning to the explanation of Fig. 1, the learning unit 22 outputs the generated control model to the robot control unit 3. The robot control unit 3 includes an instruction sending unit 31, a situation acquiring unit 32, a control instruction generating unit 33, and a control model storage unit 34. The robot control unit 3 is an example of a motion control unit (motion control device) that controls a humanoid.

[0042] The control model storage unit 34 stores the control model output from the learning unit 22. The situation acquisition unit 32 acquires situation data indicating the situation of the cooperative operation between the robot 7 and the user 5 acquired by the situation detection device 8 by receiving it from the situation detection device 8, and outputs the acquired situation data to the control instruction generation unit 33. The situation detection device 8 may be provided on the robot 7, may be provided around the robot 7, or may be provided on both the robot 7 and the periphery of the robot 7. The situation detection device 8 acquires situation data used for controlling the robot 7 according to the type of the robot 7 and the content of the cooperative operation. The situation detection device 8 may acquire the situation of the operation of the user 5, such as the voice uttered by the user 5 and the movement of the user 5. There may be a plurality of situation detection devices 8. The situation detection device 8 may be, for example, a photographing device that detects the position of the robot 7, the state of the surroundings of the robot 7, etc., or may be an acceleration sensor, a force sensor, or the like. Furthermore, the situation detection device 8 may be a photographing device for grasping the positional relationship between the target object and the robot 7 when the robot 7 moves the target object or applies a force to the target object. The situation detection device 8 may be two or more of these, or may be other than these, and any sensor generally used for controlling the robot 7 may be used. Note that situation data does not need to be used for controlling the robot 7, and in this case, the situation detection device 8 does not need to be provided.

[0043] The control instruction generation unit 33 generates control instructions for the robot 7 using the situation data received from the situation acquisition unit 32 and the control model stored in the control model storage unit 34, and outputs the generated control instructions to the instruction transmission unit 31. The instruction transmission unit 31 transmits the control instructions received from the control instruction generation unit 33 to the robot 7. The robot 7 that receives the control instructions operates based on the control instructions.

[0044] As described above, in this embodiment, a control model is generated by the learning unit 22 using personal behavior data when the user 5 is performing a cooperative action with the familiar cooperative action target person 6, and a control instruction based on the generated control model is transmitted to the robot 7. Since the control model is generated before the cooperative action between the user 5 and the robot 7, the robot 7 can perform actions that are the same as or similar to the actions of the cooperative action target person 6 from the start of the cooperative action, and stress on the user 5 caused by at least one of the actions and speech of the robot 7 during the cooperative action can be reduced.

[0045] Next, the operation of this embodiment will be described. Fig. 4 is a flowchart showing an example of a processing procedure in the control model generating unit 2 of this embodiment. The control model generating unit 2 acquires personal behavior data of the user 5 including cooperative action data acquired during cooperative action between the user 5 and the cooperative action target person 6 (step S1). In detail, the data acquiring unit 24 acquires the personal behavior data of the user 5 by receiving the personal behavior data from the detection device 4. As described above, the data acquiring unit 24 may acquire the personal behavior data by a recording medium. Also, a video or the like that is the source of the personal behavior data may be acquired by the detection device 4 and an extraction process may be performed.

[0046] The control model generating unit 2 stores the individual behavior data of the user 5 (step S2). In detail, the data acquiring unit 24 stores the received individual behavior data in the data storage unit 23.

[0047] The control model generating unit 2 generates a control model using the accumulated individual behavior data (step S3). In detail, the learning unit 22 extracts a feature amount using the individual behavior data of the user 5 stored in the data storage unit 23, and generates a control model using the feature amount and the basic control model stored in the basic model storage unit 21. The accumulated individual behavior data is individual behavior data acquired for each cooperative motion in one or more cooperative motions. If the feature amount is, for example, the speaking speed of the user 5, when individual behavior data corresponding to multiple cooperative motions are accumulated, the average speed per character may be obtained using all the individual behavior data corresponding to the multiple cooperative motions. If the feature amount indicates, for example, the tendency of the user 5 to move, the individual behavior data corresponding to the multiple cooperative motions may be used to calculate an averaged position at a specified time point in the cooperative motion, and the difference between the averaged position and a predetermined standard position may be used as the feature amount. The method of calculating the feature amount is not limited to the above-mentioned example.

[0048] The control model generation unit 2 outputs the control model (step S4). In detail, the learning unit 22 outputs the generated control model to the robot control unit 3. The control model storage unit 34 of the robot control unit 3 stores the control model output from the learning unit 22.

[0049] Fig. 5 is a flowchart showing an example of a processing procedure in the robot control unit 3 of this embodiment. The processing shown in Fig. 5 is performed when the robot 7 and the user 5 perform a cooperative operation after the control model generation unit 2 generates a control model.

[0050] The robot control unit 3 acquires situation data (step S11). In detail, the situation acquisition unit 32 acquires situation data indicating the situation of the robot 7 acquired by the situation detection device 8 by receiving the situation data from the situation detection device 8, and outputs the acquired situation data to the control instruction generation unit 33.

[0051] The robot control unit 3 generates a control instruction using the situation data and the control model (step S12). In detail, the control instruction generation unit 33 generates a control instruction for the robot 7 using the situation data received from the situation acquisition unit 32 and the control model stored in the control model storage unit 34, and outputs the generated control instruction to the instruction transmission unit 31.

[0052] The robot control unit 3 transmits a control instruction (step S13). In detail, the instruction transmission unit 31 transmits the control instruction received from the control instruction generation unit 33 to the robot 7. As a result, the robot 7 performs an operation based on the control instruction.

[0053] In addition, once a control model is generated and the robot 7 and the user 5 perform a cooperative action, when the user 5 and the cooperative action target person 6 perform a cooperative action, cooperative action data in the cooperative action may be acquired, and a control model may be generated based on personal behavior data including the acquired cooperative action data. In this case, the process shown in FIG. 5 is performed using the newly generated control model. In this way, once a control model is generated, the control model may be updated using new cooperative action data. Thereby, even if the personality of the user 5 changes, the robot 7 can be controlled by reflecting the latest state.

[0054] Next, an example of a cooperative operation performed using the cooperative operation system 100 of this embodiment will be described. First, as a first example of a cooperative operation, a case where the robot 7 is a serving robot in a restaurant will be described. In the first example, the cooperative operation is a serving operation. For example, A, B, C, and D all work at the same restaurant, and B sometimes serves food with A, sometimes serves food with C, and sometimes serves food with D. Assume that B can perform a good job when serving food with A, and also when serving food with C. On the other hand, assume that when B serves food with D, B cannot perform the serving operation comfortably and feels stressed. A and C are planning to retire, and after A and C retire, B is scheduled to perform the serving operation with the robot 7. In such a case, in preparation for a cooperative operation with the robot 7, individual behavior data including cooperative operation data when B, who is a user 5, performs a cooperative operation with a cooperative operation target person 6 is acquired. In this case, the cooperating participants 6 who are familiar with cooperating with B are A and C.

[0055] FIG. 6 is a schematic diagram showing an example of acquiring individual behavior data in the first example. In the example shown in FIG. 6, user 5 and cooperating action target 6 serve dishes placed on a serving counter 201 to tables 202 in the hall of a restaurant. As shown in FIG. 6, when user 5 (person B) is serving food together with cooperating action target 6 (person A or person C) in a restaurant, individual behavior data of person B is acquired by detection device 4. For example, as shown in FIG. 6, when user 5 is serving food together with cooperating action target 6, user 5 serves multiple small plates, and cooperating action target 6 serves large plates. On the other hand, when user 5 is serving food together with person D, person D serves multiple small plates, and person B serves large plates.

[0056] In the example shown in FIG. 6, the personal behavior data includes information indicating which dishes were served. For example, in the case where the positions of the small and large plates are generally determined in advance on the serving counter 201 where the food prepared in the kitchen is temporarily placed before serving, the detection device 4 may be used to detect the position of the user 5, and the time series data of the positions detected by the detection device 4 may be used as the personal behavior data. In this case, for example, the learning unit 22 may obtain a history of the movement of the user 5, and obtain the size of the plates served by the user 5 as a feature based on the obtained history and the positions of the small and large plates. Alternatively, the detection device 4 capable of photographing the serving counter 201 may be used, and the size and number of the plates served by the user 5 may be calculated as personal behavior data by the data acquisition unit 24 or another device by analyzing the image photographed by the detection device 4. Alternatively, the detection device 4 capable of photographing the serving counter 201 may be used, and the image photographed by the detection device 4 may be used as personal behavior data, and the learning unit 22 may obtain the size of the plates served by the user 5 as a feature from the image.

[0057] In the first example, it is assumed that the plates to be served in the serving task include large plates and small plates. For this reason, the feature value includes the size of the plates to be served, or the size and number of the plates to be served, and the type in the modification information is defined as a type in which multiple small plates are served, or a type in which a large plate is served. Then, as the modification content corresponding to the type in the modification information, a numerical value is set as a control parameter for setting the target of the robot 7 to be a large plate. For example, in the first example, the control model includes a serving judgment model and a movement model, and the serving judgment model includes a definition of the size of the plate to be served by the robot 7. Then, as the modification content corresponding to the above-mentioned type in the modification information, information is set for setting a control parameter for setting the target of the serving to be a plate with a diameter of a certain value or more. In this way, the learning unit 22 of the control model generation unit 2 can generate a control model for making the robot 7 serve large plates. In addition, when the positions at which small and large plates are placed on the serving counter 201 are roughly determined in advance, instead of specifying the size of the plate, the range at the serving counter 201 within which the plates to be served are located may be defined as a control parameter.

[0058] FIG. 7 is a schematic diagram showing an example of cooperative operation between the user 5 and the robot 7 in the first example. In the example shown in FIG. 7, as described with reference to FIG. 6, a control model for making the robot 7 serve food on a large plate is generated, so the robot 7 serves food on a large plate. This allows the user 5 to efficiently serve food with less stress, as in the case where the user 5 serves food together with the cooperative action target 6, person A or person C. In this way, in the first example, the control system 1 can learn a method of serving food that allows the user 5, person B, to act efficiently, and generate a control model that reflects the results of the learning.

[0059] Next, as a second example of the cooperative operation, a case where the robot 7 and the user 5 perform assembly work will be described. In the second example, the robot 7 is, for example, an assembly robot, which is a type of industrial machine. For example, A, B, C, and D are all workers who perform assembly work, and B sometimes performs assembly work with A, sometimes performs assembly work with C, and sometimes performs assembly work with D. Assume that B can perform the work well when performing assembly work with A, and also when performing assembly work with C. On the other hand, when B performs assembly work with D, B, the user 5, cannot perform the assembly work comfortably and feels stressed. A and C are scheduled to be transferred, and after A and C are transferred, B is scheduled to perform assembly work with the robot 7. In such a case, as in the first example, individual behavior data including cooperative action data when B, the user 5, performs cooperative action with the cooperative action target 6 is acquired in preparation for cooperative action with the robot 7. In this case, the cooperating participants 6 who are familiar with cooperating with B are A and C.

[0060] FIG. 8 is a schematic diagram showing an example of acquiring individual behavior data in the second example. In the example shown in FIG. 8, the user 5 and the collaborative action target person 6 cooperate to perform an assembly work. More specifically, the user 5 (Mr. B) places the first part 204, and the collaborative action target person 6 (Mr. A or Ms. C) places the second part 205 on the first part 204. The standard position 203 indicates a standard position where the first part 204 is placed, and the user 5 has a habit of placing the first part 204 to the right of the standard position in FIG. 8. The collaborative action target person 6, who is familiar with the user 5, places the second part 205 in accordance with the position where the user 5 places the first part 204, and therefore can perform the assembly work efficiently. On the other hand, Mr. D, who is not familiar with user 5, attempts to place the second part 205 on the assumption that the first part 204 will be placed in the standard position 203, which takes time for alignment or requires user 5 to change the position of the first part 204, making it difficult to perform the assembly work efficiently and causing user 5 to feel stressed.

[0061] In such a case, the detection device 4 detects the location where the user 5 places the first part 204, or the position of the hand of the user 5 when the user 5 places the first part 204. Then, using the difference from the standard position 203 of the first part 204 as a feature amount, a type is defined in the correction information that the placement position of the first part 204 is shifted from the standard position 203 by a threshold value or more. Then, as a correction content corresponding to the type in the correction information, a content for determining a control parameter to shift the position of the second part 205 placed by the robot 7 by the same amount as the difference between the placement position of the first part 204 and the standard position 203 is included. This allows the learning unit 22 of the control model generation unit 2 to generate a control model that causes the robot 7 to place the second part 205 according to the amount by which the user 5 shifts the first part 204 from the standard position 203.

[0062] Fig. 9 is a schematic diagram showing an example of cooperative operation between the user 5 and the robot 7 in the second example. In the example shown in Fig. 9, as described with reference to Fig. 8, a control model is generated in which the robot 7 places the second part 205 in accordance with the amount by which the user 5 has shifted the first part 204 from the standard position 203, so the robot 7 places the second part 205 by shifting it to the right. This allows the user 5 to reduce stress and efficiently perform assembly work, similar to when performing assembly work together with the cooperative operation target person 6, person A or person C.

[0063] In both the first and second examples, the robot 7 is able to work together with the user 5, taking into consideration the behavior of person B who is the user 5, and even in an environment where humans work together with the robot 7, which is an example of a humanoid for labor saving purposes, the user 5 can perform a cooperative operation in a state in which stress such as difficulty in working and discomfort is reduced. Note that the cooperative operation performed using the cooperative operation system 100 described above is merely an example, and the cooperative operation performed using the cooperative operation system 100 is not limited to the above example.

[0064] The cooperative action is not limited to two people, but may be performed by three or more people. For example, in the case of a cooperative action performed by three people, two robots 7 may be used, or the robot 7 and the cooperative action target person 6 may perform the cooperative action together with the user 5. In this case, for example, if user 5 B performs a cooperative action with users A and C, and B and A do not feel stressed, a control model is generated based on the personal behavior data of C, and the robot 7 is controlled based on this control model.

[0065] Next, the hardware configuration of each device of this embodiment will be described. In the control system 1 of this embodiment shown in FIG. 1, a computer program in which processing in the control system 1 is described is executed on the computer system, so that the computer system functions as the control system 1. FIG. 10 is a diagram showing an example of the configuration of a computer system that realizes the control system 1 of this embodiment. As shown in FIG. 10, this computer system includes a control unit 101, an input unit 102, a storage unit 103, a display unit 104, a communication unit 105, and an output unit 106, which are connected via a system bus 107. The control unit 101 and the storage unit 103 form a processing circuit.

[0066] In FIG. 10, the control unit 101 is, for example, a processor such as a CPU (Central Processing Unit), and executes a program in which the processing in the control system 1 of the present embodiment is described. Note that a part of the control unit 101 may be realized by dedicated hardware such as a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array). The input unit 102 may be an input means such as a button, a keyboard, a mouse, a joystick, a touchpad, or a game controller. The storage unit 103 includes various memories such as a RAM (Random Access Memory), a ROM (Read Only Memory), and a storage device such as a hard disk, and stores the program to be executed by the control unit 101, necessary data obtained in the process of processing, and the like. The storage unit 103 is also used as a temporary storage area for the program. As described above, the display unit 104 is, for example, a display. Note that the display unit 104 and the input unit 102 may be integrated and realized by a touch panel or the like. The communication unit 105 is a receiver and a transmitter that perform communication processing. The output unit 106 is, for example, a speaker. Note that Fig. 10 is just an example, and the configuration of the computer system is not limited to the example of Fig. 10. For example, in this embodiment, the computer system that realizes the control system 1 does not need to include the output unit 106.

[0067] Here, an example of the operation of the computer system until the program of this embodiment is in an executable state will be described. In the computer system having the above-mentioned configuration, for example, a computer program is installed in the storage unit 103 from a CD-ROM or DVD-ROM set in a CD (Compact Disc)-ROM drive or DVD (Digital Versatile Disc)-ROM drive (not shown). Then, when the program is executed, the program read from the storage unit 103 is stored in the main storage area of ​​the storage unit 103. In this state, the control unit 101 executes the process as the control system 1 of this embodiment according to the program stored in the storage unit 103.

[0068] In the above description, a program describing the processing in the control system 1 is provided on a CD-ROM or DVD-ROM as a recording medium. However, this is not limited to this, and depending on the configuration of the computer system and the capacity of the program to be provided, for example, a program provided via a transmission medium such as the Internet may be used.

[0069] The program of this embodiment causes a computer system to execute, for example, the steps of accumulating personal behavioral data, which is data regarding the behavior of a person who has performed collaborative actions, and using the accumulated personal behavioral data, generating a control model of a humanoid that enables a humanoid to perform collaborative actions with a user 5, the control model reflecting the personality of the person.

[0070] The learning unit 22 and the control instruction generating unit 33 shown in FIG. 1 are realized by executing a computer program stored in the storage unit 103 shown in FIG. 10 by the control unit 101 shown in FIG. 10. The learning unit 22 and the control instruction generating unit 33 shown in FIG. 1 are realized by the storage unit 103 shown in FIG. 10. The data acquiring unit 24, the instruction transmitting unit 31, and the situation acquiring unit 32 shown in FIG. 1 are realized by the communication unit 105 shown in FIG. 10. The data acquiring unit 24 may be realized by a device that reads a recording medium. The basic model storing unit 21, the data storing unit 23, the correction information storing unit 25, and the control model storing unit 34 shown in FIG. 1 are part of the storing unit 103 shown in FIG. 10.

[0071] The control system 1 shown in Fig. 1 may be realized by a plurality of computer systems. For example, the control system 1 may be realized by a cloud system. As described above, the control model generation unit 2 and the robot control unit 3 may be configured as separate devices, and in this case, each of the separate devices may be realized by a plurality of computer systems.

[0072] As described above, the collaborative operation system 100 of this embodiment generates a control model for controlling the robot 7 using personal behavior data of the user 5 including collaborative operation data acquired when the user 5 and the collaborative operation target person 6 performed a collaborative operation before the collaborative operation between the robot 7 and the user 5 is performed. The robot 7 can perform an operation that is the same as or similar to the operation of the collaborative operation target person 6 from the start of the collaborative operation, and can reduce stress on the user 5 caused by at least one of the behavior and speech of the robot 7 during the collaborative operation.

[0073] Embodiment 2 11 is a diagram showing a configuration example of a cooperative operation system according to the second embodiment. A cooperative operation system 100a according to the present embodiment is similar to the cooperative operation system 100 according to the first embodiment, except that the cooperative operation system 100a according to the present embodiment includes a control system 1a instead of the control system 1 and a detection device 4a instead of the detection device 4. Components having the same functions as those according to the first embodiment are given the same reference numerals as those according to the first embodiment, and duplicated explanations will be omitted. Below, differences from the first embodiment will be mainly explained.

[0074] In the first embodiment, a control model is generated using the personal behavior data of the user 5 including the cooperative action data acquired when the user 5 and the cooperative action target 6 are performing a cooperative action. In the present embodiment, a control model is generated using the personal behavior data of the cooperative action target 6 including the cooperative action data acquired when the user 5 and the cooperative action target 6 are performing a cooperative action. In the present embodiment, the personal behavior data, which is data on the actions of a person who has performed a cooperative action, is the personal behavior data of the cooperative action target 6. That is, the person from whom the personal behavior data is acquired is the user 5 in the first embodiment, and is the cooperative action target 6 in the present embodiment. The cooperative action target 6 is a person familiar to the user 5, and is a person with whom the user 5 is unlikely to feel stressed when performing a cooperative action together, as in the first embodiment.

[0075] The detection device 4a acquires personal behavior data of the collaborative action target person 6 and transmits the personal behavior data to the control system 1a. As in the first embodiment, the personal behavior data may include biological information or psychological or internal information such as emotions of the collaborative action target person 6. The detection device 4a is the same as the detection device 4 in the first embodiment, but the target from which the personal behavior data is acquired is the collaborative action target person 6. The detection device 4a may be a wearable terminal that can be worn by the collaborative action target person 6, a portable terminal that can be carried by the collaborative action target person 6, a device installed so as to be able to detect the behavior of the collaborative action target person 6, a device that detects behavior in a virtual space such as a metaverse, a combination of these, or other than these. That is, the personal behavior data includes, for example, at least one of data acquired by a wearable terminal and data in which the behavior of a person in a virtual space is recorded.

[0076] The control system 1a is similar to the control system 1 of the first embodiment, except that it includes a control model generation unit 2a instead of the control model generation unit 2. The control model generation unit 2a does not include a correction information storage unit 25, includes a learning unit 22a instead of the learning unit 22, and the data acquisition source of the data acquisition unit 24 is a detection device 4a instead of the detection device 4, but otherwise is similar to the control model generation unit 2 of the first embodiment. In this embodiment as well, the control model generation unit 2a and the robot control unit 3 may be provided as separate devices.

[0077] Next, the operation of the control model generating unit 2a of this embodiment will be described. FIG. 12 is a flowchart showing an example of a processing procedure in the control model generating unit 2a of this embodiment. The control model generating unit 2a acquires personal behavior data of the cooperating action target person 6 including cooperative action data acquired during cooperative action between the user 5 and the cooperating action target person 6 (step S21). In detail, the data acquiring unit 24 acquires the personal behavior data of the cooperating action target person 6 by receiving the personal behavior data from the detection device 4a. Note that, as in the first embodiment, the data acquiring unit 24 may acquire the personal behavior data by a recording medium. Also, a video or the like that is the source of the personal behavior data may be acquired by the detection device 4a and an extraction process may be performed.

[0078] The control model generating unit 2a stores the individual behavior data of the cooperating action target individual 6 (step S22). In detail, the data acquiring unit 24 stores the received individual behavior data in the data storage unit 23.

[0079] The control model generating unit 2a generates a control model using the accumulated individual behavior data of the cooperating action target person 6 (step S23). In detail, the learning unit 22a extracts a feature amount using the individual behavior data of the cooperating action target person 6 stored in the data storage unit 23, and generates a control model based on the feature amount.

[0080] In this embodiment, the control parameters in the control model are set so that the robot 7 performs the action indicated by the feature amount. This generates a control model for making the robot 7 perform the same action as the action reflecting the individuality of the cooperative action target person 6. The feature amount can be the same as that in the first embodiment, but in this embodiment, information on the dialect, speaking habits (including catchphrases), and the content of the topic (genre of frequent conversation) may be used as the feature amount. The information on the dialect includes, for example, information on the presence or absence of a dialect, and if there is a dialect, what kind (what region) of dialect it is. The dialect may be identified, for example, by storing a dialect dictionary for each dialect type in advance and using the dictionary, or by other methods. The speaking habits include, for example, the frequency of using a specific word at the end of a sentence, the frequent utterance of a specific word, the way of adding intonation such as a high pitch at the end of a sentence, the pitch of the voice, the speed of conversation, and the like, but may be other than these. The learning unit 22a extracts these speaking habits by, for example, performing a voice recognition process on the voice data obtained as the individual behavior data of the collaborative action target individual 6.

[0081] Step S24 after step S23 is the same as in the first embodiment, and the learning unit 22a outputs the generated control model to the robot control unit 3. The output control model is stored in the control model storage unit 34 of the robot control unit 3. The operation of the robot control unit 3 is the same as in the first embodiment. In this embodiment, a control model is generated based on the individual behavior data of the collaborative operation target 6 so as to perform an operation that reflects the individuality of the collaborative operation target 6. This allows the robot 7 to perform an operation that reflects the individuality of the collaborative operation target 6 that the user 5 is familiar with in the collaborative operation with the user 5, and can reduce stress on the user 5 caused by at least one of the behavior and speech of the robot 7 in the collaborative operation.

[0082] Next, an example of a cooperative operation performed using the cooperative operation system 100a of this embodiment will be described. As an example, an example will be described in which the robot 7 is a communication robot and the cooperative operation is a conversation. Mr. A and Mr. B are a married couple, and Mr. B is accustomed to talking with Mr. A and does not feel stressed when talking with Mr. A. Mr. A plans to work alone overseas, and during Mr. A's work alone, Mr. B plans to talk with the robot 7. In this case, the user 5 of the robot 7 is Mr. B, Mr. A is set as the cooperative operation target person 6, and personal behavior data of Mr. A is acquired. The control model generation unit 2a generates a control model based on the accumulated personal behavior data of Mr. A. For example, the basic control model includes a conversation model and a voice model, and the conversation model and the voice model are modified so as to have characteristics similar to those of Mr. A based on the personal behavior data. As a result, a control model is generated that reflects, for example, Mr. A's catchphrase, intonation, dialect, way of answering, and topic of conversation.

[0083] When user 5, B, converses with robot 7 as a cooperative operation while A is away from home, robot 7 is controlled using a control model based on the personal behavior data of A described above. This allows robot 7 to converse in a way that reflects A's personality, thereby reducing stress for user 5, B. Note that the cooperative operation performed using cooperative operation system 100a described above is merely an example, and cooperative operation performed using cooperative operation system 100a is not limited to the above example.

[0084] The control system 1a of this embodiment is realized by a computer system, similar to the control system 1 of the first embodiment. The control system 1a of this embodiment may also be realized by a plurality of computer systems, for example, by a cloud system. As described above, the control model generating unit 2a and the robot control unit 3 may each be configured as separate devices, and in this case, each separate device may also be realized by a plurality of computer systems.

[0085] In addition, the cooperative action is not limited to two people, but may be performed by three or more people. For example, in the case of a cooperative action performed by three people, two robots 7 may be used, or the robot 7 and the cooperative action target person 6 may perform a cooperative action together with the user 5. In this case, for example, if user 5 B feels no stress when performing a cooperative action with A and C, a control model is generated based on the personal behavior data of A and C, and the two robots 7 are controlled based on the respective control models. In addition, in the case where the robot 7 and the cooperative action target person 6 perform a cooperative action together with the user 5, when the cooperative action target person 6 performing the cooperative action is A, the robot 7 may be controlled based on the control model corresponding to C, and when the cooperative action target person 6 performing the cooperative action is C, the robot 7 may be controlled based on the control model corresponding to A.

[0086] As described above, the collaborative operation system 100a of this embodiment generates a control model for controlling the robot 7 using personal behavior data of the collaborative operation target person 6 including collaborative operation data acquired when the user 5 and the collaborative operation target person 6 performed a collaborative operation before the collaborative operation between the robot 7 and the user 5 is performed. The robot 7 can perform actions that are the same as or similar to the actions of the collaborative operation target person 6 from the start of the collaborative operation, and can reduce stress on the user 5 caused by at least one of the actions and speech of the robot 7 during the collaborative operation.

[0087] Embodiment 3 13 is a diagram showing a configuration example of a cooperative operation system according to the third embodiment. A cooperative operation system 100b according to the present embodiment is similar to the cooperative operation system 100a according to the second embodiment, except that the cooperative operation system 100b includes a control system 1b instead of the control system 1a. Components having the same functions as those in the second embodiment are given the same reference numerals as those in the second embodiment, and duplicated explanations will be omitted. Below, differences from the second embodiment will be mainly explained.

[0088] The control system 1b is similar to the control system 1a of the second embodiment, except that it includes a control model generation unit 2b instead of the control model generation unit 2a. The control model generation unit 2b additionally includes a behavior result acquisition unit 26 and includes a learning unit 22b instead of the learning unit 22a, but otherwise is similar to the control model generation unit 2a of the second embodiment. In this embodiment as well, the control model generation unit 2b and the robot control unit 3 may be provided as separate devices.

[0089] Next, the operation of the control model generating unit 2b of this embodiment will be described. The generation process of the control model in the control model generating unit 2b is the same as the process in the embodiment 2 described with reference to FIG. 12. In this embodiment, the control model generating unit 2b updates the control model based on the behavior result, which is the result of the cooperative action performed by the robot 7 and the user 5, and the control model corresponding to the behavior result, after the control model is generated. As described in the embodiment 2, the control model is generated so as to reduce the stress of the user 5, but in this embodiment, the behavior of the robot 7 can be made more suitable for the user 5 by updating the control model using the behavior result.

[0090] 14 is a flowchart showing an example of a control model update processing procedure in the control model generation unit 2b of this embodiment. First, the control model generation unit 2b acquires a behavior result corresponding to the control model (step S31). In detail, the behavior result acquisition unit 26 acquires a behavior result corresponding to a cooperative action (cooperative action between the robot 7 and the user 5) performed by control based on the control model stored in the robot control unit 3, and outputs the acquired behavior result to the learning unit 22b.

[0091] The behavior result indicates, for example, whether a positive result or a negative result has been obtained. The behavior result is determined, for example, by the user 5 and input to the control model generation unit 2b. In this case, the behavior result acquisition unit 26 has a function of receiving input from the user 5. Alternatively, the user 5 may input the behavior result to another device such as a user terminal (not shown), and the other device may transmit the behavior result to the control model generation unit 2b. In this case, the behavior result acquisition unit 26 has a communication function of receiving the behavior result. For example, when the user 5 feels that the cooperative operation with the robot 7 was performed comfortably, stress-free, or efficient, the behavior result is determined as a positive result, and when the user feels that the cooperative operation with the robot 7 was stressful, unpleasant, or inefficient, the behavior result is determined as a negative result.

[0092] Also, for example, when the cooperative action is a task or the like, the behavior result may be determined by another means. For example, when the cooperative action is a task with a set procedure, the task time of the task may be measured, and if the measurement result is equal to or less than a threshold, a person other than the user 5 may determine that the task was performed efficiently and set the behavior result as a positive result, and if the measurement result exceeds the threshold, the person may determine that the task was not performed efficiently and set the behavior result as a negative result. In this case, the behavior result may be input to the control model generating unit 2b or may be transmitted from another device. If the task is performed efficiently, it can be estimated that the stress of the user 5 is low, so that the behavior result may be determined based on the measurement result of the task time in this way. Also, the control model generating unit 2b may make a judgment based on the above-mentioned measurement result. For example, the behavior result acquiring unit 26 may receive the measurement result from a device that measures the task time, and determine the behavior result using the received measurement result. The method of determining the behavior result is not limited to the above-mentioned example.

[0093] The control model generating unit 2b judges whether or not the behavior result is a negative result (step S32). In detail, the learning unit 22b judges whether or not the behavior result received from the behavior result acquiring unit 26 is a negative result.

[0094] If the behavior result is not a negative result (No in step S32), that is, if the behavior result is a positive result, the control model generation unit 2b ends the control model update process.

[0095] If the action result is a negative result (Yes in step S32), the control model generation unit 2b updates the control model (step S33) and repeats the process from step S31. In step S33, in detail, the learning unit 22b updates the control model and outputs the updated control model to the robot control unit 3. This updates the control model stored in the control model storage unit 34 of the robot control unit 3. The learning unit 22b updates the control model, for example, by changing a part of the control parameters in the control model. The method of changing the control parameters may be determined in advance or may be specified by the user 5. For example, when updating a control parameter that changes the position of the robot 7, a rule for changing the position of the robot 7 may be determined in advance, or the user 5 may specify the direction and amount of change regarding the position of the robot 7.

[0096] As described above, if the behavior result is a negative result, the control model is updated, control is performed using the updated control model, and the process is repeated from step S31. If the behavior result is a negative result, the change of the control parameters is repeated until a positive result is obtained as the behavior result.

[0097] In the above example, when the behavior result is a positive result, the control model is not changed and the current control model is used as the updated control model. However, the present invention is not limited to this, and the control model may be updated by changing the current control parameter to a control parameter estimated to be better. The control parameter estimated to be better is, for example, a control parameter changed in the opposite direction to the control parameter set when the behavior result was previously a negative result. For example, when the conversation speed is a first speed, if the behavior result is a negative result and changing the conversation speed to a second speed slower than the first speed results in a positive behavior result, the control model may be updated to change the conversation speed to a third speed slower than the second speed. Then, the behavior result is acquired again, and if the behavior result is a negative result, the control model is updated to return the conversation speed to the second speed.

[0098] Alternatively, the control model update process is not limited to the procedure shown in FIG. 14, and the control parameters may be changed sequentially to obtain behavior results corresponding to the values ​​of the control parameters, a data set of the control parameter values ​​and the corresponding behavior results may be stored, and the control model may be updated using the multiple data sets. For example, data sets in which the behavior results are positive may be extracted, one of the extracted data sets may be selected, and the control model may be updated using the control parameters in the selected data set. In addition, the control model may be updated by determining control parameters that improve the behavior results by machine learning using a set of data sets that includes the behavior results and the corresponding control parameters. For example, the learning unit 22b generates a learned model by the supervised learning described in the first embodiment using multiple data sets that are composed of the behavior results and the control parameters that are the corresponding correct answer data. Then, at the time of inference, i.e., at the time of updating the control model, the learning unit 22b can infer control parameters that will result in a positive behavior result by inputting a value that will result in a positive behavior result as the behavior result.

[0099] Moreover, the behavioral result is not limited to the binary values ​​of positive and negative, and may be expressed by three or more levels of numerical values. For example, the behavioral result may be a score from 0 to 5, and it may be defined that when the behavioral result is 5, the user 5 is least stressed, and when the behavioral result is 0, the user 5 is most stressed. The definition of the score is not limited to this example. When expressed by three or more levels of numerical values, in step S32 in the process shown in FIG. 14, the learning unit 22b may determine whether the behavioral result is a numerical value that indicates the most positive. Furthermore, when updating the control model using the above-mentioned multiple data sets, the learning unit 22b may select a data set that indicates the most positive behavioral result.

[0100] In the above example, the behavior result is a result of evaluating the entire control model, but it is not limited thereto, and may be a result of evaluating the time series of the robot 7 in a section. For example, the control instructions to the robot 7 may be recorded in an operation history storage unit (not shown), and the behavior result may be determined, for example, at regular intervals or at each section of the robot 7's operation. In this case, the operation history storage unit may be provided in the robot control unit 3, in the control model generation unit 2b, or outside the control system 1b. In this case, the behavior result acquisition unit 26 reads and acquires the control instructions for the period corresponding to the behavior result from the operation history storage unit together with the behavior result, and outputs the behavior result and the corresponding control instructions to the learning unit 22b. As a result, the learning unit 22b can obtain the behavior result of the robot 7 in units of the operation of the robot 7 corresponding to the control instructions performed in time series.

[0101] For example, when the cooperative action is a conversation, it is assumed that the result that the cooperative action target person 6 often provides topics about the first genre and the second genre is obtained as a feature of the cooperative action target person 6 based on the personal behavior data of the cooperative action target person 6, and a control model is generated based on this result. When the control model is generated, it is assumed that the frequency of providing topics of the first genre and the second genre is set to be approximately the same. In the conversation, in the time series, there is a period in which the conversation of the first genre is held and a period in which the conversation of the second genre is held. These periods are distinguished based on a control instruction to the robot 7, and the behavior result of each is acquired by the behavior result acquisition unit 26. For example, if the behavior result of the period of the conversation of the first genre is a positive result and the behavior result of the period of the conversation of the second genre is a negative result, the control model is updated so as to increase the frequency of providing topics of the first genre and decrease the frequency of providing topics of the second genre.

[0102] The control model update process is not limited to the above example, and may be any method in which the learning unit 22b updates the control model so as to become a control model more suitable for the user 5 based on the action result.

[0103] The control system 1b of this embodiment is realized by a computer system, similar to the control system 1a of the second embodiment. The control system 1b of this embodiment may also be realized by a plurality of computer systems, for example, by a cloud system. As described above, the control model generating unit 2b and the robot control unit 3 may each be configured as separate devices, and in this case, each separate device may also be realized by a plurality of computer systems.

[0104] As described above, the collaborative operation system 100b of this embodiment performs the operations described in the second embodiment, and updates the control model based on the behavior result that is the result of the collaborative operation between the robot 7 and the user 5. Therefore, it is possible to obtain the same effects as in the second embodiment, and to make the robot 7 perform an operation that is more suitable for the user 5.

[0105] In the above example, a control model update function is added to the collaborative operation system 100a of the second embodiment, but the present invention is not limited to this, and a control model update function may be added to the collaborative operation system 100 of the first embodiment. For example, the control model may be updated in the same manner as in the above example by adding an action result acquisition unit 26 to the control model generation unit 2 of the collaborative operation system 100 and providing the learning unit 22 with a control model update function similar to the learning unit 22b.

[0106] Embodiment 4 15 is a diagram showing a configuration example of a cooperative operation system according to the fourth embodiment. A cooperative operation system 100c according to this embodiment is similar to the cooperative operation system 100b according to the third embodiment, except that the cooperative operation system 100c includes a control system 1c instead of the control system 1b. Components having the same functions as those in the third embodiment are given the same reference numerals as those in the third embodiment, and duplicated explanations will be omitted. Below, differences from the third embodiment will be mainly explained.

[0107] The control system 1c is similar to the control system 1b of the third embodiment, except that it includes a control model generation unit 2c instead of the control model generation unit 2b. The control model generation unit 2c additionally includes a model selection receiving unit 27 and includes a basic model storage unit 21a instead of the basic model storage unit 21, but otherwise is similar to the control model generation unit 2b of the third embodiment. In this embodiment as well, the control model generation unit 2c and the robot control unit 3 may be provided as separate devices.

[0108] The basic model storage unit 21a stores a plurality of basic control models, i.e., a plurality of types of basic control models, in advance. These plurality of types of basic control models can be said to be typical patterns that characterize the cooperative action target person 6. For example, in the case of a basic control model that imitates a human, the typical patterns that characterize the cooperative action target person 6 are patterns that correspond to speech and actions (behavior) based on a typical human personality, such as impatient, easygoing, or meticulous. In other words, for example, if the cooperative action includes a conversation, three basic control models, a moderate model, a solid model, and a leading model, are stored in the basic model storage unit 21a. The moderate model, the solid model, and the leading model each differ in at least one of, for example, the content of the conversation, the speed of the conversation, the frequency of speaking, and the genre of the topic provided.

[0109] Furthermore, for example, in the case of a basic control model to be applied to industrial machinery, basic control models that behave (act) in a manner based on typical work content that requires consideration for people working together, such as collaborative design work using engineering tools, collaborative precision work such as medical procedures (surgery), or collaborative work for long periods of time such as installing large equipment, are stored in the basic model memory unit 21a.

[0110] In addition, the basic control model, i.e., the typical pattern, may be classified by typical features of other targets that perform cooperative behavior. In this way, the multiple control models correspond to different personalities (personalities of behavior). Note that the multiple basic control models are not limited to this example, and the number of basic control models is not limited to three.

[0111] The model selection receiving unit 27 receives from the user 5 a selection result indicating the basic control model selected by the user 5 from among a plurality of basic control models. For example, the model selection receiving unit 27 may receive an input of the selection result from the user 5. Alternatively, the user 5 may input the selection result to another device such as a user terminal (not shown), the other device may transmit the selection result to the control model generating unit 2c, and the model selection receiving unit 27 may receive the selection result, thereby receiving the selection result of the basic control model. The user 5 selects a basic control model from among a plurality of basic control models according to taste and compatibility. For example, the user 5 may select a basic control model that matches the cooperative motion target person 6 from among a plurality of basic control models. For example, three basic control models, a moderate model, a steady model, and a traction model, are stored in the basic model storage unit 21a, and if the cooperative motion target person 6 that the user 5 is familiar with has a moderate personality, the moderate model may be selected. Note that the control system 1c or an operator of the control system 1c may select a basic control model suitable for the cooperative motion target person 6, instead of the user 5 selecting the basic control model.

[0112] The model selection receiving unit 27 reads out a basic control model corresponding to the received selection result from the basic model storage unit 21a, and outputs the read basic control model to the learning unit 22b. The learning unit 22b generates a control model using the basic control model received from the model selection receiving unit 27, i.e., the basic control model indicated by the selection result, and the individual behavior data, in the same manner as in the third embodiment, and outputs the generated control model to the robot control unit 3. Also, the learning unit 22b updates the control model using the behavior result, in the same manner as in the third embodiment. The operation of this embodiment other than that described above is the same as that of the third embodiment.

[0113] The control system 1c of this embodiment is realized by a computer system, similar to the control system 1b of the third embodiment. The control system 1c of this embodiment may also be realized by a plurality of computer systems, for example, by a cloud system. As described above, the control model generating unit 2c and the robot control unit 3 may each be configured as separate devices, and in this case, each separate device may also be realized by a plurality of computer systems.

[0114] As described above, the collaborative operation system 100c of this embodiment generates a control model using a basic control model selected from a plurality of basic control models with different characteristics of operation and personal behavior data. Furthermore, the collaborative operation system 100c of this embodiment updates the control model based on the behavior result that is the result of the collaborative operation between the robot 7 and the user 5. Therefore, it is possible to obtain the same effect as in the third embodiment and to make the robot 7 perform an operation that is more suited to the taste and compatibility of the user 5.

[0115] In the above example, the collaborative operation system 100b of the third embodiment is added with a function of generating a control model using a basic control model selected from a plurality of basic control models. However, the present invention is not limited to this. A function of generating a control model using a basic control model selected from a plurality of basic control models may be added to the collaborative operation system 100 of the first embodiment. For example, the control model generating unit 2 of the collaborative operation system 100 may be added with a model selection receiving unit 27 and a basic model storage unit 21a instead of the basic model storage unit 21, so that a control model may be generated using a basic control model selected from a plurality of basic control models as in the above example. Also, a function of generating a control model using a basic control model selected from a plurality of basic control models may be added to the collaborative operation system 100a of the second embodiment. For example, the control model generating unit 2a of the collaborative operation system 100 may be added with a model selection receiving unit 27 and a basic model storage unit 21a instead of the basic model storage unit 21, so that a control model may be generated using a basic control model selected from a plurality of basic control models as in the above example.

[0116] Embodiment 5. FIG. 16 is a diagram showing a configuration example of a collaborative operation system according to a fifth embodiment. A collaborative operation system 100d according to the present embodiment includes a control system 1d, a detection device 4, and a situation detection device 8a. The control system 1d generates a virtual space such as a metaverse, and transmits virtual space information to a terminal device 94 for allowing a user 5 to perceive the virtual space. The terminal device 94 outputs an image in the virtual space to an image presentation device 95 based on the virtual space information received from the control system 1d, and outputs an audio in the virtual space to an audio presentation device 96. The user 5 performs a collaborative operation with a virtual character 902 via his / her own avatar 901 in the virtual space.

[0117] In the first to fourth embodiments, the robot 7 is taken as an example of a humanoid that cooperates with the user 5, but in the present embodiment, an example will be described in which the humanoid that cooperates with the user 5 is a virtual character 902 in a virtual space. In this embodiment, when the user 5 cooperates with the virtual character 902, the virtual character 902 is controlled using a control model generated based on personal behavior data of the user 5 acquired in advance when the user 5 cooperates with the cooperative action target person 6, as in the first embodiment. Components having the same functions as those in the first embodiment are given the same reference numerals as those in the first embodiment, and duplicated explanations will be omitted. Below, differences from the first embodiment will be mainly explained.

[0118] 16 shows an example in which the terminal device 94 transmits video and audio to the video presentation device 95 and audio presentation device 96 by wireless communication, but one or more of the video and audio may be transmitted by wired communication. The terminal device 94 may be included in the control system 1d, or the terminal device 94, the video presentation device 95, and the audio presentation device 96 may be included in the control system 1d.

[0119] In addition, in FIG. 16, the image presentation device 95 and the audio presentation device 96 are used as means for the user 5 to perceive the virtual space, but a means capable of perceiving one or more of the sense of force, smell, and taste may be used. The sense of force may include information perceived by the skin, such as temperature, in addition to stress. In addition, in FIG. 16, the image presentation device 95 and the audio presentation device 96 are used, but any of them may not be used. In addition, in FIG. 16, an example is shown in which VR goggles and a head-mounted display are used as the image presentation device 95 and headphones are used as the audio presentation device 96, but this is not limited thereto, for example, the image presentation device 95 may be a display or a monitor, and the audio presentation device 96 may be a speaker, and specific examples of the image presentation device 95 and the audio presentation device 96 are not limited to the example shown in FIG. In addition, two or more of the terminal device 94, the image presentation device 95, and the audio presentation device 96 may be integrated. For example, the display of the terminal device 94 may be used as the image presentation device 95. Also, for example, a head mounted display having the functions of both the terminal device 94 and the image presentation device 95 may be used, or a head mounted display with headphones may be used.

[0120] The situation detection device 8a acquires the situation of the user 5 in the cooperative action between the user 5 and the virtual character 902. For example, the situation detection device 8a detects the voice, movement, etc. of the user 5 and transmits the detection result to the terminal device 94. The terminal device 94 transmits the detection result received from the situation detection device 8a to the control system 1d. There may be a plurality of situation detection devices 8a. Also, for example, the voice presentation device 96 may be integrated with the situation detection device 8a by using a headset as the voice presentation device 96. Also, the terminal device 94 may include the situation detection device 8a. Also, the situation detection device 8a may be worn by the user 5, or may be provided in the vicinity of the user 5, such as a photographing device that photographs the user 5 from the outside.

[0121] The control system 1d includes a control model generation unit 2 and a virtual space control unit 9 similar to those in the first embodiment. The control model generation unit 2 and the virtual space control unit 9 may be provided as separate devices. The configuration and operation of the control model generation unit 2 are similar to those in the first embodiment, but the control model generated by the control model generation unit 2 is a control model for controlling the movement of the virtual character 902, and the basic control model stored in the basic model storage unit 21 is also a basic control model for controlling the movement of the virtual character 902.

[0122] The virtual space control unit 9 includes a transmission / reception unit 91, a virtual space generation unit 92, and a virtual character control unit 93. The transmission / reception unit 91 communicates with a terminal device 94 and exchanges information with the terminal device 94. The transmission / reception unit 91 acquires, for example, situation data indicating the situation of the user 5 in the collaborative action from the terminal device 94, and outputs the acquired situation data to the virtual space generation unit 92 and the control instruction generation unit 33. The transmission / reception unit 91 may receive the situation data from the situation detection device 8a. The transmission / reception unit 91 also transmits, for example, virtual space information received from the virtual space generation unit 92, which will be described later, to the terminal device 94.

[0123] The virtual space generating unit 92 generates a virtual space, generates virtual space information for allowing the user 5 to perceive the generated virtual space, and outputs the generated virtual space information to the transmitting / receiving unit 91. The virtual space information includes data indicating an image (image data) and data indicating an audio (audio data). Depending on the content of the collaborative operation and the virtual space, the virtual space information may not include data indicating an audio. The virtual space information may also include at least one of information detectable by the user 5 using a force-tactile sense, a sense of smell, and a sense of taste. The virtual space generating unit 92 generates virtual space information so that the avatar 901 of the user 5 in the virtual space performs an action based on the situation data received from the transmitting / receiving unit 91. When the virtual space generating unit 92 receives a control instruction from the virtual character control unit 93, which will be described later, it generates virtual space information so that the virtual character 902 performs an action based on the control instruction.

[0124] The virtual character control unit 93 is an example of a motion control unit (motion control device) that controls a humanoid. The virtual character control unit 93 includes a control instruction generation unit 33 and a control model storage unit 34. The control model storage unit 34 stores the control model generated by the control model generation unit 2, as in the first embodiment. As described above, this control model is a control model for controlling the motion of the virtual character 902. The control instruction generation unit 33 generates a control instruction for controlling the motion of the virtual character 902 using the situation data indicating the situation of the user 5 received from the transmission / reception unit 91 and the control model stored in the control model storage unit 34, and outputs the generated control instruction to the virtual space generation unit 92.

[0125] In this embodiment, the control target is a virtual character 902 instead of a robot 7, but similarly to the first embodiment, before the virtual character 902 and the user 5 perform a cooperative action, a control model is generated using personal behavior data of the user 5 including cooperative action data acquired when the user 5 and the cooperative action target 6 perform a cooperative action. Therefore, the virtual character 902 can perform actions that are the same as or similar to the actions of the cooperative action target 6 from the start of the cooperative action, and stress on the user 5 caused by at least one of the actions and speech of the virtual character 902 during the cooperative action can be reduced.

[0126] In the example shown in FIG. 16, the virtual character control unit 93 is provided in the virtual space control unit 9, but the present invention is not limited to this. For example, the virtual space control unit 9 may be provided as a separate virtual space control device outside the control system 1d. In this case, the transmission / reception unit 91 and the virtual character control unit 93 are provided in the control system 1d, and the virtual space generation unit 92 is provided in the virtual space control device. The virtual space control device also includes a transmission / reception unit 91, and a control instruction generated by the virtual character control unit 93 is transmitted to the virtual space control device via the transmission / reception unit 91 of the control system 1d, and the virtual space generation unit 92 of the virtual space control device receives the control instruction via the transmission / reception unit 91 of the virtual space control device. The virtual space generation unit 92 of the virtual space control device transmits the generated virtual space information to the terminal device 94 via the transmission / reception unit 91 of the virtual space control device. The transmission / reception unit 91 may also be provided in the virtual character control unit 93. In this case, the virtual character control unit 93 and the control model generation unit 2 may also be provided as separate devices.

[0127] The control system 1d of this embodiment is realized by a computer system, similar to the control system 1 of the first embodiment. The control system 1d of this embodiment may also be realized by a plurality of computer systems, for example, by a cloud system. As described above, the control model generating unit 2 and the virtual space control unit 9 may each be configured as separate devices, and in this case, each separate device may also be realized by a plurality of computer systems.

[0128] In the above-mentioned example, the control model generating unit 2 of the first embodiment generates a control model for controlling the virtual character 902, and the virtual character control unit 93 controls the virtual character 902 using the generated control model. However, when controlling the virtual character 902, the action result acquiring unit 26 may be provided and the control model may be updated using the action result as in the third embodiment, or the basic control model to be used may be selected from a plurality of basic control models for controlling the virtual character 902 using the model selection receiving unit 27 as in the fourth embodiment. In addition, both the update of the control model using the action result and the selection of the basic control model to be used from a plurality of basic control models may be performed.

[0129] Embodiment 6 FIG. 17 is a diagram showing a configuration example of a collaborative operation system according to the sixth embodiment. The collaborative operation system 100e according to the present embodiment includes a control system 1e, a detection device 4a, and a situation detection device 8a. The control system 1e generates a virtual space and transmits virtual space information to a terminal device 94 for allowing a user 5 to perceive the virtual space, as in the fifth embodiment. The situation detection device 8a, the terminal device 94, the video presentation device 95, and the audio presentation device 96 are the same as those in the fifth embodiment. The terminal device 94 may be included in the control system 1e, and the terminal device 94, the video presentation device 95, and the audio presentation device 96 may be included in the control system 1e.

[0130] The control system 1e includes a control model generating unit 2a similar to that of the second embodiment and a virtual space control unit 9 similar to that of the fifth embodiment. The control model generating unit 2a and the virtual space control unit 9 may be provided as separate devices. The configuration and operation of the control model generating unit 2a are similar to those of the second embodiment, but the control model generated by the control model generating unit 2a is a control model for controlling the movement of the virtual character 902, and the basic control model stored in the basic model storage unit 21 is also a basic control model for controlling the movement of the virtual character 902. Components having the same functions as those of the second or fifth embodiment are given the same reference numerals as those of the second or fifth embodiment, and duplicated explanations are omitted. Below, differences from the second or fifth embodiment will be mainly explained.

[0131] In this embodiment, similarly to the second embodiment, the control model generation unit 2a generates a control model based on the personal behavior data of the cooperative action target person 6 acquired by the detection device 4a while the cooperative action is being performed with the user 5. This control model is a control model for controlling the action of the virtual character 902. The virtual character control unit 93 of the virtual space control unit 9 uses the control model generated by the control model generation unit 2a to control the virtual character 902 in the same manner as the fifth embodiment.

[0132] As described in the fifth embodiment, for example, the virtual space control unit 9 may be provided as an individual virtual space control device outside the control system 1e.

[0133] In this embodiment, the control target is a virtual character 902 instead of a robot 7, but similarly to the second embodiment, before the virtual character 902 and the user 5 perform a cooperative action, a control model is generated using personal behavior data of the cooperative action target 6 including cooperative action data acquired when the user 5 and the cooperative action target 6 performed a cooperative action. Therefore, the virtual character 902 can perform actions that are the same as or similar to the actions of the cooperative action target 6 from the start of the cooperative action, and stress on the user 5 caused by at least one of the actions and speech of the virtual character 902 during the cooperative action can be reduced.

[0134] The control system 1e of this embodiment is realized by a computer system, similar to the control system 1a of the second embodiment. The control system 1e of this embodiment may also be realized by a plurality of computer systems, for example, by a cloud system. As described above, the control model generating unit 2a and the virtual space control unit 9 may each be configured as separate devices, and in this case, each separate device may also be realized by a plurality of computer systems.

[0135] In the above example, the virtual characters 902 perform cooperative actions and the personal behavior data includes cooperative action data. However, as long as the control model of the virtual character 902 is generated using personal behavior data indicating the personality of a specific person, the action to be applied is not limited to cooperative actions. That is, the personal behavior data, which is data related to the actions of a specific person, is stored in the data storage unit 23, and the learning unit 22a uses the personal behavior data to generate a control model of the virtual character 902 in the virtual space that reflects the personality of the specific person. For example, the user 5 sets a specific person, and the control model of the virtual character 902 is generated using the personal behavior data of the specific person, so that a control model reflecting the personality of the specific person according to the desire of the user 5 is generated. This can reduce the stress of the user 5 when the user 5 talks to the virtual character 902 or watches the words and actions of the virtual character 902. In addition, the method of setting the specific person is not limited to this example. By generating a control model of the virtual character 902 based on the personal behavioral data of a specific person, a control model reflecting at least one of the behavior and speaking style of the specific person can be generated, and at least one of the individual characteristics of the specific person's behavior and speaking style can be reflected in the virtual character 902.

[0136] In the above example, the control model generating unit 2a of the second embodiment generates a control model for controlling the virtual character 902, and the virtual character control unit 93 controls the virtual character 902 using the generated control model. However, when controlling the virtual character 902, the action result acquiring unit 26 may be provided and the control model may be updated using the action result as in the third embodiment, or the basic control model to be used may be selected from a plurality of basic control models for controlling the virtual character 902 using the model selection receiving unit 27 as in the fourth embodiment. In addition, both the update of the control model using the action result and the selection of the basic control model to be used from a plurality of basic control models may be performed.

[0137] The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, or the embodiments may be combined with each other. Also, parts of the configurations may be omitted or modified without departing from the spirit of the invention. [Explanation of symbols]

[0138] 1, 1a, 1b, 1c, 1d, 1e control system, 2, 2a, 2b, 2c control model generation unit, 3 robot control unit, 4, 4a detection device, 5 user, 6 collaborative operation target, 7 robot, 8, 8a situation detection device, 9 virtual space control unit, 21, 21a basic model memory unit, 22, 22a, 22b learning unit, 23 data memory unit, 24 data acquisition unit, 25 correction information memory unit, 26 action result acquisition unit, 27 model selection reception unit, 31 instruction transmission unit, 32 situation acquisition unit, 33 control instruction generation unit, 34 control model memory unit, 91 transmission / reception unit, 92 virtual space generation unit, 93 virtual character control unit, 94 terminal device, 95 video presentation device, 96 audio presentation device, 100, 100a, 100b, 100c, 100d, 100e collaborative operation system.

Claims

1. a data storage unit for storing individual behavior data which is data related to the behavior of a person who has performed a cooperative behavior; a learning unit that uses the individual behavior data stored in the data storage unit to generate a control model of the humanoid for causing the humanoid to perform the collaborative action with a user, the control model reflecting the individuality of the person; and Equipped with A control model generating device, wherein the personal behavior data includes data acquired when the user is performing the cooperative action with a cooperative action target with which the user is familiar.

2. 2. The control model generating device according to claim 1, wherein the person is the user.

3. 2. The control model generating device according to claim 1, wherein the person is a target of the cooperative action.

4. 4. The control model generating device according to claim 1, wherein the humanoid is a robot.

5. 4. The control model generating device according to claim 1, wherein the humanoid is a virtual character in a virtual space.

6. 4. The control model generating device according to claim 1, wherein the learning unit updates the control model based on an action result that is a result of the collaborative action performed by the humanoid and the user and the control model that corresponds to the action result.

7. 6. The control model generating device according to claim 5, wherein the learning unit updates the control model based on an action result that is a result of the collaborative action performed by the humanoid and the user and the control model that corresponds to the action result.

8. 4. The control model generating device according to claim 1, wherein the learning unit generates the control model by using a predetermined basic control model and the individual behavior data.

9. 6. The control model generating device according to claim 5, wherein the learning unit generates the control model by using a predetermined basic control model and the individual behavior data.

10. a selection result receiving unit that receives a selection result indicating the basic control model selected by the user from among the plurality of basic control models; Equipped with 9. The control model generating device according to claim 8, wherein the learning unit generates the control model by using the basic control model indicated by the selection result and the individual behavior data.

11. The learning unit extracts a feature amount indicating a personality of the person from the personal behavior data, 4. The control model generating device according to claim 1, wherein the personality includes at least one of a catchphrase, an intonation, a dialect, and a habit of a manner of movement.

12. The learning unit extracts a feature amount indicating a personality of the person from the personal behavior data, 6. The control model generating device according to claim 5, wherein the personality includes at least one of a speech pattern, an intonation, a dialect, and a habit of a movement.

13. The control model generating device according to any one of claims 1 to 3, characterized in that the personal behavior data includes at least one of data acquired by a wearable terminal and data recording the movements of the person in a virtual space.

14. The control model generating device according to claim 5 , wherein the personal behavior data includes at least one of data acquired by a wearable terminal and data recording the person's movements in a virtual space.

15. A control model storage unit; a control instruction generation unit that generates control instructions for a humanoid; Equipped with the control model storage unit stores a control model of the humanoid for causing the humanoid to perform the cooperative action with a user, the control model being generated using personal behavior data, which is data related to the actions of a person who has performed a cooperative action, and the control model reflects the individuality of the person; the control instruction generation unit generates control instructions for the humanoid using the control model; A robot control device, characterized in that the personal behavior data includes data acquired when the user is performing the cooperative action with a cooperative action target with which the user is familiar.

16. A control model generation unit; A motion control unit that controls the humanoid; Equipped with The control model generation unit a data storage unit for storing individual behavior data which is data related to the behavior of a person who has performed a cooperative behavior; a learning unit that uses the individual behavior data stored in the data storage unit to generate a control model of the humanoid for causing the humanoid to perform the collaborative action with a user, the control model reflecting the individuality of the person; and Equipped with the motion control unit controls the humanoid using the control model; A control system, characterized in that the personal behavior data includes data acquired when the user is performing the collaborative activity with a collaborative activity partner with whom the user is familiar.

17. A control model generation method in a control model generation device, comprising: a step of storing personal behavior data, which is data on the behavior of a person who has performed a cooperative behavior, by the control model generating device; a step of generating, by the control model generation device, a control model of the humanoid for performing the collaborative action with a user, the control model reflecting the personality of the person, using the accumulated individual behavior data; Including, A control model generating method, characterized in that the personal behavior data includes data acquired when the user is performing the cooperative action with a cooperative action target person with whom the user is familiar.

18. In the computer system, A step of accumulating personal behavior data which is data on the behavior of a person who has performed a cooperative behavior; generating a control model of the humanoid that allows the humanoid to perform the collaborative action with a user, the control model reflecting the personality of the person, using the accumulated individual behavior data; Run the command, The program, wherein the personal behavior data includes data acquired when the user is performing the cooperative activity with a cooperative activity partner with whom the user is familiar.

Images