Control model generation device, robot control device, control system, control model generation method, and program

Abstract

A control model generation unit (2) according to the present disclosure comprising: a data storage unit (23) that stores personal behavior data which is data pertaining to an action of a person who has performed a cooperative action; and a learning unit (22) that generates, using the personal behavior data stored in the data storage unit (23), a control model for a humanoid, which is for causing the humanoid to cooperatively act with a user (5) and in which the person's personality is reflected. The personal behavior data includes data which is acquired when the user (5) is performing a cooperative action with a cooperative action subject (6) with whom the user (5) is familiar.

Description

Control model generation device, robot control device, control system, control model generation method and program

[0001] The present disclosure relates to a control model generation device, a robot control device, a control system, a control model generation method, and a program.

[0002] In recent years, the development of devices capable of interacting with humans, such as service robots and smart speakers, has progressed. Furthermore, in virtual spaces on computers, such as the Metaverse, it is now possible for virtual characters to interact with users. An example of a device or virtual character capable of interacting with humans is a humanoid. Humanoids are required to behave and speak in a way that is appropriate for each individual, so as not to cause stress when living with humans. However, the same device (or virtual character) generally behaves and speaks in the same way regardless of the user, i.e., the person interacting with it. Even the same device (or virtual character) can cause stress depending on the user.

[0003] Patent document 1 discloses a robot control device that controls the behavior of a robot so that the expected reaction to the robot's behavior matches the user's actual reaction to that behavior, enabling the robot to build an affiliative relationship with the user.

[0004] JP 2013-027937 A

[0005] However, in the technology described in Patent Document 1, when the expected reaction of the robot's behavior does not match the user's actual reaction to that behavior, the robot's behavior is controlled so that the expected reaction matches the user's actual reaction. In other words, if the robot's actual behavior is found to be inappropriate, the robot's behavior is controlled ex post. This results in inappropriate behavior, which can be quite stressful for the user. Furthermore, users may wish to have at least one of the individual characteristics of a particular person, either their behavior or their speaking style, reflected in the humanoid.

[0006] The present disclosure has been made in consideration of the above, and aims to provide a control model generation device that can reduce stress on a user caused by at least one of the behavior and speaking style of a humanoid.

[0007] In order to solve the above-mentioned problems and achieve the object, the control model generation device according to the present disclosure includes a data storage unit that stores personal behavior data, which is data related to the behavior of persons who have performed cooperative behavior, and a learning unit that uses the personal behavior data stored in the data storage unit to generate a control model of a humanoid that allows a humanoid to perform cooperative behavior with a user, the control model reflecting the personality of the person. The personal behavior data includes data acquired when the user performs cooperative behavior with a cooperative behavior target with whom the user is familiar.

[0008] The control model generation device according to the present disclosure has the effect of reducing stress on the user caused by at least one of the behavior and speech of the humanoid.

[0009] FIG. 1 is a diagram showing a configuration example of a cooperative operation system according to a first embodiment. FIG. 1 is a diagram showing an example of correction information according to a first embodiment. Schematic diagram showing an example of a neural network. Flowchart showing an example of a processing procedure in a control model generation unit according to the first embodiment. Flowchart showing an example of a processing procedure in a robot control unit according to the first embodiment. Schematic diagram showing an example of acquisition of personal behavior data in the first example. Schematic diagram showing an example of cooperative operation between a user and a robot in the first example. Schematic diagram showing an example of acquisition of personal behavior data in the second example. Schematic diagram showing an example of cooperative operation between a user and a robot in the second example. FIG. 1 is a diagram showing a configuration example of a computer system for realizing a control system according to the first embodiment. FIG. 1 is a diagram showing a configuration example of a cooperative operation system according to a second embodiment. Flowchart showing an example of a processing procedure in a control model generation unit according to the second embodiment. FIG. 1 is a diagram showing a configuration example of a cooperative operation system according to a third embodiment. Flowchart showing an example of a control model update processing procedure in the control model generation unit according to the third embodiment.

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A control model generating device, a robot control device, a control system, a control model generating method, and a program according to embodiments will be described in detail below with reference to the accompanying drawings.

[0011] 1 is a diagram showing an example of the configuration of a collaborative operation system according to embodiment 1. A collaborative operation system 100 of this embodiment includes a control system 1, a robot 7, a detection device 4, and a situation detection device 8.

[0012] The collaborative operation system 100 of this embodiment accumulates personal behavior data of the user 5, including collaborative operation data acquired when the robot 7 and the user 5 performed a collaborative operation with a collaborative operation target 6, before the collaborative operation between the robot 7 and the user 5 is performed, and generates a control model for controlling the robot 7 using the accumulated personal behavior data. The collaborative operation data is personal behavior data acquired while the robot 7 is performing a collaborative operation. The acquired personal behavior data may consist of only collaborative operation data, or may include data other than collaborative operation data.

[0013] The collaborative action includes, but is not limited to, at least one of conversation, work, games, and sports. The collaborative action target 6 is another person who performs the collaborative action with the user 5 and who has become familiar with the user 5 in the collaborative action. A person who is familiar with the user 5 in the collaborative action is, for example, a person who is not meeting the user for the first time and who knows the other person's personality or mannerisms to some extent, and who therefore feels a good chemistry with the user 5 in the collaborative action and is less likely to feel stressed when the user 5 performs the collaborative action together. The collaborative action target 6 may be determined, for example, by the user 5 or by an operator of the collaborative action system 100 other than the user 5. When determined by an operator of the collaborative action system 100, for example, a person who is clearly engaged in collaborative action with the user 5 over a long period of time is determined as the collaborative action target 6. The collaborative action target 6 may be one person or multiple people. When there are multiple collaborative action targets 6, individual behavior data including collaborative action data when the user 5 performed a collaborative action with each of the collaborative action targets 6 is acquired. For example, if the collaborative operation targets 6 include a first collaborative operation target and a second collaborative operation target, the personal behavior data of user 5 includes collaborative operation data acquired during collaborative operation between the first collaborative operation target and user 5, and collaborative operation data acquired during collaborative operation between the second collaborative operation target and user 5.

[0014] The personal behavior data is data related to the behavior of a person who has performed a cooperative action. In this embodiment, the personal behavior data of user 5 is acquired, and therefore the personal behavior data is data that reflects the individuality of user 5. Since the personal behavior data of user 5 includes data acquired when user 5 was performing a cooperative action with a cooperating action target 6 with which user 5 is familiar, it can also be said that the personal behavior data of user 5 indirectly reflects the individuality of the cooperating action target 6. Furthermore, since the control model is generated using personal data including data of user 5 acquired when user 5 was performing a cooperative action with a cooperating action target 6 with which user 5 is familiar, the individuality of user 5 is reflected in the control model. Here, the individuality of user 5 reflected in the control model is not simply the individuality of user 5, but the individuality of user 5 when performing a cooperative action with a cooperating action target 6 with which user 5 is familiar. Therefore, the control model indirectly reflects not only the individuality of user 5 but also the individuality of the cooperating action target 6 with which user 5 is familiar. The indirectly reflected control here refers to control that reflects the behavior or manner of speech of the collaborative action target 6 when interacting with the user 5, and can also be said to be control that includes elements that are characteristic of the behavior or manner of speech that constitute the individuality of the collaborative action target 6. This allows the robot 7 to perform actions that do not cause stress to the user 5 when performing collaborative actions.

[0015] The personal behavior data includes the user 5's behavioral habits. Here, the behavior is considered to be a target used for controlling the robot 7, and may include not only movement but also at least one of the user 5's speaking style and the user 5's position and posture. That is, the behavior includes, for example, at least one of the user's movement style, speaking style, and at least one of the user 5's position and posture. The movement style may not only indicate continuous movement, but also at least one of the user's position and posture at a given moment. The behavioral habits include at least one of the user's movement habits and speaking habits. The movement habits are, for example, at least one of the user's movement style, such as the trajectory and speed of movement, and gestures. The speaking habits are, for example, at least one of the user's speaking speed, speaking habits, speaking style, intonation, dialect, etc., but are not limited to these. The movements, actions, speaking styles, habits, etc. of the user 5, the collaborative operation target 6, and the robot 7 may also be referred to as behavior.

[0016] The robot 7 is an example of a humanoid that operates in cooperation with the user 5. Note that a humanoid may also be referred to as a controlled object. More specifically, in this embodiment, the robot 7 is an example of a machine that operates in cooperation with the user 5. The robot 7 may be humanoid, a machine that does not have any moving parts and only communicates with the user 5, an industrial machine that has a manipulator or the like, or something other than these, and there are no particular restrictions on the shape or function of the robot 7.

[0017] The control system 1 controls a robot 7. The control system 1 includes a control model generation unit 2 that generates a control model of the robot 7, and a robot control unit 3 that controls the robot 7 using the control model generated by the control model generation unit 2 and situation data detected by a situation detection device 8. The control model generation unit 2, which is a control model generation device, and the robot control unit 3, which is a robot control device, may be integrated together, or may be provided separately.

[0018] The detection device 4 detects the behavior of the user 5 when the user 5 is performing a cooperative action with the cooperative action target 6 as individual behavior data, and transmits the individual behavior data to the control model generation unit 2. The detection device 4 is a device that detects at least one of, for example, position, speed, acceleration, posture, voice, pulse, blood pressure, body temperature, emotion, etc., and there may be multiple detection devices 4.

[0019] In addition, the detection device 4 may detect psychological or internal information such as biometric information or emotions of the user 5 when the user 5 is performing a cooperative action with the cooperative action target 6, and include the information in the personal behavior data and transmit it to the control model generation unit 2.

[0020] The detection device 4 may be a wearable device that can be worn by the user 5 or a portable device that can be carried by the user 5. Furthermore, the detection device 4 may be a device installed so as to be able to detect the user 5's movements, or a device that detects movements in a virtual space such as the Metaverse. The detection device 4 may be a combination of these, or something else. When the detection device 4 is a wearable device or a portable device, the detection device 4 may be a device capable of detecting at least one of its own position, velocity, and acceleration, a device capable of detecting its own rotation, a device equipped with a microphone capable of collecting and recording sound, or a combination of these. The position of the wearable device may be detected using a Global Positioning System (GPS) receiver, an RFID (Radio Frequency Identification) tag, or other devices. Using a wearable device or a portable device as the detection device 4 allows for daily acquisition of the user 5's personal behavior data, regardless of the user 5's location. When recording voice, sounds uttered by persons other than the user 5 may also be recorded, but the detection device 4, the control model generation unit 2, or a device not shown in Fig. 1 performs voice recognition processing to extract the voice of the user 5. The voice recognition processing may be any processing, but may also be, for example, a processing in which the voice of the user 5 is acquired in advance and the voice uttered by the user 5 is identified using the previously acquired voice.

[0021] When a device installed so as to be able to detect the movements of the user 5 is used as the detection device 4, the device may be, for example, a photographing device such as a camera that photographs the location where the cooperative movement is performed, or a device such as a microphone that can collect and record sound at the location where the cooperative movement is performed, or a combination of these. When the detection device 4 is a photographing device, the control model generation unit 2 or a device not shown in FIG. 1 recognizes the user 5 in the video captured by the photographing device and detects the position, speed, acceleration, movement trajectory, etc. of the user 5. The user 5 can be recognized by a general image recognition method, such as using an image of the user 5 captured in advance. Alternatively, the detection device 4 may recognize the user 5 and detect the position, speed, acceleration, movement trajectory, etc. of the user 5. A general method can also be used to detect the position, speed, acceleration, movement trajectory, etc. of the user 5.

[0022] When the detection device 4 is a device that detects movements in a virtual space such as the Metaverse, the device may be, for example, a computer system that manages the virtual space, a terminal device that the user 5 uses to make an avatar of the user 5 move in the virtual space, or a device that records video in the virtual space. When a device that records video in the virtual space is used as the detection device 4, the control model generation unit 2 or a device not shown in FIG. 1 recognizes the user 5 in the video captured by the imaging device and detects the position, speed, acceleration, movement trajectory, etc. of the user 5.

[0023] Furthermore, if the collaborative action includes an action using an object, such as moving or processing an object, the detection device 4 may include a device that detects the force that the user 5 applies to the object.

[0024] The control model generation unit 2 includes a basic model storage unit 21, a learning unit 22, a data storage unit 23, a data acquisition unit 24, and a correction information storage unit 25. The basic model storage unit 21 stores a predetermined basic control model that serves as a reference for the control model for controlling the robot 7. The basic control model is a general control model that does not depend on the user 5, i.e., does not reflect the individuality of the user 5, and is a model that defines the basic operations of the robot 7.

[0025] The basic control model may be stored in advance in the data storage unit 23 by a vendor of the control system 1 or a vendor of the robot 7, or may be transmitted from another device, received by a communication unit (not shown in FIG. 1 ), and stored in the data storage unit 23. For example, the user 5 may operate the control system 1, causing the control system 1 to receive a basic control model from an external server or the like that provides a basic control model suitable for the robot 7. The basic control model may also be provided for each type of robot 7, or may be provided according to the type of collaborative movement performed by the robot 7. For example, basic control models suitable for the type of collaborative movement performed by the robot 7 may be stored in the basic model storage unit 21, and the user 5 may select a basic control model suitable for the type of collaborative movement performed by the robot 7 and the user 5. Alternatively, an external server or the like may provide basic control models suitable for the type of collaborative movement performed by the robot 7, and the user 5 may select and download a basic control model suitable for the type of collaborative movement performed by the robot 7 and the user 5, thereby storing the basic control model in the basic model storage unit 21.

[0026] The basic control model and the control model include, for example, one or more control parameters for controlling the robot 7. The control parameters include at least one of parameters for controlling the movement of the robot 7, such as the trajectory, speed, and acceleration of the robot 7, parameters for controlling the movement of each part of the robot 7, such as the hands and joints of the robot 7, and parameters for controlling the way the robot 7 speaks, such as the speaking speed of the robot 7 and the pitch (frequency) of the voice emitted by the robot 7.

[0027] The data acquisition unit 24 acquires personal behavior data of the user 5 from the detection device 4 and stores the acquired personal behavior data in the data storage unit 23. As described above, to obtain the personal behavior data, processing such as image processing on video acquired by the detection device 4 and voice recognition processing on audio acquired by the detection device 4 may be performed. In this case, these processes may be performed by another device (not shown), and the data acquisition unit 24 acquires the personal behavior data from the other device. The data acquisition unit 24 may also perform extraction processing such as image processing on video acquired by the detection device 4 and voice recognition processing on audio acquired by the detection device 4. In this case, the data acquisition unit 24 may perform extraction processing on the data acquired from the detection device 4 and store the processed data in the data storage unit 23 as personal behavior data. Alternatively, the data acquired from the detection device 4 itself may be stored in the data storage unit 23 as personal behavior data, and the learning unit 22 may perform extraction processing in the process of generating a control model, which will be described later. Note that, although the data acquisition unit 24 acquires the personal behavior data by receiving the personal behavior data here, this is not limiting, and the personal behavior data may also be recorded on a recording medium or the like. In this case, the data acquisition unit 24 acquires the personal behavior data by reading the personal data from the recording medium.

[0028] The correction information storage unit 25 stores correction information indicating corrections to the basic control model according to the individual behavior data. The correction information is, for example, information associating one or more feature amounts indicated by the individual behavior data with the correction amount. The feature amounts indicate the personality of the person from whom the individual behavior data is acquired. The personality amount includes, for example, at least one of behavior, catchphrases, intonation, dialect, and habitual movements. The correction amount is determined according to the personality of the user 5 so that the robot 7 performs movements that do not cause stress to the user 5 when cooperatively operating with the robot 7. The learning unit 22 uses the individual behavior data stored in the data storage unit 23, i.e., the accumulated individual behavior data, to generate a control model for the robot 7 that reflects the personality of the user 5 and allows the robot 7 to cooperatively operate with the user 5. The learning unit 22 generates the control model using, for example, the basic control model stored in the basic model storage unit 21, the individual behavior data accumulated in the data storage unit 23, and the correction information stored in the correction information storage unit 25.

[0029] FIG. 2 is a diagram illustrating an example of correction information according to the present embodiment. In the example illustrated in FIG. 2, the personality of the user 5 is classified into types based on N (N is an integer equal to or greater than 1) feature quantities, and the correction information includes correction details for parameters (control parameters) in the control model for each type. The feature quantities may indicate the direction of movement of the user 5 during cooperative motion as an angle from a reference direction, or may numerically indicate the amount of movement from a reference point during cooperative motion by the user 5, or may be information obtained by frequency-converting time-series data on the position of the user 5 during cooperative motion, the speaking speed of the user 5, or information obtained by frequency-converting the voice of the user 5. Furthermore, the feature quantities may indicate whether the user 5 performed a specific predetermined movement, the number of times the user 5 performed a specific predetermined movement per unit time, whether the user 5 uttered a specific word, or the number of times the user 5 uttered a specific word per unit time. Furthermore, as feature quantities indicating the habits or personality of the user 5, for example, behaviors that tend to vary significantly from individual to individual and that exhibit a specific pattern may be extracted. Furthermore, the feature may be the individual behavior data itself, or the collaborative action data in the individual behavior data itself. The feature is not limited to the above example, and may be anything that indicates at least one of the characteristics of the user 5's way of moving and speaking, that is, the habits of the user 5. Note that, although FIG. 2 shows an example in which N is 3 or more, the present invention is not limited to this, and N may be anything that is 1 or more.

[0030] Although an example in which the correction information is defined in a table format has been described in Fig. 2, the format of the correction information is not limited to the example shown in Fig. 2. When the correction information shown in Fig. 2 is used, the learning unit 22 extracts features from the personal behavior data accumulated in the data storage unit 23, identifies a type corresponding to the extracted features using the correction information, extracts correction content corresponding to the identified type from the correction information, and corrects the basic control model based on the extracted correction content, thereby generating a control model.

[0031] The correction information may be determined manually in advance. For example, the correction information may be determined by a vendor or an administrator of the robot 7 or the control system 1, or may be determined by learning through machine learning (pre-learning).

[0032] In the former case, the vendor or administrator determines the correction information by inferring how to correct the behavior of the robot 7 based on the basic control model for each type of user 5, depending on the content of the collaborative behavior, so that the user 5 does not feel stressed.

[0033] In the latter case, for example, before the robot 7 is put into operation, the robot 7 is caused to perform a cooperative action with an arbitrary person, and for each cooperative action, a set of data sets is obtained, which includes feature amounts extracted from the personal behavioral data of the person who performed the cooperative action and corrections to each control parameter in the control model of the robot 7 (corrections from the basic control model). Furthermore, for each cooperative action, an evaluation is performed to indicate whether the person who performed the cooperative action with the robot 7 felt stress. By appropriately changing the action of the robot 7 and the person performing the cooperative action, multiple data sets under different conditions and corresponding evaluation results are obtained. Note that the person performing the cooperative action at this time may be any person, and may or may not include the user 5. Furthermore, the person performing the cooperative action at this time does not have to be the robot 7 itself that performs the cooperative action with the user 5, but may be another robot of the same type as the robot 7 or a different type of robot that can perform the same action as the robot 7.

[0034] When multiple data sets with different conditions and corresponding evaluation results that are correct data are acquired, a trained model is generated by supervised learning using corrections to each control parameter in the data set that obtained an evaluation result indicating that the user did not feel stress as correct data. The pre-learning may be performed by the learning unit 22, a pre-learning unit (not shown) of the control model generation unit 2, or a learning device separate from the control system 1. When pre-learning is performed, the correction information is a trained model for inferring corrections to the control parameters from features extracted from the personal behavior data. The learning unit 22 can infer corrections to the control parameters that are appropriate for the user 5 by inputting the features extracted from the personal behavior data of the user 5 into the trained model. The learning unit 22 generates a control model by reflecting the inferred corrections in the basic control model.

[0035] Any algorithm may be used for supervised learning, but for example, a neural network model may be used. A neural network is composed of an input layer consisting of multiple neurons, an intermediate layer (hidden layer) consisting of multiple neurons, and an output layer consisting of multiple neurons. The intermediate layer may be one layer, or two or more layers.

[0036] Figure 3 is a schematic diagram showing an example of a neural network. For example, in a three-layer neural network as shown in Figure 3, when multiple inputs are input to the input layer (X1-X3), the values ​​are multiplied by weight W1 (w11-w16) and input to the intermediate layer (Y1-Y2), and the result is further multiplied by weight W2 (w21-w26) and output from the output layer (Z1-Z3). This output result varies depending on the values ​​of weight W1 and weight W2.

[0037] In this embodiment, the weights W1 and W2 are adjusted so that the output from the output layer when the feature extracted from the individual behavior data is input approaches the correction content of the control parameter, which is the correct data, thereby learning the relationship between the feature and the correct data. Note that the machine learning algorithm is not limited to a neural network and may be other algorithms such as a support vector machine. Furthermore, the machine learning used to generate the trained model is not limited to supervised learning and may be reinforcement learning or the like.

[0038] Furthermore, when pre-learning is performed, table-format information may be used as the correction information. For example, after the trained model described above is generated, multiple pieces of input data are generated by changing the values ​​of each feature, and control parameters obtained by inputting each piece of input data into the trained model are inferred. Then, input data for which the correction contents of the control parameters obtained by inference are all the same or the difference is within a certain range may be defined as one type, and control parameters for each type may be determined, and correction information in the table format shown in FIG. 2 may be generated.

[0039] Furthermore, even when supervised learning, reinforcement learning, or the like is not used, the movement of the robot 7 and the person who performs the cooperative movement may be changed as appropriate to obtain a plurality of data sets under different conditions, and correction information may be determined using the obtained data sets and the corresponding evaluation results. For example, the values ​​of the type classification thresholds such as X1 and X2 and the control parameters may be determined manually using the correction details of each control parameter in a data set in which the evaluation result that the robot 7 did not feel stressed and the feature quantities of the personal behavior data.

[0040] Although an example using correction information has been described above, depending on the content of the collaborative action, overall information indicating the sum of an index obtained by the action of the robot 7 and an index obtained by the action of the user 5 in the collaborative action may be defined instead of the correction information. For example, if the collaborative action is an action in which the robot 7 and the user 5 cooperate to pull an object with a certain force, if the user 5 pulls with a weak pulling force, the collaborative action will fail, causing stress for the user 5. Also, if the user 5 pulls with a strong pulling force, the robot 7 will need to weaken the pulling force, causing stress for the user 5. In such a collaborative action, the pulling force of the user 5 is acquired as individual behavior data, and the total force is defined as overall information. Then, the learning unit 22 calculates the pulling force of the robot 7 by subtracting the pulling force of the user 5 from the total force, and calculates the correction amount of the control parameter based on the calculated force. The above-described overall information is an example, and the overall information is not limited to the above example.

[0041] Returning to the explanation of Fig. 1, the learning unit 22 outputs the generated control model to the robot control unit 3. The robot control unit 3 includes an instruction sending unit 31, a situation acquisition unit 32, a control instruction generation unit 33, and a control model storage unit 34. The robot control unit 3 is an example of a movement control unit (movement control device) that controls a humanoid.

[0042] The control model storage unit 34 stores the control model output from the learning unit 22. The situation acquisition unit 32 acquires situation data indicating the situation of the cooperative movement between the robot 7 and the user 5 acquired by the situation detection device 8 by receiving the data from the situation detection device 8, and outputs the acquired situation data to the control instruction generation unit 33. The situation detection device 8 may be provided on the robot 7, may be provided in the vicinity of the robot 7, or may be provided on both the robot 7 and the vicinity of the robot 7. The situation detection device 8 acquires situation data used for controlling the robot 7 according to the type of robot 7 and the content of the cooperative movement. The situation detection device 8 may acquire the situation of the movement of the user 5, such as the voice uttered by the user 5 and the movement of the user 5. There may be multiple situation detection devices 8. The situation detection device 8 may be, for example, a camera that detects the position of the robot 7, the state of the surroundings of the robot 7, etc., or may be an acceleration sensor, a force sensor, etc. Furthermore, the situation detection device 8 may be a photographing device or the like for grasping the positional relationship between the target object and the robot 7 when the robot 7 moves the target object or applies force to the target object. The situation detection device 8 may be two or more of these, or may be other than these, and any sensor generally used for controlling the robot 7 may be used. Note that situation data may not be used for controlling the robot 7, and in this case, the situation detection device 8 may not be provided.

[0043] The control instruction generation unit 33 generates control instructions for the robot 7 using the situation data received from the situation acquisition unit 32 and the control model stored in the control model storage unit 34, and outputs the generated control instructions to the instruction transmission unit 31. The instruction transmission unit 31 transmits the control instructions received from the control instruction generation unit 33 to the robot 7. The robot 7 that has received the control instructions operates based on the control instructions.

[0044] As described above, in this embodiment, the learning unit 22 generates a control model using personal behavior data from when the user 5 is performing cooperative action with the familiar cooperative action target 6, and control instructions based on the generated control model are transmitted to the robot 7. Because the control model is generated before the cooperative action between the user 5 and the robot 7, the robot 7 can perform actions that are the same as or similar to the actions of the cooperative action target 6 from the start of the cooperative action, thereby reducing stress on the user 5 caused by at least one of the actions and speaking style of the robot 7 during the cooperative action.

[0045] Next, the operation of this embodiment will be described. FIG. 4 is a flowchart showing an example of a processing procedure in the control model generation unit 2 of this embodiment. The control model generation unit 2 acquires personal behavior data of the user 5, including cooperative behavior data acquired during cooperative behavior between the user 5 and the cooperative behavior target 6 (step S1). In detail, the data acquisition unit 24 acquires the personal behavior data of the user 5 by receiving the personal behavior data from the detection device 4. As described above, the data acquisition unit 24 may acquire the personal behavior data from a recording medium. Alternatively, a video or the like that is the source of the personal behavior data may be acquired by the detection device 4 and subjected to extraction processing.

[0046] The control model generation unit 2 stores the individual behavior data of the user 5 (step S2). Specifically, the data acquisition unit 24 stores the received individual behavior data in the data storage unit 23.

[0047] The control model generation unit 2 generates a control model using the accumulated individual behavior data (step S3). Specifically, the learning unit 22 extracts features using the individual behavior data of the user 5 stored in the data storage unit 23, and generates a control model using the features and the basic control model stored in the basic model storage unit 21. The accumulated individual behavior data is individual behavior data acquired for each cooperative behavior in one or more cooperative behaviors. For example, if the feature is the speaking speed of the user 5, and individual behavior data corresponding to multiple cooperative behaviors has been accumulated, the average speed per character may be calculated using all of the individual behavior data corresponding to the multiple cooperative behaviors. For example, if the feature indicates the movement tendency of the user 5, the individual behavior data corresponding to the multiple cooperative behaviors may be used to calculate an average position at a predetermined time point in the cooperative behavior, and the difference between the average position and a predetermined standard position may be used as the feature. The method of calculating the feature is not limited to the above example.

[0048] The control model generation unit 2 outputs the control model (step S4). Specifically, the learning unit 22 outputs the generated control model to the robot control unit 3. The control model storage unit 34 of the robot control unit 3 stores the control model output from the learning unit 22.

[0049] 5 is a flowchart showing an example of a processing procedure in the robot control unit 3 of this embodiment. The processing shown in FIG. 5 is performed when the robot 7 and the user 5 perform a cooperative operation after the control model generation unit 2 generates a control model.

[0050] The robot control unit 3 acquires situation data (step S11). Specifically, the situation acquisition unit 32 acquires situation data indicating the situation of the robot 7 acquired by the situation detection device 8 by receiving the situation data from the situation detection device 8, and outputs the acquired situation data to the control instruction generation unit 33.

[0051] The robot control unit 3 generates a control instruction using the situation data and the control model (step S12). Specifically, the control instruction generation unit 33 generates a control instruction for the robot 7 using the situation data received from the situation acquisition unit 32 and the control model stored in the control model storage unit 34, and outputs the generated control instruction to the instruction transmission unit 31.

[0052] The robot control unit 3 transmits the control instruction (step S13). Specifically, the instruction transmission unit 31 transmits the control instruction received from the control instruction generation unit 33 to the robot 7. As a result, the robot 7 performs an operation based on the control instruction.

[0053] Note that, once a control model is generated and the robot 7 and the user 5 perform a cooperative action, if the user 5 and the cooperative action target person 6 perform a cooperative action, cooperative action data for the cooperative action may be acquired, and a control model may be generated based on personal behavior data including the acquired cooperative action data. In this case, the process shown in FIG. 5 is performed using the newly generated control model. In this way, once a control model is generated, the control model may be updated using new cooperative action data. This allows the robot 7 to be controlled to reflect the latest state even if the personality of the user 5 changes.

[0054] Next, an example of a cooperative operation performed using the cooperative operation system 100 of this embodiment will be described. First, as a first example of cooperative operation, a case will be described in which the robot 7 is a serving robot in a restaurant. In the first example, the cooperative operation is serving food. For example, Person A, Person B, Person C, and Person D all work at the same restaurant, and Person B sometimes serves food with Person A, sometimes with Person C, and sometimes with Person D. Person B performs the task well when serving food with Person A, and also when serving food with Person C. On the other hand, when Person B serves food with Person D, Person B is unable to perform the task comfortably and feels stressed. Persons A and C are planning to retire, and after Person A and C retire, Person B is scheduled to perform the task of serving food with the robot 7. In this case, in preparation for cooperative operation with the robot 7, individual behavior data is acquired, including cooperative operation data when User B, who is User 5, performs cooperative operation with Collaborative Operation Target Person 6. In this case, the cooperating action target persons 6 who are familiar with cooperating action with B are A and C.

[0055] FIG. 6 is a schematic diagram showing an example of acquiring personal behavior data in the first example. In the example shown in FIG. 6, user 5 and cooperating activity target person 6 serve plates placed on a serving counter 201 to tables 202 in the dining area of ​​a restaurant. As shown in FIG. 6, when user 5 (person B) is serving food together with cooperating activity target person 6 (person A or person C) at a restaurant, personal behavior data of person B is acquired by detection device 4. For example, as shown in FIG. 6, when user 5 is serving food together with cooperating activity target person 6, user 5 serves multiple small plates, while cooperating activity target person 6 serves large plates. On the other hand, when user 5 is serving food together with person D, person D serves multiple small plates, and person B serves large plates.

[0056] In the example shown in FIG. 6 , the personal behavior data includes information indicating which dishes were served. For example, if the positions of small and large plates are generally predetermined at the serving counter 201, where food prepared in the kitchen is temporarily placed before serving, a detection device 4 that detects the position of the user 5 may be used, and time-series data of the positions detected by the detection device 4 may be used as personal behavior data. In this case, for example, the learning unit 22 may obtain a history of the user 5's movements and determine the size of the plates served by the user 5 as a feature based on the obtained history and the positions of the small and large plates. Alternatively, a detection device 4 capable of photographing the serving counter 201 may be used, and the size and number of plates served by the user 5 may be calculated as personal behavior data by the data acquisition unit 24 or another device by analyzing video captured by the detection device 4. Alternatively, a detection device 4 capable of photographing the serving counter 201 may be used, and video captured by the detection device 4 may be used as personal behavior data, and the learning unit 22 may obtain the size of the plates served by the user 5 as a feature from the video.

[0057] In the first example, it is assumed that the plates to be served in the serving task include both large and small plates. Therefore, the feature values ​​include the size of the plates to be served, or the size and number of plates to be served. The correction information defines the type as either a type for serving multiple small plates or a type for serving large plates. Then, the correction information defines the correction content corresponding to the type, by specifying a numerical value as a control parameter for the robot 7 to serve large plates. For example, in the first example, the control model includes a serving judgment model and a movement model, and the serving judgment model includes a definition of the size of plates to be served by the robot 7. Then, the correction content corresponding to the above-described type in the correction information defines information for setting a control parameter for the robot 7 to serve plates with a diameter equal to or larger than a certain value. This allows the learning unit 22 of the control model generation unit 2 to generate a control model that causes the robot 7 to serve large plates. In addition, if the positions at the serving counter 201 where small and large plates are placed are roughly determined in advance, instead of specifying the size of the plate, the range at the serving counter 201 within which the plates to be served are located may be defined as a control parameter.

[0058] 7 is a schematic diagram showing an example of cooperative operation between user 5 and robot 7 in the first example. In the example shown in FIG. 7, a control model for causing robot 7 to serve food on a platter has been generated as described with reference to FIG. 6, and therefore robot 7 serves food on a platter. This allows user 5 to perform the food serving task efficiently with reduced stress, similar to when serving food together with cooperative operation target persons 6, A and C. In this way, in the first example, control system 1 can learn a food serving method that allows user 5, B, to act efficiently, and generate a control model that reflects the learned results.

[0059] Next, as a second example of cooperative behavior, a case in which the robot 7 and the user 5 perform assembly work will be described. In the second example, the robot 7 is, for example, an assembly robot, which is a type of industrial machine. For example, Person A, Person B, Person C, and Person D are all workers who perform assembly work, and Person B sometimes performs assembly work with Person A, sometimes with Person C, and sometimes with Person D. It is assumed that Person B performs the assembly work well when performing the assembly work with Person A, and also when performing the assembly work with Person C. On the other hand, it is assumed that when Person B performs the assembly work with Person D, Person B, who is the user 5, is unable to perform the assembly work comfortably and feels stressed. Persons A and C are scheduled to be transferred, and after Person A and C are transferred, Person B is scheduled to perform the assembly work with the robot 7. In this case, as in the first example, in preparation for cooperative behavior with the robot 7, individual behavior data including cooperative behavior data when Person B, who is the user 5, performs a cooperative behavior with the cooperative behavior target person 6 is acquired. In this case, the cooperating action target persons 6 who are familiar with cooperating action with B are A and C.

[0060] FIG. 8 is a schematic diagram illustrating an example of acquiring individual behavior data in the second example. In the example shown in FIG. 8 , user 5 and collaborative target person 6 cooperate to perform assembly work. More specifically, user 5 (person B) places a first part 204, and collaborative target person 6 (person A or C) places a second part 205 on top of the first part 204. Standard position 203 indicates the standard position where the first part 204 is placed. User 5 has a habit of placing the first part 204 to the right of the standard position in FIG. 8 . Collaborative target person 6, who is familiar with user 5, places the second part 205 in the same position where user 5 places the first part 204, thereby enabling efficient assembly work. On the other hand, User D, who is not familiar with User 5, attempts to place second component 205 on the assumption that first component 204 will be placed in standard position 203, which takes time to align and requires User 5 to change the position of first component 204, making it impossible to assemble efficiently and causing User 5 stress.

[0061] In such a case, the detection device 4 detects the location where the user 5 places the first part 204, or the position of the user 5's hand when the user 5 places the first part 204. Then, using the difference of the first part 204 from the standard position 203 as a feature, a type is defined in the correction information in which the placement position of the first part 204 is deviated from the standard position 203 by a threshold or more. Then, as a correction content corresponding to the type in the correction information, content for determining a control parameter such that the position of the second part 205 to be placed by the robot 7 is shifted by the same amount as the difference between the placement position of the first part 204 and the standard position 203 is included. This allows the learning unit 22 of the control model generation unit 2 to generate a control model that causes the robot 7 to place the second part 205 in accordance with the amount by which the user 5 has shifted the first part 204 from the standard position 203.

[0062] 9 is a schematic diagram showing an example of cooperative operation between the user 5 and the robot 7 in the second example. In the example shown in FIG. 9, as described with reference to FIG. 8, a control model is generated that causes the robot 7 to place the second part 205 in accordance with the amount by which the user 5 has shifted the first part 204 from the standard position 203, so the robot 7 places the second part 205 by shifting it to the right. This allows the user 5 to perform the assembly work efficiently with reduced stress, just as if they were performing the assembly work together with the cooperative operation target person 6, Person A or Person C.

[0063] In both the first and second examples, the robot 7 is able to work together with the user 5, taking into consideration the behavior of the user 5, person B, and even in an environment where humans work together to reduce manpower, and the robot 7, which is an example of a humanoid, the user 5 can perform cooperative operations in a state where stress such as difficulty in working and discomfort is reduced. Note that the cooperative operations performed using the above-described cooperative operation system 100 are merely examples, and the cooperative operations performed using the cooperative operation system 100 are not limited to the above-described examples.

[0064] Note that the cooperative action is not limited to two people, but may be performed by three or more people. For example, in the case of a cooperative action performed by three people, two robots 7 may be used, or the robot 7 and the cooperative action target person 6 may perform the cooperative action together with the user 5. In this case, for example, if user B, who is user 5, performs a cooperative action with users A and C, and users B and A do not feel stressed, a control model is generated based on the personal behavior data of user C, and the robot 7 is controlled based on this control model.

[0065] Next, the hardware configuration of each device of this embodiment will be described. In the control system 1 of this embodiment shown in FIG. 1, a computer program describing the processing in the control system 1 is executed on the computer system, causing the computer system to function as the control system 1. FIG. 10 is a diagram showing an example of the configuration of a computer system that realizes the control system 1 of this embodiment. As shown in FIG. 10, this computer system includes a control unit 101, an input unit 102, a memory unit 103, a display unit 104, a communication unit 105, and an output unit 106, which are connected via a system bus 107. The control unit 101 and the memory unit 103 form a processing circuit.

[0066] In FIG. 10 , the control unit 101 is a processor such as a CPU (Central Processing Unit) and executes a program describing the processes in the control system 1 of this embodiment. Note that a portion of the control unit 101 may be realized by dedicated hardware such as a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array). The input unit 102 may be an input means such as a button, keyboard, mouse, joystick, touchpad, or game controller. The storage unit 103 includes various types of memory such as RAM (Random Access Memory) and ROM (Read Only Memory) and a storage device such as a hard disk, and stores programs to be executed by the control unit 101, necessary data obtained during processing, and the like. The storage unit 103 is also used as a temporary storage area for programs. As described above, the display unit 104 is, for example, a display. Note that the display unit 104 and the input unit 102 may be integrated and realized by a touch panel or the like. The communication unit 105 is a receiver and transmitter that perform communication processing. The output unit 106 is a speaker or the like. Note that Fig. 10 is an example, and the configuration of the computer system is not limited to the example of Fig. 10. For example, in this embodiment, the computer system that realizes the control system 1 does not need to include the output unit 106.

[0067] Here, an example of the operation of the computer system until the program of this embodiment is ready to be executed will be described. In the computer system having the above configuration, for example, a computer program is installed in storage unit 103 from a CD-ROM or DVD-ROM inserted in a CD (Compact Disc)-ROM drive or DVD (Digital Versatile Disc)-ROM drive (not shown). Then, when the program is executed, the program read from storage unit 103 is stored in the main storage area of ​​storage unit 103. In this state, control unit 101 executes processing as control system 1 of this embodiment in accordance with the program stored in storage unit 103.

[0068] In the above explanation, a program describing the processing in the control system 1 is provided using a CD-ROM or DVD-ROM as a recording medium, but this is not limited to this, and depending on the configuration of the computer system, the capacity of the program to be provided, etc., it is also possible to use a program provided via a transmission medium such as the Internet.

[0069] The program of this embodiment causes a computer system to execute, for example, the steps of accumulating personal behavior data, which is data relating to the behavior of people who have performed cooperative actions, and using the accumulated personal behavior data, generating a control model of a humanoid that allows the humanoid to perform cooperative actions with a user 5, the control model reflecting the personality of the person.

[0070] The learning unit 22 and the control instruction generation unit 33 shown in FIG. 1 are realized by the control unit 101 shown in FIG. 10 executing a computer program stored in the storage unit 103 shown in FIG. 10. The storage unit 103 shown in FIG. 10 is also used to realize the learning unit 22 and the control instruction generation unit 33 shown in FIG. 1. The data acquisition unit 24, the instruction transmission unit 31, and the situation acquisition unit 32 shown in FIG. 1 are realized by the communication unit 105 shown in FIG. 10. The data acquisition unit 24 may also be realized by a device that reads a recording medium. The basic model storage unit 21, the data storage unit 23, the correction information storage unit 25, and the control model storage unit 34 shown in FIG. 1 are part of the storage unit 103 shown in FIG. 10.

[0071] 1 may be realized by multiple computer systems. For example, the control system 1 may be realized by a cloud system. As described above, the control model generation unit 2 and the robot control unit 3 may each be configured as separate devices, and in this case, each separate device may also be realized by multiple computer systems.

[0072] As described above, the collaborative operation system 100 of this embodiment generates a control model for controlling the robot 7 using personal behavior data of the user 5, including collaborative operation data acquired when the user 5 and the collaborative operation target person 6 performed collaborative operations, before the collaborative operation between the robot 7 and the user 5 is performed. The robot 7 can perform actions that are the same as or similar to the actions of the collaborative operation target person 6 from the start of the collaborative operation, thereby reducing stress on the user 5 caused by at least one of the actions and speaking style of the robot 7 during the collaborative operation.

[0073] Second Embodiment Fig. 11 is a diagram showing an example of the configuration of a collaborative operation system according to a second embodiment. A collaborative operation system 100a according to this embodiment is similar to the collaborative operation system 100 according to the first embodiment, except that it includes a control system 1a instead of the control system 1 and a detection device 4a instead of the detection device 4. Components having the same functions as those in the first embodiment are assigned the same reference numerals as those in the first embodiment, and redundant explanations will be omitted. Differences from the first embodiment will be mainly explained below.

[0074] In the first embodiment, a control model is generated using personal behavior data of the user 5, including cooperative behavior data acquired when the user 5 and the cooperating behavior target person 6 are performing a cooperative behavior. In the present embodiment, a control model is generated using personal behavior data of the cooperating behavior target person 6, including cooperative behavior data acquired when the user 5 and the cooperating behavior target person 6 are performing a cooperative behavior. In the present embodiment, the personal behavior data, which is data related to the behavior of a person who has performed a cooperative behavior, is the personal behavior data of the cooperating behavior target person 6. That is, the person from whom personal behavior data is acquired is the user 5 in the first embodiment, and the cooperating behavior target person 6 in the present embodiment. As in the first embodiment, the cooperating behavior target person 6 is someone the user 5 is familiar with and someone with whom the user 5 is less likely to feel stressed when performing a cooperative behavior together.

[0075] The detection device 4a acquires personal behavior data of the collaborative activity target person 6 and transmits the personal behavior data to the control system 1a. As in the first embodiment, the personal behavior data may include psychological or internal information, such as biometric information or emotions, of the collaborative activity target person 6. The detection device 4a is similar to the detection device 4 in the first embodiment, but acquires personal behavior data from the collaborative activity target person 6. The detection device 4a may be a wearable device that can be worn by the collaborative activity target person 6, a portable terminal that can be carried by the collaborative activity target person 6, a device installed so as to detect the actions of the collaborative activity target person 6, a device that detects actions in a virtual space such as the Metaverse, or a combination of these, or other types of data. In other words, the personal behavior data may include, for example, at least one of data acquired by a wearable device and data recording a person's actions in a virtual space.

[0076] The control system 1a is similar to the control system 1 of the first embodiment, except that it includes a control model generation unit 2a instead of the control model generation unit 2. The control model generation unit 2a does not include the correction information storage unit 25, includes a learning unit 22a instead of the learning unit 22, and the data acquisition unit 24 acquires data from a detection device 4a instead of from the detection device 4, but other than these, it is similar to the control model generation unit 2 of the first embodiment. In this embodiment as well, the control model generation unit 2a and the robot control unit 3 may be provided as separate devices.

[0077] Next, the operation of the control model generation unit 2a of this embodiment will be described. FIG. 12 is a flowchart showing an example of a processing procedure in the control model generation unit 2a of this embodiment. The control model generation unit 2a acquires personal behavior data of the cooperating behavior target person 6, including cooperative behavior data acquired during cooperative behavior between the user 5 and the cooperating behavior target person 6 (step S21). In detail, the data acquisition unit 24 acquires the personal behavior data of the cooperating behavior target person 6 by receiving the personal behavior data from the detection device 4a. Note that, as in the first embodiment, the data acquisition unit 24 may acquire the personal behavior data from a recording medium. Alternatively, video or the like that is the source of the personal behavior data may be acquired by the detection device 4a and subjected to extraction processing.

[0078] The control model generating unit 2a stores the individual behavior data of the cooperating behavior target person 6 (step S22). Specifically, the data acquiring unit 24 stores the received individual behavior data in the data storage unit 23.

[0079] The control model generating unit 2a generates a control model using the accumulated individual behavior data of the cooperating behavior target 6 (step S23). In detail, the learning unit 22a extracts features using the individual behavior data of the cooperating behavior target 6 stored in the data storage unit 23, and generates a control model based on the features.

[0080] In this embodiment, control parameters in the control model are set so that the robot 7 performs actions indicated by the feature amounts. This generates a control model for causing the robot 7 to perform actions similar to actions that reflect the individuality of the collaborative behavior target person 6. The feature amounts can be the same as those in the first embodiment. However, in this embodiment, the feature amounts may also include information about dialects, speaking habits (including catchphrases), and topics (genres frequently spoken). The information about dialects includes, for example, whether or not a dialect is spoken, and if so, what type (region) of dialect it is. Dialect identification may be performed, for example, by storing a dialect dictionary for each dialect type in advance and using the dictionary, or by other methods. Speaking habits include, for example, frequently using specific words at the end of words, frequently uttering specific words, intonation (such as raising the pitch at the end of words), pitch of voice, and speed of speech, but may also be other factors. The learning unit 22a extracts these speaking habits by, for example, performing a voice recognition process on the voice data obtained as the individual behavior data of the collaborative action target person 6.

[0081] Step S24 after step S23 is the same as in the first embodiment, where the learning unit 22a outputs the generated control model to the robot control unit 3. The output control model is stored in the control model storage unit 34 of the robot control unit 3. The operation of the robot control unit 3 is the same as in the first embodiment. In this embodiment, a control model is generated based on the individual behavior data of the collaborative operation target 6 so as to perform an operation that reflects the individuality of the collaborative operation target 6. As a result, in collaborative operation with the user 5, the robot 7 can perform an operation that reflects the individuality of the collaborative operation target 6 that the user 5 is familiar with, and stress on the user 5 caused by at least one of the behavior and speaking style of the robot 7 during collaborative operation can be reduced.

[0082] Next, an example of a cooperative operation performed using the cooperative operation system 100a of this embodiment will be described. As an example, the robot 7 is a communication robot, and the cooperative operation is a conversation. Mr. A and Mr. B are a married couple, and Mr. B is accustomed to conversing with Mr. A and does not feel stressed when conversing with Mr. A. Mr. A is scheduled to be transferred overseas and will converse with the robot 7 during his transfer. In this case, the user 5 of the robot 7 is Mr. B, and Mr. A is set as the collaborative operation target 6, and personal behavioral data of Mr. A is acquired. The control model generation unit 2a generates a control model based on the accumulated personal behavioral data of Mr. A. For example, the basic control model includes a conversation model and a voice model, and the conversation model and voice model are modified based on the personal behavioral data to have characteristics similar to those of Mr. A. As a result, a control model is generated that reflects, for example, Mr. A's catchphrases, intonation, dialect, way of responding, and topic content.

[0083] When user B, who is user 5, converses with robot 7 as a cooperative operation while user A is away from home, robot 7 is controlled using a control model based on the above-described personal behavior data of user A. This allows robot 7 to converse in a way that reflects the personality of user A, thereby reducing stress for user B, who is user 5. Note that the above-described cooperative operation performed using cooperative operation system 100a is an example, and cooperative operation performed using cooperative operation system 100a is not limited to the above-described example.

[0084] The control system 1a of this embodiment is realized by a computer system, similar to the control system 1 of embodiment 1. The control system 1a of this embodiment may also be realized by multiple computer systems, for example, by a cloud system. As described above, the control model generation unit 2a and the robot control unit 3 may each be configured as separate devices, and in this case, each separate device may also be realized by multiple computer systems.

[0085] Note that the cooperative action is not limited to two people and may be performed by three or more people. For example, in a cooperative action performed by three people, two robots 7 may be used, or the robot 7 and the cooperating action target person 6 may cooperate with the user 5. In this case, for example, if user B, who is user 5, does not feel stressed when cooperating with users A and C, control models may be generated based on the personal behavior data of users A and C, and the two robots 7 may be controlled based on the respective control models. Furthermore, when the robot 7 and the cooperating action target person 6 cooperate with the user 5, if the cooperating action target person 6 is A, the robot 7 may be controlled based on the control model corresponding to user C, and if the cooperating action target person 6 is C, the robot 7 may be controlled based on the control model corresponding to user A.

[0086] As described above, the collaborative operation system 100a of this embodiment generates a control model for controlling the robot 7 using personal behavior data of the collaborative operation target person 6, including collaborative operation data acquired when the user 5 and the collaborative operation target person 6 performed collaborative operations, before the collaborative operation between the robot 7 and the user 5 is performed. The robot 7 can perform actions that are the same as or similar to the actions of the collaborative operation target person 6 from the start of the collaborative operation, thereby reducing stress on the user 5 caused by at least one of the behavior and speaking style of the robot 7 during the collaborative operation.

[0087] Third Embodiment Fig. 13 is a diagram showing an example of the configuration of a collaborative system according to a third embodiment. A collaborative system 100b according to this embodiment is similar to the collaborative system 100a according to the second embodiment, except that it includes a control system 1b instead of the control system 1a. Components having the same functions as those in the second embodiment are given the same reference numerals as those in the second embodiment, and redundant explanations will be omitted. Differences from the second embodiment will be mainly explained below.

[0088] The control system 1b is similar to the control system 1a of embodiment 2, except that it includes a control model generation unit 2b instead of the control model generation unit 2a. The control model generation unit 2b additionally includes a behavior result acquisition unit 26 and a learning unit 22b instead of the learning unit 22a, but other than that, it is similar to the control model generation unit 2a of embodiment 2. In this embodiment as well, the control model generation unit 2b and the robot control unit 3 may be provided as separate devices.

[0089] Next, the operation of the control model generation unit 2b of this embodiment will be described. The process of generating a control model in the control model generation unit 2b is the same as the process in embodiment 2 described with reference to FIG. 12. In this embodiment, after the control model is generated, the control model generation unit 2b updates the control model based on the behavioral result, which is the result of the cooperative action performed by the robot 7 and the user 5, and the control model corresponding to the behavioral result. As described in embodiment 2, the control model is generated so as to reduce stress on the user 5. However, in this embodiment, by updating the control model using the behavioral result, the behavior of the robot 7 can be made more suitable for the user 5.

[0090] 14 is a flowchart showing an example of a control model update process procedure in the control model generation unit 2b of this embodiment. First, the control model generation unit 2b acquires an action result corresponding to the control model (step S31). In detail, the action result acquisition unit 26 acquires an action result corresponding to a cooperative action (cooperative action between the robot 7 and the user 5) performed by control based on the control model stored in the robot control unit 3, and outputs the acquired action result to the learning unit 22b.

[0091] The behavioral result indicates, for example, whether a positive result or a negative result was obtained. The behavioral result is determined, for example, by the user 5 and input to the control model generation unit 2b. In this case, the behavioral result acquisition unit 26 has a function of receiving input from the user 5. Alternatively, the user 5 may input the behavioral result to another device, such as a user terminal (not shown), and the other device may transmit the behavioral result to the control model generation unit 2b. In this case, the behavioral result acquisition unit 26 has a communication function for receiving the behavioral result. For example, when the user 5 feels that the cooperative movement with the robot 7 was comfortable, stress-free, or efficient, the behavioral result is determined to be a positive result. When the user 5 feels that the cooperative movement with the robot 7 was stressful, unpleasant, or inefficient, the behavioral result is determined to be a negative result.

[0092] Furthermore, for example, if the collaborative behavior is a task, the behavior result may be determined by another means. For example, if the collaborative behavior is a task with a set procedure, the task time may be measured, and if the measurement result is below a threshold, a person other than the user 5 may determine that the task was performed efficiently and give the behavior result a positive result. If the measurement result exceeds the threshold, the person may determine that the task was not performed efficiently and give the behavior result a negative result. In this case, the behavior result may be input to the control model generation unit 2b or transmitted from another device. Since it can be estimated that the user 5 is less stressed when the task is performed efficiently, the behavior result may be determined based on the measurement result of the task time. Alternatively, the control model generation unit 2b may make a determination based on the measurement result described above. For example, the behavior result acquisition unit 26 may receive measurement results from a device that measures the task time and determine the behavior result using the received measurement results. The method for determining the behavior result is not limited to the above example.

[0093] The control model generating unit 2b determines whether the behavior result is a negative result (step S32). Specifically, the learning unit 22b determines whether the behavior result received from the behavior result acquiring unit 26 is a negative result.

[0094] If the behavior result is not a negative result (No in step S32), that is, if the behavior result is a positive result, the control model generation unit 2b ends the control model update process.

[0095] If the behavior result is negative (Yes in step S32), the control model generation unit 2b updates the control model (step S33) and repeats the processing from step S31. In step S33, in detail, the learning unit 22b updates the control model and outputs the updated control model to the robot control unit 3. This updates the control model stored in the control model storage unit 34 of the robot control unit 3. The learning unit 22b updates the control model, for example, by changing some of the control parameters in the control model. The method of changing the control parameters may be determined in advance or may be specified by the user 5. For example, when updating a control parameter that changes the position of the robot 7, a rule for changing the position of the robot 7 may be determined in advance, or the user 5 may specify the direction and amount of change in the position of the robot 7.

[0096] As described above, if the behavior result is negative, the control model is updated, control is performed using the updated control model, and the process is repeated from step S31. If the behavior result is negative, repeated changes to the control parameters will result in a positive behavior result.

[0097] In the above example, when the behavioral result was positive, the control model was not changed and the current control model was used as the updated control model. However, this is not limited to this. The control model may be updated by changing the current control parameter to a control parameter that is estimated to be better. The control parameter that is estimated to be better is, for example, a control parameter that is changed in the opposite direction to the control parameter that was set when the behavioral result was previously negative. For example, if the conversation speed was a first speed, the behavioral result was negative, and changing the conversation speed to a second speed slower than the first speed resulted in a positive behavioral result, the control model may be updated to change the conversation speed to a third speed slower than the second speed. Then, when the behavioral result is obtained again and the behavioral result is negative, the control model is updated to return the conversation speed to the second speed.

[0098] Alternatively, the control model update process is not limited to the procedure shown in FIG. 14 . Alternatively, the control parameters may be sequentially changed, behavioral results corresponding to the respective control parameter values ​​may be obtained, and data sets of the control parameter values ​​and the corresponding behavioral results may be stored. The control model may be updated using the multiple data sets. For example, data sets in which the behavioral results are positive may be extracted, one of the extracted data sets may be selected, and the control model may be updated using the control parameters in the selected data set. Alternatively, the control model may be updated by using multiple data sets each containing a behavioral result and a corresponding control parameter as correct data to determine control parameters that improve the behavioral result through machine learning. For example, the learning unit 22b generates a trained model by supervised learning as described in the first embodiment using multiple data sets each containing a behavioral result and a corresponding control parameter as correct data. Then, during inference, i.e., when updating the control model, the learning unit 22b can infer control parameters that result in a positive behavioral result by inputting a value corresponding to the positive behavioral result as the behavioral result.

[0099] Furthermore, the behavioral result is not limited to the binary values ​​of positive and negative, and may be expressed using three or more numerical levels. For example, the behavioral result may be scored from 0 to 5, and a behavioral result of 5 may be defined as the least stressful for the user 5, and a behavioral result of 0 may be defined as the most stressful for the user 5. Note that the definition of the scores is not limited to this example. When the behavioral result is expressed using three or more numerical levels, in step S32 of the process shown in FIG. 14, the learning unit 22b may determine whether the numerical value indicates that the behavioral result is the most positive. Furthermore, when updating the control model using the above-mentioned multiple data sets, the learning unit 22b may select a data set that indicates that the behavioral result is the most positive.

[0100] Furthermore, in the above example, the behavior result is the result of evaluating the entire control model. However, this is not limiting and the behavior result may be the result of evaluating a time-series motion of the robot 7 in a separate section. For example, control instructions to the robot 7 may be recorded in an operation history storage unit (not shown), and the behavior result may be determined, for example, at regular intervals or for each section of the robot 7's motion. In this case, the operation history storage unit may be provided in the robot control unit 3, the control model generation unit 2b, or outside the control system 1b. In this case, the behavior result acquisition unit 26 reads and acquires the behavior result and the control instruction for the period corresponding to the behavior result from the operation history storage unit, and outputs the behavior result and the corresponding control instruction to the learning unit 22b. As a result, the learning unit 22b can obtain the behavior result for each motion of the robot 7 corresponding to the control instruction given in time series.

[0101] For example, when the collaborative behavior is a conversation, it is assumed that a result is obtained based on the individual behavior data of the collaborative behavior target person 6 that the collaborative behavior target person 6 often provides topics about the first genre and the second genre, and a control model is generated based on this result. When generating the control model, it is assumed that the frequency of providing topics about the first genre and the second genre is set to be approximately the same. In the conversation, the time series includes a period in which the conversation in the first genre takes place and a period in which the conversation in the second genre takes place. These periods are distinguished based on a control instruction to the robot 7, and the behavior result acquisition unit 26 acquires the respective behavior results. For example, if the behavior result during the period in which the conversation in the first genre took place is positive and the behavior result during the period in which the conversation in the second genre took place is negative, the control model is updated to increase the frequency of providing topics about the first genre and decrease the frequency of providing topics about the second genre.

[0102] The control model update process is not limited to the above example, and may be any method in which the learning unit 22b updates the control model based on the behavioral results so that the control model becomes more suitable for the user 5.

[0103] The control system 1b of this embodiment is realized by a computer system, similar to the control system 1a of embodiment 2. The control system 1b of this embodiment may also be realized by multiple computer systems, for example, by a cloud system. As described above, the control model generation unit 2b and the robot control unit 3 may each be configured as separate devices, and in this case, each separate device may also be realized by multiple computer systems.

[0104] As described above, the collaborative operation system 100b of this embodiment performs the operations described in the second embodiment, and also updates the control model based on the behavioral results that are the results of the collaborative operation between the robot 7 and the user 5. Therefore, the same effects as those of the second embodiment can be achieved, and the robot 7 can be made to perform operations that are more suitable for the user 5.

[0105] In the above example, a control model update function is added to the collaborative operation system 100a of embodiment 2, but this is not limiting, and a control model update function may be added to the collaborative operation system 100 of embodiment 1. For example, the control model may be updated in the same manner as in the above example by adding an action result acquisition unit 26 to the control model generation unit 2 of the collaborative operation system 100 and providing the learning unit 22 with a control model update function similar to the learning unit 22b.

[0106] Fourth Embodiment Fig. 15 is a diagram showing an example of the configuration of a cooperative operation system according to a fourth embodiment. A cooperative operation system 100c according to this embodiment is similar to the cooperative operation system 100b according to the third embodiment, except that it includes a control system 1c instead of the control system 1b. Components having the same functions as those in the third embodiment are given the same reference numerals as those in the third embodiment, and redundant explanations will be omitted. Differences from the third embodiment will be mainly explained below.

[0107] The control system 1c is similar to the control system 1b of embodiment 3, except that it includes a control model generation unit 2c instead of the control model generation unit 2b. The control model generation unit 2c additionally includes a model selection receiving unit 27 and a basic model storage unit 21a instead of the basic model storage unit 21, but is otherwise similar to the control model generation unit 2b of embodiment 3. In this embodiment as well, the control model generation unit 2c and the robot control unit 3 may be provided as separate devices.

[0108] The basic model storage unit 21a stores a plurality of basic control models, i.e., a plurality of types of basic control models, in advance. These plurality of types of basic control models can also be considered typical patterns that characterize the collaborative behavior target 6. For example, in the case of a basic control model that imitates a human, the typical patterns that characterize the collaborative behavior target 6 are patterns that correspond to speech and actions (behaviors) based on typical human personality traits, such as impatient, easygoing, or meticulous. In other words, for example, if the collaborative behavior includes conversation, three basic control models, namely, a moderate model, a steady model, and a leading model, are stored in the basic model storage unit 21a. The moderate model, the steady model, and the leading model each differ in at least one of, for example, the content of the conversation, the speed of the conversation, the frequency of speaking, and the genre of the topic provided.

[0109] Furthermore, for example, in the case of a basic control model to be applied to industrial machinery, basic control models that behave in a manner that is based on typical work content that requires consideration for the people working together, such as collaborative design work using engineering tools, collaborative precision work such as medical procedures (surgery), or collaborative long-term work such as installing large equipment, are stored in the basic model storage unit 21a.

[0110] Furthermore, the basic control models, i.e., typical patterns, may be classified based on typical characteristics of other cooperatively operating targets. In this way, the multiple control models correspond to different personalities (personalities of behavior). Note that the multiple basic control models are not limited to this example, and the number of basic control models is not limited to three.

[0111] The model selection receiving unit 27 receives from the user 5 a selection result indicating the basic control model selected by the user 5 from among multiple basic control models. For example, the model selection receiving unit 27 may receive an input of the selection result from the user 5. Alternatively, the user 5 may input the selection result to another device, such as a user terminal (not shown), which then transmits the selection result to the control model generation unit 2c, and the model selection receiving unit 27 receives the selection result, thereby receiving the selection result of the basic control model. The user 5 selects a basic control model from among multiple basic control models based on preferences and compatibility. For example, the user 5 may select a basic control model that matches the collaborative behavior target person 6 from among the multiple basic control models. For example, if three basic control models, namely, a moderate model, a steady model, and a traction model, are stored in the basic model storage unit 21a and the collaborative behavior target person 6 that the user 5 is familiar with has a moderate personality, the user 5 may select the moderate model. Note that, instead of the user 5 making the selection, the control system 1c or an operator of the control system 1c may select a basic control model that is suitable for the collaborative behavior target person 6.

[0112] The model selection receiving unit 27 reads out a basic control model corresponding to the received selection result from the basic model storage unit 21a, and outputs the read-out basic control model to the learning unit 22b. The learning unit 22b generates a control model using the basic control model received from the model selection receiving unit 27, i.e., the basic control model indicated by the selection result, and the individual behavior data, as in the third embodiment, and outputs the generated control model to the robot control unit 3. Also, the learning unit 22b updates the control model using the behavior result, as in the third embodiment. Operation of this embodiment other than that described above is the same as that of the third embodiment.

[0113] The control system 1c of this embodiment is realized by a computer system, similar to the control system 1b of embodiment 3. The control system 1c of this embodiment may also be realized by multiple computer systems, for example, by a cloud system. As described above, the control model generation unit 2c and the robot control unit 3 may each be configured as separate devices, and in this case, each separate device may also be realized by multiple computer systems.

[0114] As described above, the collaborative operation system 100c of this embodiment generates a control model using a basic control model selected from a plurality of basic control models with different movement characteristics and personal behavior data. Furthermore, the collaborative operation system 100c of this embodiment updates the control model based on the behavioral results that are the results of collaborative operation between the robot 7 and the user 5. Therefore, the same effects as those of the third embodiment can be achieved, and the robot 7 can be made to perform actions that are more in line with the preferences and compatibility of the user 5.

[0115] In the above example, the collaborative system 100b of the third embodiment has been added with a function for generating a control model using a basic control model selected from a plurality of basic control models. However, this is not limited to this. A function for generating a control model using a basic control model selected from a plurality of basic control models may be added to the collaborative system 100 of the first embodiment. For example, by adding a model selection receiving unit 27 to the control model generation unit 2 of the collaborative system 100 and providing a basic model storage unit 21a instead of the basic model storage unit 21, a control model may be generated using a basic control model selected from a plurality of basic control models, as in the above example. Furthermore, a function for generating a control model using a basic control model selected from a plurality of basic control models may be added to the collaborative system 100a of the second embodiment. For example, by adding a model selection receiving unit 27 to the control model generation unit 2a of the collaborative system 100 and providing a basic model storage unit 21a instead of the basic model storage unit 21, a control model may be generated using a basic control model selected from a plurality of basic control models, as in the above example.

[0116] Fifth Embodiment Fig. 16 is a diagram showing an example of the configuration of a collaborative operation system according to a fifth embodiment. A collaborative operation system 100d of this embodiment includes a control system 1d, a detection device 4, and a situation detection device 8a. The control system 1d generates a virtual space such as a metaverse and transmits virtual space information to a terminal device 94 so that a user 5 can perceive the virtual space. The terminal device 94 outputs an image of the virtual space to an image presentation device 95 and outputs audio of the virtual space to an audio presentation device 96 based on the virtual space information received from the control system 1d. The user 5 collaborates with a virtual character 902 via his or her own avatar 901 in the virtual space.

[0117] In the first to fourth embodiments, a robot 7 was used as an example of a humanoid that cooperates with a user 5. In the present embodiment, however, an example will be described in which the humanoid that cooperates with a user 5 is a virtual character 902 in a virtual space. In this embodiment, when the user 5 cooperates with the virtual character 902, the virtual character 902 is controlled using a control model that is generated based on personal behavior data of the user 5 that was acquired in advance while the user 5 was cooperatively operating with a cooperative operation target person 6, as in the first embodiment. Components that have the same functions as in the first embodiment are assigned the same reference numerals as in the first embodiment, and redundant explanations will be omitted. Below, differences from the first embodiment will be mainly explained.

[0118] 16 shows an example in which the terminal device 94 transmits video and audio to the video presentation device 95 and audio presentation device 96 via wireless communication, but one or more of the video and audio may be transmitted via wired communication. The terminal device 94 may be included in the control system 1d, or the terminal device 94, the video presentation device 95, and the audio presentation device 96 may be included in the control system 1d.

[0119] 16 illustrates the use of a video presentation device 95 and an audio presentation device 96 as means for the user 5 to perceive the virtual space. However, means capable of perceiving one or more of haptics, smell, and taste may also be used. In addition to stress, haptics may also include information perceived by the skin, such as temperature. While FIG. 16 illustrates the use of a video presentation device 95 and an audio presentation device 96, none of these may be used. While FIG. 16 illustrates an example in which VR goggles or a head-mounted display is used as the video presentation device 95 and headphones are used as the audio presentation device 96, this is not a limitation. For example, the video presentation device 95 may be a display or a monitor, and the audio presentation device 96 may be a speaker. Specific examples of the video presentation device 95 and the audio presentation device 96 are not limited to the example illustrated in FIG. 16. Furthermore, two or more of the terminal device 94, the video presentation device 95, and the audio presentation device 96 may be integrated. For example, the display of the terminal device 94 may be used as the video presentation device 95. Also, for example, a head-mounted display having the functions of both the terminal device 94 and the image presentation device 95 may be used, or a head-mounted display with headphones may be used.

[0120] The situation detection device 8a acquires the situation of the user 5 in the cooperative action between the user 5 and the virtual character 902. For example, the situation detection device 8a detects the voice, movement, etc. of the user 5 and transmits the detection results to the terminal device 94. The terminal device 94 transmits the detection results received from the situation detection device 8a to the control system 1d. There may be multiple situation detection devices 8a. Also, for example, the audio presentation device 96 may be a headset, so that the audio presentation device 96 and the situation detection device 8a are integrated. Also, the terminal device 94 may be equipped with the situation detection device 8a. Also, the situation detection device 8a may be worn by the user 5, or may be provided in the vicinity of the user 5, such as a camera that takes a picture of the user 5 from outside.

[0121] The control system 1d includes a control model generation unit 2 and a virtual space control unit 9 similar to those in the first embodiment. The control model generation unit 2 and the virtual space control unit 9 may each be provided as separate devices. The configuration and operation of the control model generation unit 2 are similar to those in the first embodiment, but the control model generated by the control model generation unit 2 is a control model for controlling the movement of the virtual character 902, and the basic control model stored in the basic model storage unit 21 is also a basic control model for controlling the movement of the virtual character 902.

[0122] The virtual space control unit 9 includes a transmission / reception unit 91, a virtual space generation unit 92, and a virtual character control unit 93. The transmission / reception unit 91 communicates with a terminal device 94 and exchanges information with the terminal device 94. The transmission / reception unit 91, for example, acquires situation data indicating the situation of the user 5 in the collaborative action from the terminal device 94, and outputs the acquired situation data to the virtual space generation unit 92 and the control instruction generation unit 33. The transmission / reception unit 91 may receive the situation data from the situation detection device 8a. The transmission / reception unit 91 also transmits, for example, virtual space information (described later) received from the virtual space generation unit 92 to the terminal device 94.

[0123] The virtual space generation unit 92 generates a virtual space, generates virtual space information for allowing the user 5 to perceive the generated virtual space, and outputs the generated virtual space information to the transmission / reception unit 91. The virtual space information includes data representing images (video data) and data representing audio (audio data). Note that, depending on the collaborative action and the content of the virtual space, the virtual space information may not include data representing audio. Furthermore, the virtual space generation unit 92 may include at least one of information detectable by the user 5 through haptics, smell, and taste. Furthermore, the virtual space generation unit 92 generates virtual space information so that the avatar 901 of the user 5 in the virtual space performs an action based on the situation data received from the transmission / reception unit 91. Furthermore, upon receiving a control instruction (described later) from the virtual character control unit 93, the virtual space generation unit 92 generates virtual space information so that the virtual character 902 performs an action based on the control instruction.

[0124] The virtual character control unit 93 is an example of a movement control unit (movement control device) that controls a humanoid. The virtual character control unit 93 includes a control instruction generation unit 33 and a control model storage unit 34. As in the first embodiment, the control model storage unit 34 stores the control model generated by the control model generation unit 2. As described above, this control model is a control model for controlling the movement of the virtual character 902. The control instruction generation unit 33 generates control instructions for controlling the movement of the virtual character 902 using situation data indicating the situation of the user 5 received from the transmission / reception unit 91 and the control model stored in the control model storage unit 34, and outputs the generated control instructions to the virtual space generation unit 92.

[0125] In this embodiment, the control target is a virtual character 902 instead of a robot 7, but as in the first embodiment, before a cooperative action is performed between the virtual character 902 and the user 5, a control model is generated using personal behavior data of the user 5, including cooperative action data acquired when the user 5 and the cooperative action target 6 performed a cooperative action. Therefore, the virtual character 902 can perform actions that are the same as or similar to the actions of the cooperative action target 6 from the start of the cooperative action, and stress on the user 5 caused by at least one of the actions and speaking style of the virtual character 902 during the cooperative action can be reduced.

[0126] 16, the virtual character control unit 93 is provided within the virtual space control unit 9. However, this is not limiting. For example, the virtual space control unit 9 may be provided as a separate virtual space control device outside the control system 1d. In this case, the transmission / reception unit 91 and the virtual character control unit 93 are provided within the control system 1d, and the virtual space generation unit 92 is provided in the virtual space control device. The virtual space control device also includes a transmission / reception unit 91. Control instructions generated by the virtual character control unit 93 are transmitted to the virtual space control device via the transmission / reception unit 91 of the control system 1d, and the virtual space generation unit 92 of the virtual space control device receives the control instructions via the transmission / reception unit 91 of the virtual space control device. The virtual space generation unit 92 of the virtual space control device transmits the generated virtual space information to the terminal device 94 via the transmission / reception unit 91 of the virtual space control device. The transmission / reception unit 91 may also be provided within the virtual character control unit 93. In this case, the virtual character control unit 93 and the control model generation unit 2 may also be provided as separate devices.

[0127] The control system 1d of this embodiment is realized by a computer system, similar to the control system 1 of embodiment 1. The control system 1d of this embodiment may also be realized by multiple computer systems, for example, by a cloud system. As described above, the control model generation unit 2 and the virtual space control unit 9 may each be configured as separate devices, and in this case, each separate device may also be realized by multiple computer systems.

[0128] In the above-described example, the control model generation unit 2 of the first embodiment generates a control model for controlling the virtual character 902, and the virtual character control unit 93 controls the virtual character 902 using the generated control model. However, when controlling the virtual character 902, an action result acquisition unit 26 may be provided as in the third embodiment, and the control model may be updated using the action result, or a basic control model to be used may be selected from a plurality of basic control models for controlling the virtual character 902 using the model selection reception unit 27 as in the fourth embodiment. Furthermore, both updating of the control model using the action result and selection of the basic control model to be used from a plurality of basic control models may be performed.

[0129] Sixth Embodiment Fig. 17 is a diagram showing an example of the configuration of a collaborative operation system according to a sixth embodiment. A collaborative operation system 100e according to this embodiment includes a control system 1e, a detection device 4a, and a situation detection device 8a. As in the fifth embodiment, the control system 1e generates a virtual space and transmits virtual space information to a terminal device 94 so that a user 5 can perceive the virtual space. The situation detection device 8a, the terminal device 94, the video presentation device 95, and the audio presentation device 96 are the same as those in the fifth embodiment. Note that the terminal device 94 may be included in the control system 1e, or the terminal device 94, the video presentation device 95, and the audio presentation device 96 may be included in the control system 1e.

[0130] The control system 1e includes a control model generation unit 2a similar to that of the second embodiment and a virtual space control unit 9 similar to that of the fifth embodiment. The control model generation unit 2a and the virtual space control unit 9 may be provided as separate devices. The configuration and operation of the control model generation unit 2a are similar to those of the second embodiment, but the control model generated by the control model generation unit 2a is a control model for controlling the movement of the virtual character 902, and the basic control model stored in the basic model storage unit 21 is also a basic control model for controlling the movement of the virtual character 902. Components having the same functions as those of the second or fifth embodiment are assigned the same reference numerals as those of the second or fifth embodiment, and redundant description will be omitted. The following description will mainly focus on differences from the second or fifth embodiment.

[0131] In this embodiment, similarly to the second embodiment, the control model generation unit 2a generates a control model based on the personal behavior data of the cooperating action target person 6 acquired by the detection device 4a while the cooperating action target person 6 is performing a cooperative action with the user 5. This control model is a control model for controlling the action of the virtual character 902. The virtual character control unit 93 of the virtual space control unit 9 uses the control model generated by the control model generation unit 2a to control the virtual character 902, similarly to the fifth embodiment.

[0132] Furthermore, as described in the fifth embodiment, for example, the virtual space control unit 9 may be provided as an individual virtual space control device outside the control system 1e.

[0133] In this embodiment, the control target is a virtual character 902 instead of a robot 7, but as in the second embodiment, before a cooperative action is performed between the virtual character 902 and the user 5, a control model is generated using personal behavior data of the cooperating action target person 6, including cooperative action data acquired when the user 5 and the cooperating action target person 6 performed a cooperative action. Therefore, the virtual character 902 can perform actions that are the same as or similar to the actions of the cooperating action target person 6 from the start of the cooperative action, and stress on the user 5 caused by at least one of the actions and speaking style of the virtual character 902 during the cooperative action can be reduced.

[0134] The control system 1e of this embodiment is realized by a computer system, similar to the control system 1a of embodiment 2. The control system 1e of this embodiment may also be realized by multiple computer systems, for example, by a cloud system. As described above, the control model generation unit 2a and the virtual space control unit 9 may each be configured as separate devices, and in this case, each separate device may also be realized by multiple computer systems.

[0135] In the above example, the virtual characters 902 perform cooperative actions and the personal behavior data includes cooperative action data. However, the applied actions are not limited to cooperative actions as long as a control model of the virtual characters 902 is generated using personal behavior data that indicates the personality of a specific person. That is, personal behavior data, which is data related to the actions of a specific person, is stored in the data storage unit 23, and the learning unit 22a uses the personal behavior data to generate a control model of the virtual characters 902 in the virtual space that reflects the personality of the specific person. For example, the user 5 sets a specific person, and the control model of the virtual character 902 is generated using the personal behavior data of the specific person. This generates a control model that reflects the personality of the specific person according to the user 5's wishes. This reduces stress for the user 5 when the user 5 converses with the virtual characters 902 or watches the virtual characters 902 speak and act. The method of setting a specific person is not limited to this example. By generating a control model for the virtual character 902 based on the personal behavioral data of a specific person, it is possible to generate a control model that reflects at least one of the behavior and speaking style of the specific person, and it is possible to reflect the individual characteristics of at least one of the behavior and speaking style of the specific person in the virtual character 902.

[0136] In the above-described example, the control model generation unit 2a of the second embodiment generates a control model for controlling the virtual character 902, and the virtual character control unit 93 controls the virtual character 902 using the generated control model. However, when controlling the virtual character 902, an action result acquisition unit 26 may be provided and the control model may be updated using the action result, as in the third embodiment, or a basic control model to be used may be selected from a plurality of basic control models for controlling the virtual character 902 using the model selection receiving unit 27, as in the fourth embodiment. Furthermore, both the update of the control model using the action result and the selection of the basic control model to be used from a plurality of basic control models may be performed.

[0137] The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, or different embodiments may be combined with each other. It is also possible to omit or modify parts of the configurations as long as they do not deviate from the gist of the invention.

[0138] 1, 1a, 1b, 1c, 1d, 1e Control system, 2, 2a, 2b, 2c Control model generation unit, 3 Robot control unit, 4, 4a Detection device, 5 User, 6 Collaborative operation target person, 7 Robot, 8, 8a Situation detection device, 9 Virtual space control unit, 21, 21a Basic model storage unit, 22, 22a, 22b Learning unit, 23 Data storage unit, 24 Data acquisition unit, 25 Correction information storage unit, 26 Action result acquisition unit, 27 Model selection reception unit, 31 Instruction transmission unit, 32 Situation acquisition unit, 33 Control instruction generation unit, 34 Control model storage unit, 91 Transmission / reception unit, 92 Virtual space generation unit, 93 Virtual character control unit, 94 Terminal device, 95 Video presentation device, 96 Audio presentation device, 100, 100a, 100b, 100c, 100d, 100e Collaborative operation system.

Claims

1. a data storage unit for storing individual behavior data which is data related to the behavior of a person who has performed a cooperative behavior; a learning unit that uses the individual behavior data stored in the data storage unit to generate a control model of the humanoid for causing the humanoid to perform the collaborative action with a user, the control model reflecting the individuality of the person; and Equipped with A control model generating device, wherein the personal behavior data includes data acquired when the user is performing the cooperative action with a cooperative action target with which the user is familiar.

2. 2. The control model generating device according to claim 1, wherein the person is the user.

3. 2. The control model generating device according to claim 1, wherein the person is a target of the cooperative action.

4. 4. The control model generating device according to claim 1, wherein the humanoid is a robot.

5. 4. The control model generating device according to claim 1, wherein the humanoid is a virtual character in a virtual space.

6. 4. The control model generating device according to claim 1, wherein the learning unit updates the control model based on an action result that is a result of the collaborative action performed by the humanoid and the user and the control model that corresponds to the action result.

7. The control model generating device described in Claim 5, characterized in that the learning unit updates the control model based on an action result that is the result of the collaborative action performed by the humanoid and the user and the control model corresponding to the action result.

8. 4. The control model generating device according to claim 1, wherein the learning unit generates the control model by using a predetermined basic control model and the individual behavior data.

9. The control model generating device described in Claim 5, characterized in that the learning unit generates the control model using a predetermined basic control model and the personal behavior data.

10. a selection result receiving unit that receives a selection result indicating the basic control model selected by the user from among the plurality of basic control models; Equipped with 9. The control model generating device according to claim 8, wherein the learning unit generates the control model by using the basic control model indicated by the selection result and the individual behavior data.

11. The learning unit extracts a feature amount indicating a personality of the person from the personal behavior data, 4. The control model generating device according to claim 1, wherein the personality includes at least one of a catchphrase, an intonation, a dialect, and a habit of a manner of movement.

12. The learning unit extracts a feature quantity indicating a personality of the person from the personal behavior data, 6. The control model generating device according to claim 5, wherein the personality includes at least one of a speech pattern, an intonation, a dialect, and a habit of a movement.

13. The control model generating device according to any one of claims 1 to 3, characterized in that the personal behavior data includes at least one of data acquired by a wearable terminal and data recording the movements of the person in a virtual space.

14. The control model generating device as described in Claim 5, characterized in that the personal behavior data includes at least one of data acquired by a wearable terminal and data recording the person's movements in a virtual space.

15. A data storage unit that stores personal behavior data, which is data related to a person's movements; a learning unit that uses the individual behavior data stored in the data storage unit to generate a control model of a virtual character in a virtual space, the control model reflecting the individuality of the person; A control model generating device comprising:

16. A control model storage unit; a control instruction generation unit that generates control instructions for a humanoid; Equipped with the control model storage unit stores a control model of the humanoid for causing the humanoid to perform the cooperative action with a user, the control model being generated using personal behavior data, which is data related to the actions of a person who has performed a cooperative action, and the control model reflects the individuality of the person; the control instruction generation unit generates control instructions for the humanoid using the control model; A robot control device, characterized in that the personal behavior data includes data acquired when the user is performing the cooperative action with a cooperative action target with which the user is familiar.

17. A control model generation unit; A motion control unit that controls the humanoid; Equipped with The control model generation unit a data storage unit for storing individual behavior data which is data related to the behavior of a person who has performed a cooperative behavior; a learning unit that uses the individual behavior data stored in the data storage unit to generate a control model of the humanoid for causing the humanoid to perform the collaborative action with a user, the control model reflecting the individuality of the person; and Equipped with the motion control unit controls the humanoid using the control model; A control system, characterized in that the personal behavior data includes data acquired when the user is performing the collaborative activity with a collaborative activity partner with whom the user is familiar.

18. A control model generation method in a control model generation device, comprising: a step of storing personal behavior data, which is data on the behavior of a person who has performed a cooperative behavior, by the control model generating device; a step of generating, by the control model generation device, a control model of the humanoid for performing the collaborative action with a user, the control model reflecting the personality of the person, using the accumulated individual behavior data; Including, A control model generating method, characterized in that the personal behavior data includes data acquired when the user is performing the cooperative action with a cooperative action target person with whom the user is familiar.

19. In the computer system, A step of accumulating personal behavior data which is data on the behavior of a person who has performed a cooperative behavior; generating a control model of the humanoid that allows the humanoid to perform the collaborative action with a user, the control model reflecting the personality of the person, using the accumulated individual behavior data; Run the command, The program, wherein the personal behavior data includes data acquired when the user is performing the cooperative activity with a cooperative activity partner with whom the user is familiar.