Control model generation device, control system, control model generation method, and program

Abstract

To obtain a control model generation device that can reduce a user's stress caused by at least one of the behavior and the way of speaking of a humanoid.SOLUTION: A control model generation device comprises: a data storage unit 23 that accumulates individual behavior data that is data related to the movement of a person; and a learning unit 22a that uses the individual behavior data accumulated in the data storage unit 23, to generate a control model of a virtual character in a virtual space, the control model reflecting the personality of the person.SELECTED DRAWING: Figure 17

Description

Technical Field

[0005] , ,

[0001] The present disclosure relates to a control model generation device, a control system, a control model generation method, and a program.

Background Art

[0002] In recent years, the development of devices capable of interacting with people, such as service robots and smart speakers, has been progressing. Also, in a virtual space (virtual space) on a computer such as the metaverse, interaction between a virtual character and a user has become possible. A device or virtual character capable of interacting with people is an example of a humanoid. A humanoid is required to perform actions and speech appropriate to an individual so as not to cause stress when living with people. However, generally, for the same device (or virtual character), the actions and ways of speaking are the same regardless of the user, that is, the partner of the interaction, and even for the same device (or virtual character), some users may feel stressed.

[0003] Patent Document 1 discloses a robot control device that controls the actions of a robot so that the expected reaction by the actions of the robot and the actual reaction of the user to the actions match in order to enable the robot to build an amicable relationship with the user.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the technology described in Patent Document 1, if the expected response from the robot's actions does not match the user's actual response to those actions, the robot's actions are controlled so that the expected response matches the user's actual response. In other words, the robot's actions are controlled retrospectively only if they were inappropriate after the robot has actually acted. As a result, inappropriate actions are performed, causing considerable stress to the user. Furthermore, users may wish for the humanoid to reflect at least one of the characteristics of a particular person, either their behavior or their way of speaking.

[0006] This disclosure is made in view of the above, and aims to provide a control model generation device that can reduce user stress caused by at least one of the behavior and speech patterns of a humanoid. [Means for solving the problem]

[0007] To solve the aforementioned problems and achieve the objective, the control model generation device according to this disclosure comprises a data storage unit that stores personal behavior data, which is data relating to a person's movements, and a learning unit that uses the personal behavior data stored in the data storage unit to generate a control model for controlling the movements of a virtual character in a virtual space, the control model reflecting the individuality of the person. The actions include the coordinated actions of the virtual character with the avatar of a person in the virtual space. A person's personality includes at least one of their behavior and manner of speaking. [Effects of the Invention]

[0008] The control model generation device described herein has the effect of reducing user stress caused by at least one of the humanoid's behavior and speech patterns. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example configuration of the cooperative operation system according to Embodiment 1. [Figure 2] A diagram showing an example of modification information in Embodiment 1. [Figure 3] A schematic diagram illustrating an example of a neural network. [Figure 4] A flowchart showing an example of the processing procedure in the control model generation unit of Embodiment 1. [Figure 5] A flowchart showing an example of the processing procedure in the robot control unit of Embodiment 1. [Figure 6] A schematic diagram illustrating an example of acquiring personal behavioral data in the first example. [Figure 7] A schematic diagram illustrating an example of cooperative operation between the user and the robot in the first example. [Figure 8] A schematic diagram illustrating an example of acquiring personal behavioral data in the second example. [Figure 9] A schematic diagram illustrating an example of cooperative operation between the user and the robot in the second example. [Figure 10] This figure shows an example configuration of a computer system that implements the control system of Embodiment 1. [Figure 11] This figure shows an example configuration of the cooperative operation system according to Embodiment 2. [Figure 12] A flowchart showing an example of the processing procedure in the control model generation unit of Embodiment 2. [Figure 13] This figure shows an example configuration of the cooperative operation system according to Embodiment 3. [Figure 14] A flowchart showing an example of the control model update process procedure in the control model generation unit of Embodiment 3. [Figure 15] This figure shows an example configuration of the cooperative operation system according to Embodiment 4. [Figure 16] This figure shows an example configuration of the cooperative operation system according to Embodiment 5. [Figure 17] This figure shows an example configuration of the cooperative operation system according to Embodiment 6. [Modes for carrying out the invention]

[0010] The control model generation apparatus, robot control device, control system, control model generation method, and program according to the embodiment will be described in detail below with reference to the drawings.

[0011] Embodiment 1. FIG. 1 is a diagram showing a configuration example of a cooperative operation system according to Embodiment 1. The cooperative operation system 100 of the present embodiment includes a control system 1, a robot 7, a detection device 4, and a situation detection device 8.

[0012] The cooperative operation system 100 of the present embodiment accumulates personal action data of the user 5 including cooperative operation data obtained when performing a cooperative operation with the cooperative operation target person 6 before the cooperative operation between the robot 7 and the user 5 is performed, and generates a control model for controlling the robot 7 using the accumulated personal action data. The cooperative operation data is personal action data obtained when performing a cooperative operation. The personal action data to be obtained may be only the cooperative operation data, or may include data other than the cooperative operation data.

[0013] The collaborative action includes, for example, at least one of conversation, work, games, and sports, but is not limited thereto. The collaborative action partner 6 is another person who performs the collaborative action with the user 5 and is a person who is familiar with the user 5 in the collaborative action. A person who is familiar with the user 5 in the collaborative action is, for example, a person who is not a first acquaintance and who knows quite a bit about each other's personality or speech and behavior habits. Therefore, the user 5 feels compatible with the person in the collaborative action and is less likely to feel stressed when performing the collaborative action together with the person. The collaborative action partner 6 may be determined by, for example, the user 5 or may be determined by an operator of another collaborative action system 100 different from the user 5. When determined by an operator of the collaborative action system 100 or the like, for example, a person who is clearly performing a collaborative action with the user 5 over a long period of time is determined as the collaborative action partner 6. The collaborative action partner 6 may be one person or a plurality of persons. When there are a plurality of collaborative action partners 6, personal action data including collaborative action data when the user 5 performs a collaborative action with each of the collaborative action partners 6 is acquired. For example, when the collaborative action partner 6 includes a first collaborative action partner and a second collaborative action partner, the personal action data of the user 5 includes collaborative action data acquired during the collaborative action between the first collaborative action partner and the user 5 and collaborative action data acquired during the collaborative action between the second collaborative action partner and the user 5.

[0014] Personal behavior data is data relating to the actions of a person who has performed coordinated actions. In this embodiment, personal behavior data of user 5 is acquired, and therefore, personal behavior data reflects the personality of user 5. User 5's personal behavior data includes data acquired when user 5 is performing coordinated actions with a coordinated action target 6 with whom user 5 is familiar, and therefore, it can be said that the personality of the coordinated action target 6 is indirectly reflected in the data. Furthermore, since the control model is generated using personal data including data of user 5 acquired when user 5 is performing coordinated actions with a coordinated action target 6 with whom user 5 is familiar, the personality of user 5 is reflected in the control model. Here, the personality of user 5 reflected in the control model is not simply the personality of user 5, but the personality of user 5 when performing coordinated actions with a coordinated action target 6 with whom user 5 is familiar. Therefore, the control model indirectly reflects not only the personality of user 5, but also the personality of the coordinated action target 6 with whom user 5 is familiar. The indirectly reflected control here refers to control that reflects the behavior or speech patterns of the collaborative agent 6 when interacting with user 5, and can also be described as control that includes elements that constitute the unique behavior or speech patterns of the collaborative agent 6. As a result, robot 7 can perform actions that do not cause stress to user 5 when performing collaborative actions.

[0015] Furthermore, personal behavior data includes data on user 5's behavioral habits. Here, behavior is defined as something used to control robot 7, and may include not only movement but also at least one of user 5's way of speaking or user 5's position and posture. That is, behavior includes, for example, at least one of the way of moving, way of speaking, and at least one of user 5's position and posture. Also, way of moving may include not only continuous movement but also at least one of the position and posture at a given moment. Behavioral habits include at least one of movement habits and speech habits. Movement habits include, for example, at least one of movement patterns such as trajectory and speed, and gestures, while speech habits include, but are not limited to, at least one of speaking speed, verbal tics, way of speaking, intonation, and dialect. Note that the movements, actions, way of speaking, habits, etc. of user 5, the collaborative action target 6, and robot 7 may also be referred to as behavior.

[0016] Robot 7 is an example of a humanoid that performs cooperative actions with User 5. The humanoid may also be referred to as the controlled object. More specifically, in this embodiment, Robot 7 is an example of a machine that performs cooperative actions with User 5. Robot 7 may be humanoid, a machine without movable parts that only communicates with User 5, an industrial machine equipped with manipulators, or something else; there are no particular restrictions on its shape and function.

[0017] The control system 1 controls the robot 7. The control system 1 includes a control model generation unit 2 that generates a control model for the robot 7, and a robot control unit 3 that controls the robot 7 using the control model generated by the control model generation unit 2 and the situation data detected by the situation detection device 8. The control model generation unit 2, which is a control model generation device, and the robot control unit 3, which is a robot control device, may be integrated, or they may be provided separately.

[0018] The detection device 4 detects the user 5's actions as personal action data when the user 5 is performing a coordinated action with the person 6 who is the target of the coordinated action, and transmits the personal action data to the control model generation unit 2. The detection device 4 is a device that detects at least one of the following: position, velocity, acceleration, posture, voice, pulse, blood pressure, body temperature, emotion, etc. There may be multiple detection devices 4.

[0019] Furthermore, the detection device 4 may also detect the user 5's biological information or psychological or internal information such as emotions when the user 5 is performing a coordinated action with the person 6, and include this information in the personal behavior data before transmitting it to the control model generation unit 2.

[0020] The detection device 4 may be a wearable device that user 5 can wear, or a portable device that user 5 can carry. Furthermore, the detection device 4 may be a device installed to detect user 5's movements, or a device that detects movements in a virtual space such as the metaverse. The detection device 4 may be a combination of these, or otherwise. If the detection device 4 is a wearable device or a portable device, it may be a device capable of detecting at least one of its own position, velocity, and acceleration, or a device capable of detecting its own rotation, or a device equipped with a microphone capable of collecting and recording sound, or a combination of these. For detecting the wearable device's position, a GPS (Global Positioning System) receiver may be used, an RFID (Radio Frequency Identification) tag may be used, or other devices may be used. By using a wearable device or a portable device as the detection device 4, it is possible to acquire user 5's personal behavior data on a daily basis, not just its location. When recording audio, sounds other than those made by user 5 may also be recorded. However, the detection device 4, the control model generation unit 2, or a device not shown in Figure 1 will perform speech recognition processing to extract user 5's voice. The speech recognition processing can be any type of process, but for example, it may involve acquiring user 5's voice in advance and using the previously acquired voice to identify the voices made by user 5.

[0021] When using a device installed to detect the movements of user 5 as the detection device 4, for example, the device may be a camera or other imaging device that photographs the location where the coordinated operation takes place, or a microphone or other device that can collect and record sound at the location where the coordinated operation takes place, or a combination of these. When the detection device 4 is an imaging device, the control model generation unit 2 or a device not shown in Figure 1 recognizes user 5 in the video captured by the imaging device and detects user 5's position, velocity, acceleration, movement trajectory, etc. The method for recognizing user 5 can be a general image recognition method, such as using a pre-captured image of user 5. Alternatively, the detection device 4 may recognize user 5 and detect user 5's position, velocity, acceleration, movement trajectory, etc. The method for detecting user 5's position, velocity, acceleration, movement trajectory, etc. can also be a general method.

[0022] If the detection device 4 is a device that detects actions in a virtual space such as the metaverse, the device may be, for example, a computer system that manages the virtual space, a terminal device used by user 5 to operate a avatar of user 5 in the virtual space, or a device that records video in the virtual space. If a device that records video in the virtual space is used as the detection device 4, the control model generation unit 2 or a device not shown in Figure 1 recognizes user 5 in the video captured by the shooting device and detects user 5's position, velocity, acceleration, movement trajectory, etc.

[0023] Furthermore, if the coordinated operation involves actions using an object, such as moving or processing an object, the detection device 4 may include a device that detects the force that the user 5 applies to the object.

[0024] The control model generation unit 2 comprises a basic model storage unit 21, a learning unit 22, a data storage unit 23, a data acquisition unit 24, and a modification information storage unit 25. The basic model storage unit 21 stores a predetermined basic control model that serves as the basis for the control model used to control the robot 7. The basic control model is a general control model that is independent of the user 5, that is, does not reflect the individuality of the user 5, and defines the basic movements of the robot 7.

[0025] The basic control model may be pre-stored in the data storage unit 23 by the vendor of the control system 1 or the vendor of the robot 7, or it may be transmitted from another device and received by a communication unit (not shown in Figure 1) and stored in the data storage unit 23. For example, when user 5 operates the control system 1, the control system 1 may receive the basic control model from an external server that provides basic control models corresponding to the robot 7. The basic control model may also be provided for each type of robot 7, or according to the type of cooperative operation performed by the robot 7. For example, basic control models corresponding to the type of cooperative operation performed by the robot 7 may be stored in the basic model storage unit 21, and user 5 may select the basic control model corresponding to the type of cooperative operation performed by the robot 7 together with user 5. Alternatively, an external server may provide basic control models corresponding to the type of cooperative operation performed by the robot 7, and user 5 may select and download the basic control model corresponding to the type of cooperative operation performed by the robot 7 together with user 5, thereby storing the basic control model in the basic model storage unit 21.

[0026] The basic control model and control model include, for example, one or more control parameters for controlling the robot 7. The control parameters include, for example, at least one of the following: parameters for controlling the movement of the robot 7, such as the trajectory, velocity, and acceleration of the robot 7; parameters for controlling the movement of each part of the robot 7, such as the end effectors and joints of the robot 7; and parameters for controlling the way the robot 7 speaks, such as the speed at which the robot 7 speaks and the pitch (frequency) of the sound of the voice that the robot 7 emits.

[0027] The data acquisition unit 24 acquires the user 5's personal behavior data from the detection device 4 and stores the acquired personal behavior data in the data storage unit 23. As described above, in order to obtain personal behavior data, processing such as image processing on the video acquired by the detection device 4 and speech recognition processing on the audio acquired by the detection device 4 may be performed. In this case, other devices (not shown) perform these processes, and the data acquisition unit 24 acquires the personal behavior data from these other devices. Alternatively, the data acquisition unit 24 may perform extraction processing such as image processing on the video acquired by the detection device 4 and speech recognition processing on the audio acquired by the detection device 4. In this case, the data acquisition unit 24 may perform extraction processing on the data acquired from the detection device 4 and store the processed data as personal behavior data in the data storage unit 23, or the data acquired from the detection device 4 itself may be stored in the data storage unit 23 as personal behavior data, and the learning unit 22 may perform extraction processing in the control model generation process described later. Here, the data acquisition unit 24 acquires personal behavior data by receiving it, but it is not limited to this, and personal behavior data may also be recorded on a recording medium or the like. In this case, the data acquisition unit 24 acquires personal behavior data by reading personal data from the recording medium.

[0028] The modification information storage unit 25 stores modification information indicating modifications to the basic control model according to the individual behavior data. The modification information is, for example, information that associates one or more features shown in the individual behavior data with the modification content. The features indicate the individuality of the person from whom the individual behavior data is acquired. Individuality includes, for example, at least one of the following: behavior, catchphrases, intonation, dialect, and movement habits. The modification content is determined according to the individuality of user 5 so that robot 7 performs actions that do not cause stress to user 5 when performing cooperative actions with robot 7. The learning unit 22 uses the individual behavior data stored in the data storage unit 23, i.e., the accumulated individual behavior data, to generate a control model for robot 7 that reflects the individuality of user 5, for robot 7 to perform cooperative actions with user 5. The learning unit 22 generates the control model using, for example, the basic control model stored in the basic model storage unit 21, the individual behavior data accumulated in the data storage unit 23, and the modification information stored in the modification information storage unit 25.

[0029] Figure 2 shows an example of the modification information in this embodiment. In the example shown in Figure 2, the individuality of user 5 is categorized based on N (where N is an integer greater than or equal to 1) feature quantities, and the modification information includes the modification of parameters (control parameters) in the control model for each type. The feature quantities may be an angle from the reference direction indicating the direction of movement of user 5 during coordinated movement, a numerical value indicating the amount of movement from a reference point during user 5's coordinated movement, information obtained by frequency-converting time-series data of user 5's position during coordinated movement, the speaking speed of user 5, or information obtained by frequency-converting user 5's voice. Furthermore, the feature quantities may be whether or not user 5 performed a specific predetermined movement, the number of times user 5 performed a specific predetermined movement per unit time, whether or not user 5 uttered a specific word, or the number of times user 5 uttered a specific word per unit time. In addition, as feature quantities that indicate user 5's habits and individuality, for example, those behaviors that show a specific regularity among behaviors that tend to differ from person to person may be extracted. Furthermore, the features may be the individual behavior data itself, or the coordinated action data within the individual behavior data. The features are not limited to the examples above, and any feature that indicates at least one of the characteristics of User 5's movement and speech, i.e., User 5's habits, is acceptable. Note that Figure 2 shows an example where N is 3 or greater, but it is not limited to this, and N can be 1 or greater.

[0030] Figure 2 illustrates an example where the correction information is defined in a table format, but the format of the correction information is not limited to the example shown in Figure 2. When the correction information shown in Figure 2 is used, the learning unit 22 extracts features from the individual behavior data stored in the data storage unit 23, identifies the type corresponding to the extracted features using the correction information, extracts the correction content corresponding to the identified type from the correction information, and generates a control model by modifying the basic control model based on the extracted correction content.

[0031] The correction information may be determined manually in advance. For example, the correction information may be determined by the vendor or manager of the robot 7 or control system 1, or it may be determined by machine learning (pre-training).

[0032] In the former case, depending on the nature of the cooperative operation, the vendor or administrator determines the modification information by inferring how to modify the robot 7's operation based on the basic control model so that user 5 does not experience stress, for each type of user 5.

[0033] In the latter case, for example, before the robot 7 is put into operation, the robot 7 is made to perform cooperative actions with an arbitrary person, and for each cooperative action, a set of data is obtained consisting of features extracted from the personal behavior data of the person who performed the cooperative action and the modifications made to each control parameter in the robot 7's control model (modifications from the basic control model). In addition, for each cooperative action, an evaluation is made to indicate whether or not the person who performed the cooperative action with the robot 7 felt stressed. By appropriately changing the actions of the robot 7 and the person performing the cooperative action, multiple datasets with different conditions and corresponding evaluation results are obtained. Note that the person performing the cooperative action can be any person, and may or may not include user 5. Also, the person performing the cooperative action does not have to be the robot 7 itself that performs the cooperative action with user 5, but may be another robot of the same type as robot 7, or another type of robot that can perform actions similar to robot 7.

[0034] When multiple datasets with different conditions and corresponding evaluation results (ground truth data) are obtained, pre-training uses the modified control parameters in the dataset where the evaluation result of "no stress felt" was obtained as ground truth data to generate a trained model through supervised learning. Pre-training may be performed by the learning unit 22, by a pre-training unit (not shown) of the control model generation unit 2, or by a learning device separate from the control system 1. When pre-training is performed, the modification information is a trained model for inferring modifications to control parameters from features extracted from individual behavior data. The learning unit 22 can infer modifications to control parameters suitable for user 5 by inputting features extracted from user 5's individual behavior data into the trained model. The learning unit 22 generates a control model by reflecting the inferred modifications in the basic control model.

[0035] Any supervised learning algorithm can be used, but for example, a neural network model can be used. A neural network consists of an input layer with multiple neurons, an intermediate layer (hidden layer) with multiple neurons, and an output layer with multiple neurons. The intermediate layer can be one or more layers.

[0036] Figure 3 is a schematic diagram illustrating an example of a neural network. For example, in a three-layer neural network like the one shown in Figure 3, when multiple inputs are input to the input layer (X1-X3), these values ​​are multiplied by weights W1 (w11-w16) and input to the hidden layer (Y1-Y2), and the result is further multiplied by weights W2 (w21-w26) and output from the output layer (Z1-Z3). This output result varies depending on the values ​​of weights W1 and W2.

[0037] In this embodiment, the relationship between features and the ground truth data is learned by adjusting weights W1 and W2 so that the output from the output layer when features extracted from individual behavior data are input approaches the corrected control parameters, which are the ground truth data. Note that the machine learning algorithm is not limited to neural networks, but may be other algorithms such as support vector machines. Also, the machine learning used to generate the trained model is not limited to supervised learning, but may be reinforcement learning, etc.

[0038] Furthermore, if pre-training is performed, table-formatted information may be used as correction information. For example, after the pre-trained model described above is generated, multiple input data sets are generated by changing the values ​​of each feature, and the control parameters obtained by inputting each input data into the pre-trained model are inferred. Then, input data in which the corrections to the control parameters obtained by inference are all the same or within a certain range is defined as one type, control parameters are determined for each type, and correction information in table format as exemplified in Figure 2 may be generated.

[0039] Furthermore, even if supervised learning or reinforcement learning is not used, the robot 7's movements and the person performing the cooperative movements may be appropriately changed to obtain multiple datasets with different conditions, and the correction information may be determined using the obtained datasets and the corresponding evaluation results. For example, the values ​​of type classification thresholds such as X1 and X2, and the control parameters may be manually determined using the corrections to each control parameter in a dataset where the evaluation result of "no stress felt" was obtained, and the features of the individual behavior data.

[0040] Furthermore, while an example using correction information has been explained here, depending on the content of the cooperative action, instead of correction information, overall information may be defined that shows the sum of the indicators obtained by the actions of robot 7 and user 5 in the cooperative action. For example, if the cooperative action is one in which robot 7 and user 5 work together to pull an object with a certain force, if user 5's pulling force is weak, robot 7 must increase its pulling force, otherwise the cooperative action will fail and user 5 will experience stress. Conversely, if user 5's pulling force is strong, robot 7 must decrease its pulling force, otherwise the cooperative action will fail and user 5 will experience stress. In such a cooperative action, the pulling force of user 5 is acquired as individual action data, and the total force is defined as overall information. The learning unit 22 then calculates the pulling force of robot 7 by subtracting the pulling force of user 5 from the total force, and calculates the amount of correction for the control parameters according to the calculated force. The overall information described above is just one example, and overall information is not limited to the example described above.

[0041] Returning to the explanation of Figure 1, the learning unit 22 outputs the generated control model to the robot control unit 3. The robot control unit 3 includes an instruction transmission unit 31, a status acquisition unit 32, a control instruction generation unit 33, and a control model storage unit 34. The robot control unit 3 is an example of a motion control unit (motion control device) for controlling a humanoid robot.

[0042] The control model storage unit 34 stores the control model output from the learning unit 22. The situation acquisition unit 32 acquires situation data indicating the status of cooperative operation between the robot 7 and the user 5, which is acquired by the situation detection device 8, by receiving it from the situation detection device 8, and outputs the acquired situation data to the control instruction generation unit 33. The situation detection device 8 may be installed on the robot 7, installed around the robot 7, or installed both on the robot 7 and around the robot 7. The situation detection device 8 acquires situation data used to control the robot 7 according to the type of robot 7 and the content of the cooperative operation. The situation detection device 8 may also acquire the status of the user 5's actions, such as the voice emitted by the user 5 and the user 5's movements. There may be multiple situation detection devices 8. The situation detection device 8 may be, for example, a camera that detects the position of the robot 7 and the state of the surroundings of the robot 7, or it may be an acceleration sensor, force sensor, etc. In addition, the situation detection device 8 detects when the robot 7 moves an object or applies force to an object. Object It may also be a camera or other device for determining the positional relationship between the robot 7 and the situation detection device 8. The situation detection device 8 may consist of two or more of these, or it may consist of other sensors, and any sensor commonly used for controlling the robot 7 can be used. Note that situation data is not required to control the robot 7, in which case the situation detection device 8 may not be provided.

[0043] The control instruction generation unit 33 generates control instructions for the robot 7 using the status data received from the status acquisition unit 32 and the control model stored in the control model storage unit 34, and outputs the generated control instructions to the instruction transmission unit 31. The instruction transmission unit 31 transmits the control instructions received from the control instruction generation unit 33 to the robot 7. The robot 7, having received the control instructions, operates based on the instructions.

[0044] Thus, in this embodiment, the learning unit 22 generates a control model using personal behavior data from when the user 5 is performing cooperative actions with the cooperative action target 6, who is familiar with the user 5, and a control instruction based on the generated control model is transmitted to the robot 7. Since the control model is generated before the cooperative action between the user 5 and the robot 7, the robot 7 can perform the same or similar actions as the cooperative action target 6 from the start of the cooperative action, thereby reducing the stress on the user 5 caused by at least one of the robot 7's actions and speech during the cooperative action.

[0045] Next, the operation of this embodiment will be described. Figure 4 is a flowchart showing an example of the processing procedure in the control model generation unit 2 of this embodiment. The control model generation unit 2 acquires personal behavior data of user 5, including cooperative action data acquired during cooperative action between user 5 and the cooperative action target person 6 (step S1). In detail, the data acquisition unit 24 acquires personal behavior data of user 5 by receiving personal behavior data from the detection device 4. As mentioned above, the data acquisition unit 24 may acquire personal behavior data using a recording medium. Alternatively, video or other images that form the basis of the personal behavior data may be acquired by the detection device 4 and extracted.

[0046] The control model generation unit 2 stores the user 5's personal behavior data (step S2). Specifically, the data acquisition unit 24 stores the received personal behavior data in the data storage unit 23.

[0047] The control model generation unit 2 generates a control model using the accumulated individual behavior data (step S3). Specifically, the learning unit 22 extracts features using the individual behavior data of user 5 stored in the data storage unit 23, and generates a control model using the features and the basic control model stored in the basic model storage unit 21. The accumulated individual behavior data is the individual behavior data acquired for each cooperative action in one or more cooperative actions. If the feature is, for example, the speaking speed of user 5, and individual behavior data corresponding to multiple cooperative actions has been accumulated, the average speed per character may be calculated using all the individual behavior data corresponding to multiple cooperative actions. If the feature is, for example, the tendency of user 5's movement, the average position at a predetermined point in time during the cooperative action may be calculated using the individual behavior data corresponding to multiple cooperative actions, and the difference between the average position and a predetermined standard position may be used as the feature. The method for calculating the feature is not limited to the examples described above.

[0048] The control model generation unit 2 outputs a control model (step S4). Specifically, the learning unit 22 outputs the generated control model to the robot control unit 3. The control model storage unit 34 of the robot control unit 3 stores the control model output from the learning unit 22.

[0049] Figure 5 is a flowchart showing an example of the processing procedure in the robot control unit 3 of this embodiment. The processing shown in Figure 5 is performed when the robot 7 and the user 5 perform cooperative operations after the control model has been generated by the control model generation unit 2.

[0050] The robot control unit 3 acquires status data (step S11). Specifically, the status acquisition unit 32 acquires status data indicating the status of the robot 7 acquired by the status detection device 8 by receiving it from the status detection device 8, and outputs the acquired status data to the control instruction generation unit 33.

[0051] The robot control unit 3 generates control instructions using the situation data and the control model (step S12). Specifically, the control instruction generation unit 33 generates control instructions for the robot 7 using the situation data received from the situation acquisition unit 32 and the control model stored in the control model storage unit 34, and outputs the generated control instructions to the instruction transmission unit 31.

[0052] The robot control unit 3 transmits a control instruction (step S13). Specifically, the instruction transmission unit 31 transmits the control instruction received from the control instruction generation unit 33 to the robot 7. As a result, the robot 7 performs an action based on the control instruction.

[0053] Furthermore, once a control model has been generated and the robot 7 and user 5 have performed cooperative actions, if user 5 and the person being coordinated with 6 then perform cooperative actions, the cooperative action data from that cooperative action may be acquired, and a control model may be generated based on the individual behavior data, including the acquired cooperative action data. In this case, the process shown in Figure 5 is performed using the newly generated control model. Thus, the control model may be updated using new cooperative action data after it has been generated. This allows the robot 7 to be controlled in a way that reflects the latest state, even if the user 5's personality changes.

[0054] Next, an example of cooperative action performed using the cooperative action system 100 of this embodiment will be described. First, as a first example of cooperative action, we will describe the case where the robot 7 is a serving robot for a restaurant. In the first example, the cooperative action is serving food. For example, A, B, C, and D all work at the same restaurant, and B sometimes serves food with A, sometimes with C, and sometimes with D. Suppose that B can perform the serving work well when working with A, and also when working with C. On the other hand, suppose that when B serves food with D, B is unable to perform the serving work comfortably and feels stressed. A and C are scheduled to retire, and after A and C retire, B is scheduled to perform the serving work with the robot 7. In such a case, in preparation for cooperative action with the robot 7, personal behavior data including cooperative action data is acquired when user 5, B, is performing cooperative action with the person to be coordinated 6. In this case, the six individuals who are familiar with coordinating actions with person B are person A and person C.

[0055] Figure 6 is a schematic diagram illustrating an example of acquiring individual behavior data in the first example. In the example shown in Figure 6, User 5 and the collaborative worker 6 serve dishes placed on the serving counter 201 to tables 202 in the dining hall of the restaurant. As shown in Figure 6, when User 5 (Person B) is serving food together with the collaborative worker 6 (Person A or Person C) in the restaurant, Person B's individual behavior data is acquired by the detection device 4. For example, as shown in Figure 6, when User 5 is serving food together with the collaborative worker 6, User 5 serves multiple small dishes, and the collaborative worker 6 serves the large dishes. On the other hand, when User 5 is serving food together with Person D, Person D serves multiple small dishes, and Person B serves the large dishes.

[0056] In the example shown in Figure 6, the personal behavior data includes information indicating which dishes were served. For example, in a serving counter 201 where dishes prepared in the kitchen are temporarily placed before serving, if the positions of small and large plates are generally predetermined, a detection device 4 can be used to detect the position of user 5, and the time-series data of the positions detected by the detection device 4 may be used as personal behavior data. In this case, for example, the learning unit 22 may obtain the movement history of user 5 and, based on the obtained history and the positions of the small and large plates, determine the size of the dishes served by user 5 as a feature. Alternatively, a detection device 4 capable of photographing the serving counter 201 may be used, and the size and number of dishes served by user 5 may be calculated as personal behavior data by the data acquisition unit 24 or other device by analyzing the video captured by the detection device 4. Alternatively, a detection device 4 capable of photographing the serving counter 201 may be used, and the video captured by the detection device 4 may be used as personal behavior data, and the learning unit 22 may determine the size of the dishes served by user 5 as a feature from the video.

[0057] In the first example, it is assumed that the serving task includes both large and small plates. Therefore, the size of the plates to be served, or the size and number of plates to be served, are included as features, and the types in the modification information are defined as either a type that serves multiple small plates or a type that serves a large plate. Then, in the modification information, numerical values ​​that serve as control parameters for the robot 7 to serve large plates are defined as the modification content corresponding to the type. For example, in the first example, the control model includes a serving decision model and a movement model, and the serving decision model includes a definition of the size of the plates that the robot 7 will serve. Then, in the modification information corresponding to the above type, information is defined to set control parameters that make the plates to be served have a diameter of a certain value or more. As a result, the learning unit 22 of the control model generation unit 2 can generate a control model that causes the robot 7 to serve large plates. Furthermore, if the positions where small and large plates are placed on the serving counter 201 are generally predetermined, instead of specifying the size of the plates, the range in which the plates to be served are located on the serving counter 201 may be defined as a control parameter.

[0058] Figure 7 is a schematic diagram showing an example of cooperative operation between user 5 and robot 7 in the first example. In the example shown in Figure 7, as explained using Figure 6, a control model is generated that causes robot 7 to serve food from large platters, and therefore robot 7 serves the food. As a result, user 5 can perform the serving task efficiently with reduced stress, just as if they were performing the serving task together with person A or person C, who are the subjects of the cooperative operation 6. Thus, in the first example, the control system 1 can learn a serving method that allows user 5, person B, to act efficiently, and generate a control model that reflects the learned results.

[0059] Next, as a second example of cooperative action, we will describe a case where robot 7 and user 5 perform assembly work. In this second example, robot 7 is, for example, an assembly robot, which is a type of industrial machine. For example, A, B, C, and D are all workers who perform assembly work, and B sometimes works with A, sometimes with C, and sometimes with D. Suppose that B was able to perform the assembly work well when working with A, and also when working with C. On the other hand, suppose that when B worked with D, user 5 (B) was unable to perform the assembly work comfortably and felt stressed. A and C are scheduled to be transferred, and after A and C's transfer, B is scheduled to perform assembly work with robot 7. In this case, as in the first example, in preparation for cooperative action with robot 7, personal behavior data including cooperative action data is acquired when user 5 (B) is performing cooperative action with the person to be coordinated 6. In this case, the six individuals who are familiar with coordinating actions with person B are person A and person C.

[0060] Figure 8 is a schematic diagram illustrating an example of acquiring individual behavioral data in the second example. In the example shown in Figure 8, User 5 and the collaborative worker 6 work together to perform assembly. More specifically, User 5 (Person B) places the first part 204, and the collaborative worker 6 (Person A or Person C) places the second part 205 on top of the first part 204. The standard position 203 indicates the standard position where the first part 204 is placed, and User 5 has a habit of placing the first part 204 slightly to the right in Figure 8 compared to the standard position. The collaborative worker 6, being familiar with User 5, places the second part 205 in the same position as where User 5 places the first part 204, thus enabling efficient assembly work. On the other hand, user D, who is not familiar with user 5, attempts to position the second part 205 assuming that the first part 204 will be placed in the standard position 203. This requires time for alignment, and user 5 may need to change the position of the first part 204, making it difficult to assemble efficiently and causing user 5 stress.

[0061] In such cases, the detection device 4 detects the location where user 5 places the first part 204, or the position of user 5's hand when user 5 places the first part 204. Then, using the difference of the first part 204 from the standard position 203 as a feature, a type is defined in the correction information that indicates the placement position of the first part 204 is shifted from the standard position 203 by a threshold or more.Then, as the correction content corresponding to this type in the correction information, the control parameters are determined so that the position of the second part 205 placed by the robot 7 is shifted by the same amount as the difference between the placement position of the first part 204 and the standard position 203.As a result, the learning unit 22 of the control model generation unit 2 can generate a control model that causes the robot 7 to place the second part 205 in accordance with the amount that user 5 shifted the first part 204 from the standard position 203.

[0062] Figure 9 is a schematic diagram showing an example of cooperative operation between user 5 and robot 7 in the second example. In the example shown in Figure 9, as explained using Figure 8, a control model is generated in which robot 7 positions the second part 205 in accordance with the amount that user 5 has shifted the first part 204 from its standard position 203. As a result, robot 7 positions the second part 205 shifted to the right. This allows user 5 to perform the assembly work efficiently with reduced stress, just as if performing the assembly work together with person A or person C, who are the subjects of the cooperative operation 6.

[0063] In both the first and second examples, the robot 7 is able to work with user 5, person B, by taking into account the user's behavior. Even in an environment where a human and a humanoid robot 7 work together for labor-saving purposes, user 5 can perform cooperative actions with reduced stress such as difficulty or discomfort. The cooperative actions performed using the cooperative action system 100 described above are illustrative examples, and the cooperative actions performed using the cooperative action system 100 are not limited to the examples described above.

[0064] Furthermore, cooperative actions are not limited to those performed by two people; they may involve three or more people. For example, in the case of a cooperative action performed by three people, two robots 7 may be used, or robot 7 and the person 6 performing the cooperative action together with user 5 may perform the cooperative action. In this case, for example, if user 5, B, performs a cooperative action with A and C and B and A do not feel stressed, a control model is generated based on C's individual behavior data, and robot 7 is controlled based on this control model.

[0065] Next, the hardware configuration of each device in this embodiment will be described. In the control system 1 of this embodiment shown in Figure 1, the computer system functions as the control system 1 when a program, which is a computer program describing the processing in the control system 1, is executed on the computer system. Figure 10 is a diagram showing an example of the configuration of a computer system that realizes the control system 1 of this embodiment. As shown in Figure 10, this computer system comprises a control unit 101, an input unit 102, a storage unit 103, a display unit 104, a communication unit 105, and an output unit 106, which are connected via a system bus 107. The control unit 101 and the storage unit 103 constitute a processing circuit.

[0066] In Figure 10, the control unit 101 is a processor such as a CPU (Central Processing Unit), which executes a program describing the processing in the control system 1 of this embodiment. Part of the control unit 101 may be implemented by dedicated hardware such as a GPU (Graphics Processing Unit) or FPGA (Field-Programmable Gate Array). The input unit 102 may be an input means such as a button, keyboard, mouse, joystick, touchpad, or game controller. The storage unit 103 includes various types of memory such as RAM (Random Access Memory) and ROM (Read Only Memory), as well as storage devices such as a hard disk, and stores the program to be executed by the control unit 101, necessary data obtained during processing, etc. The storage unit 103 is also used as a temporary storage area for the program. The display unit 104 is, as described above, a display, for example. The display unit 104 and the input unit 102 may be integrated and implemented as a touch panel, etc. The communication unit 105 is a receiver and transmitter that perform communication processing. The output unit 106 is a speaker, etc. Note that Figure 10 is an example, and the configuration of the computer system is not limited to the example shown in Figure 10. For example, in this embodiment, the computer system that implements the control system 1 does not need to have an output unit 106.

[0067] Here, an example of the operation of the computer system until the program of this embodiment becomes executable will be described. In a computer system with the above configuration, for example, a computer program is installed in the storage unit 103 from a CD-ROM or DVD-ROM set in a CD (Compact Disc)-ROM drive or DVD (Digital Versatile Disc)-ROM drive (not shown). When the program is executed, the program read from the storage unit 103 is stored in the main memory area of ​​the storage unit 103. In this state, the control unit 101 performs processing as the control system 1 of this embodiment according to the program stored in the storage unit 103.

[0068] In the above explanation, a program describing the processing in control system 1 is provided on a CD-ROM or DVD-ROM as the recording medium. However, the explanation is not limited to this, and depending on the configuration of the computer system, the capacity of the program to be provided, a program provided via a transmission medium such as the Internet may be used.

[0069] The program of this embodiment, for example, causes a computer system to perform the following steps: accumulate personal behavior data, which is data relating to the actions of a person who has performed cooperative actions with the computer; and use the accumulated personal behavior data to generate a control model for a humanoid that performs cooperative actions with user 5, and which reflects the individuality of the person.

[0070] The learning unit 22 and control instruction generation unit 33 shown in Figure 1 are realized by the execution of a computer program stored in the storage unit 103 shown in Figure 10 by the control unit 101 shown in Figure 10. The storage unit 103 shown in Figure 10 is also used to realize the learning unit 22 and control instruction generation unit 33 shown in Figure 1. The data acquisition unit 24, instruction transmission unit 31 and status acquisition unit 32 shown in Figure 1 are realized by the communication unit 105 shown in Figure 10. The data acquisition unit 24 may also be realized by a device that reads the recording medium. The basic model storage unit 21, data storage unit 23, correction information storage unit 25 and control model storage unit 34 shown in Figure 1 are part of the storage unit 103 shown in Figure 10.

[0071] The control system 1 shown in Figure 1 may be implemented by multiple computer systems. For example, the control system 1 may be implemented by a cloud system. Also, as described above, the control model generation unit 2 and the robot control unit 3 may be configured as separate devices, and in this case as well, each of the separate devices may be implemented by multiple computer systems.

[0072] As described above, the cooperative action system 100 of this embodiment generates a control model for controlling the robot 7 using the user's personal behavior data, including cooperative action data acquired when the user 5 and the person being coordinated 6 performed cooperative actions, before the cooperative action between the robot 7 and the user 5 takes place. The robot 7 can perform actions that are the same as or similar to those of the person being coordinated 6 from the start of the cooperative action, thereby reducing the stress on the user 5 caused by at least one of the robot 7's actions and speech during the cooperative action.

[0073] Embodiment 2. Figure 11 shows an example of the configuration of the cooperative operation system according to Embodiment 2. The cooperative operation system 100a of this embodiment is the same as the cooperative operation system 100 of Embodiment 1, except that it is equipped with a control system 1a instead of a control system 1 and a detection device 4a instead of a detection device 4. Components having the same functions as those in Embodiment 1 are denoted by the same reference numerals as in Embodiment 1, and redundant descriptions are omitted. The differences from Embodiment 1 will be mainly described below.

[0074] In Embodiment 1, a control model was generated using User 5's personal behavior data, including cooperative action data acquired when User 5 and the cooperative action target 6 were performing cooperative actions. In this embodiment, a control model is generated using the cooperative action target 6's personal behavior data, including cooperative action data acquired when User 5 and the cooperative action target 6 were performing cooperative actions. In this embodiment, the personal behavior data, which is data relating to the actions of a person who has performed cooperative actions before, is the personal behavior data of the cooperative action target 6. That is, the person whose personal behavior data is acquired is User 5 in Embodiment 1, and the cooperative action target 6 in this embodiment. The cooperative action target 6 is, as in Embodiment 1, a person that User 5 is familiar with and with whom User 5 does not feel stressed when performing cooperative actions.

[0075] The detection device 4a acquires personal behavior data of the collaborative action subject 6 and transmits the personal behavior data to the control system 1a. As with Embodiment 1, the personal behavior data may include biometric information or psychological or internal information such as emotions of the collaborative action subject 6. The detection device 4a is the same as the detection device 4 of Embodiment 1, but the target of the acquisition of personal behavior data is the collaborative action subject 6. The detection device 4a may be a wearable terminal that the collaborative action subject 6 can wear, a portable terminal that the collaborative action subject 6 can carry, a device installed to detect the movements of the collaborative action subject 6, a device that detects movements in a virtual space such as the metaverse, a combination of these, or something else. In other words, the personal behavior data includes, for example, data acquired by a wearable terminal and data recording the movements of a person in a virtual space.

[0076] The control system 1a is the same as the control system 1 of Embodiment 1, except that it includes a control model generation unit 2a instead of a control model generation unit 2. The control model generation unit 2a does not include a correction information storage unit 25, includes a learning unit 22a instead of a learning unit 22, and the data acquisition source of the data acquisition unit 24 is detection device 4a instead of detection device 4, but other than these, it is the same as the control model generation unit 2 of Embodiment 1. In this embodiment as well, the control model generation unit 2a and the robot control unit 3 may be provided as separate devices.

[0077] Next, the operation of the control model generation unit 2a in this embodiment will be described. Figure 12 is a flowchart showing an example of the processing procedure in the control model generation unit 2a in this embodiment. The control model generation unit 2a acquires personal behavior data of the cooperative action target 6, including cooperative action data acquired during cooperative action between the user 5 and the cooperative action target 6 (step S21). In detail, the data acquisition unit 24 acquires the personal behavior data of the cooperative action target 6 by receiving personal behavior data from the detection device 4a. As in Embodiment 1, the data acquisition unit 24 may acquire the personal behavior data using a recording medium. Alternatively, video or other images that form the basis of the personal behavior data may be acquired by the detection device 4a and extracted.

[0078] The control model generation unit 2a stores the individual behavior data of the cooperative operation target 6 (step S22). Specifically, the data acquisition unit 24 stores the received individual behavior data in the data storage unit 23.

[0079] The control model generation unit 2a generates a control model using the accumulated individual behavior data of the cooperative operation target 6 (step S23). Specifically, the learning unit 22a extracts features using the individual behavior data of the cooperative operation target 6 stored in the data storage unit 23, and generates a control model based on these features.

[0080] In this embodiment, the control parameters in the control model are set so that the robot 7 performs the actions indicated as features. This generates a control model that causes the robot 7 to perform actions similar to those that reflect the individuality of the person 6 with whom the collaborative action is performed. The features can be the same as those in Embodiment 1, but in this embodiment, features may include information about dialects, speech habits (including verbal tics), and topics of conversation (genres that are frequently discussed). Information about dialects includes, for example, whether or not there is a dialect, and if there is, what type (or region) of dialect it is. Dialect identification may be performed, for example, by storing a dictionary of dialects for each type of dialect in advance and using that dictionary, or by other methods. Speech habits include, for example, frequently using a particular word or phrase at the end of a sentence, frequently uttering a particular word or phrase, intonation such as raising the pitch of the end of a sentence, the pitch of the voice, and the speed of conversation, but other habits may also be used. The learning unit 22a extracts these speech habits by performing speech recognition processing on the speech data obtained as individual behavior data of the person being coordinated 6, for example.

[0081] Step S24, following step S23, is the same as in Embodiment 1, and the learning unit 22a outputs the generated control model to the robot control unit 3. The outputted control model is stored in the control model storage unit 34 of the robot control unit 3. The operation of the robot control unit 3 is the same as in Embodiment 1. In this embodiment, a control model is generated to perform actions that reflect the personality of the collaborative action target 6, based on the personal behavior data of the collaborative action target 6. As a result, in collaborative action with the user 5, the robot 7 can perform actions that reflect the personality of the collaborative action target 6, which the user 5 is familiar with, and the stress on the user 5 caused by at least one of the robot 7's actions and speech during collaborative action can be reduced.

[0082] Next, an example of cooperative action performed using the cooperative action system 100a of this embodiment will be described. As an example, we will describe an example where robot 7 is a communication robot and the cooperative action is conversation. A and B are a married couple, and B is accustomed to conversing with A and does not feel stressed when talking to A. A is scheduled to be transferred overseas on his own, and during A's assignment, B will converse with robot 7. In this case, the user 5 of robot 7 is B, A is set as the cooperative action target 6, and A's personal behavior data is acquired. The control model generation unit 2a generates a control model based on the accumulated personal behavior data of A. For example, the basic control model includes a conversation model and a voice model, and the conversation model and voice model are modified based on the personal behavior data to have characteristics similar to those of A. As a result, a control model is generated that reflects, for example, A's catchphrases, intonation, dialect, way of responding, and the content of topics offered.

[0083] When user 5, person B, engages in a conversation with robot 7 as part of a coordinated action while person A is away on a business trip, robot 7 is controlled using a control model based on the personal behavior data of person A described above. This allows robot 7 to engage in conversation that reflects person A's personality, thereby reducing stress for user 5, person B. It should be noted that the coordinated actions performed using the coordinated action system 100a described above are illustrative examples, and the coordinated actions performed using the coordinated action system 100a are not limited to the examples described above.

[0084] The control system 1a of this embodiment is implemented by a computer system, similar to the control system 1 of Embodiment 1. The control system 1a of this embodiment may also be implemented by multiple computer systems, for example, by a cloud system. Furthermore, as described above, the control model generation unit 2a and the robot control unit 3 may each be configured as separate devices, and in this case as well, each of the separate devices may be implemented by multiple computer systems.

[0085] Furthermore, cooperative actions are not limited to those performed by two people, but may involve three or more people. For example, in the case of cooperative actions performed by three people, two robots 7 may be used, or robot 7 and the person 6 performing the cooperative action together with user 5 may perform the cooperative action. In this case, for example, if user 5, person B, did not feel stressed when performing cooperative actions with person A and person C, then control models may be generated based on the individual behavior data of person A and person C, and the two robots 7 may be controlled based on their respective control models. Also, when robot 7 and the person 6 performing the cooperative action together with user 5, if the person 6 performing the cooperative action is person A, robot 7 may be controlled based on the control model corresponding to person C, and if the person 6 performing the cooperative action is person C, robot 7 may be controlled based on the control model corresponding to person A.

[0086] As described above, the cooperative action system 100a of this embodiment generates a control model for controlling the robot 7 using personal behavior data of the person to be coordinated, including cooperative action data acquired when the user 5 and the person to be coordinated 6 performed cooperative action, before the cooperative action between the robot 7 and the user 5 takes place. The robot 7 can perform the same or similar actions as the person to be coordinated 6 from the start of the cooperative action, thereby reducing stress on the user 5 caused by at least one of the robot 7's actions and speech during the cooperative action.

[0087] Embodiment 3. Figure 13 shows an example of the configuration of the cooperative operation system according to Embodiment 3. The cooperative operation system 100b of this embodiment is the same as the cooperative operation system 100a of Embodiment 2, except that it includes a control system 1b instead of a control system 1a. Components having the same functions as in Embodiment 2 are denoted by the same reference numerals as in Embodiment 2, and redundant descriptions are omitted. The differences from Embodiment 2 will be mainly described below.

[0088] The control system 1b is the same as the control system 1a of Embodiment 2, except that it includes a control model generation unit 2b instead of a control model generation unit 2a. The control model generation unit 2b has an additional action result acquisition unit 26 and a learning unit 22b instead of a learning unit 22a, but otherwise it is the same as the control model generation unit 2a of Embodiment 2. In this embodiment as well, the control model generation unit 2b and the robot control unit 3 may be provided as separate devices.

[0089] Next, the operation of the control model generation unit 2b in this embodiment will be described. The control model generation process in the control model generation unit 2b is the same as the process in Embodiment 2 described with reference to Figure 12. In this embodiment, after the control model is generated, the control model generation unit 2b updates the control model based on the action results, which are the result of the cooperative actions performed by the robot 7 and the user 5, and the control model corresponding to the action results. As described in Embodiment 2, the control model is generated to reduce stress on the user 5, but in this embodiment, by updating the control model using the action results, the operation of the robot 7 can be made more suitable for the user 5.

[0090] Figure 14 is a flowchart showing an example of the control model update process in the control model generation unit 2b of this embodiment. First, the control model generation unit 2b acquires action results corresponding to the control model (step S31). Specifically, the action result acquisition unit 26 acquires action results corresponding to the cooperative action (cooperative action between the robot 7 and the user 5) performed by control based on the control model stored in the robot control unit 3, and outputs the acquired action results to the learning unit 22b.

[0091] The action result indicates, for example, whether a positive or negative result was obtained. The action result is determined, for example, by user 5 and input to the control model generation unit 2b. In this case, the action result acquisition unit 26 has a function to receive input from user 5. Alternatively, user 5 may input the action result to another device, such as a user terminal (not shown), and the other device may transmit the action result to the control model generation unit 2b. In this case, the action result acquisition unit 26 has a communication function to receive the action result. User 5 may, for example, determine the action result as a positive result if they felt that they were able to perform cooperative action with robot 7 comfortably, without stress, or efficiently, and as a negative result if they felt stressed, unpleasant, or inefficient.

[0092] Furthermore, if the coordinated action is a task, for example, the outcome of the action may be determined by other means. For example, if the coordinated action is a task with a defined procedure, the time taken for the task may be measured, and if the measurement result is below a threshold, a person other than user 5 may determine that the task was performed efficiently and assign a positive outcome to the action result, and if the measurement result exceeds the threshold, a person may determine that the task was not performed efficiently and assign a negative outcome to the action result. In this case as well, the action result may be input to the control model generation unit 2b or transmitted from another device. Since it can be estimated that user 5 experiences less stress when the task is performed efficiently, the action result may be determined based on the measurement result of the task time in this way. Alternatively, the control model generation unit 2b may make the judgment based on the measurement result as described above. For example, the action result acquisition unit 26 may receive the measurement result from a device that measures the task time and use the received measurement result to determine the action result. The method for determining the action result is not limited to the examples described above.

[0093] The control model generation unit 2b determines whether the action result is a negative result or not (step S32). Specifically, the learning unit 22b determines whether the action result received from the action result acquisition unit 26 is a negative result or not.

[0094] If the action result is not negative (step S32 No), that is, if the action result is positive, the control model generation unit 2b terminates the control model update process.

[0095] If the action result is negative (step S32 Yes), the control model generation unit 2b updates the control model (step S33) and repeats the process from step S31. In step S33, specifically, the learning unit 22b updates the control model and outputs the updated control model to the robot control unit 3. This updates the control model stored in the control model storage unit 34 of the robot control unit 3. The learning unit 22b updates the control model, for example, by changing some of the control parameters in the control model. The method for changing the control parameters may be predetermined or specified by the user 5. For example, when updating the control parameters that change the position of the robot 7, the rules for changing the position of the robot 7 may be predetermined, or the user 5 may specify the direction and amount of change regarding the position of the robot 7.

[0096] As described above, if the action result is negative, the control model is updated, control is performed using the updated control model, and the process from step S31 is repeated. If the action result is negative, the control parameters are repeatedly changed until a positive result is obtained as the action result.

[0097] In the example above, if the action result was positive, the current control model was used as the updated control model without changing it. However, the control model may also be updated by changing the current control parameters to control parameters that are estimated to be better. Control parameters estimated to be better are, for example, control parameters that have been changed in the opposite direction to the control parameters that were set when the action result was negative previously. For example, if the action result was negative when the conversation speed was at the first speed, and the action result became positive when the conversation speed was changed to a second speed which is slower than the first speed, the control model may be updated to change the conversation speed to a third speed which is slower than the second speed. Then, the action result is obtained again, and if the action result is negative, the control model is updated to return the conversation speed to the second speed.

[0098] Alternatively, the control model update process is not limited to the procedure shown in Figure 14. The control parameters may be changed sequentially, action results corresponding to each control parameter value may be obtained, a dataset of control parameter values ​​and corresponding action results may be stored, and the control model may be updated using multiple datasets. For example, datasets in which the action results were positive may be extracted, one of the extracted datasets may be selected, and the control model may be updated using the control parameters in the selected dataset. Alternatively, the action results and corresponding control parameters may be treated as a single dataset, and the control model may be updated by determining control parameters that improve the action results using machine learning with multiple datasets. For example, the learning unit 22b generates a trained model using supervised learning as described in Embodiment 1, using multiple datasets consisting of action results and corresponding ground truth data, which are control parameters. Then, during inference, i.e., when updating the control model, the learning unit 22b can infer control parameters that result in positive action results by inputting values ​​where the action result is positive.

[0099] Furthermore, the behavioral results are not limited to a binary value of positive or negative, but may be represented by a numerical value on a scale of three or more. For example, the behavioral results may be assigned a score from 0 to 5, with a score of 5 representing the least stress for user 5, and a score of 0 representing the most stress. Note that the definition of the score is not limited to this example. If the results are represented by a numerical value on a scale of three or more, in the process shown in Figure 14, in step S32, the learning unit 22b should determine whether the numerical value represents the most positive behavioral result. Also, when updating the control model using the multiple datasets described above, the learning unit 22b should select the dataset that represents the numerical value representing the most positive behavioral result.

[0100] Furthermore, in the above example, the action result was the result of evaluating the entire control model, but it is not limited to this, and may also be the result of evaluating the time-series movements of the robot 7 in segments. For example, control instructions to the robot 7 may be recorded in an operation history storage unit (not shown), and the action result may be determined, for example, at regular intervals or at segments of the robot 7's movements. In this case, the operation history storage unit may be provided in the robot control unit 3, in the control model generation unit 2b, or outside the control system 1b. In this case, the action result acquisition unit 26 reads and acquires the control instructions for the period corresponding to the action result from the operation history storage unit along with the action result, and outputs the action result and the corresponding control instructions to the learning unit 22b. As a result, the learning unit 22b can obtain the action result for the robot 7's movements corresponding to the control instructions performed in time series.

[0101] For example, suppose that the coordinated action is a conversation, and based on the individual behavior data of the person 6 performing the coordinated action, the result is obtained that the person 6 frequently provides topics in both the first and second genres, and a control model is generated based on this. When generating the control model, suppose the frequency of providing topics in the first and second genres is set to be roughly equal. In a conversation, the time series consists of a period in which the conversation of the first genre takes place and a period in which the conversation of the second genre takes place. Based on the control instructions to the robot 7, these periods are distinguished and the action result acquisition unit 26 acquires the action results for each. For example, if the action result during the period of conversation in the first genre is a positive result and the action result during the period of conversation in the second genre is a negative result, the control model is updated to increase the frequency of providing topics in the first genre and decrease the frequency of providing topics in the second genre.

[0102] The control model update process is not limited to the example described above; any method by which the learning unit 22b updates the control model based on the action results to create a control model more suitable for user 5 is acceptable.

[0103] The control system 1b of this embodiment is implemented by a computer system, similar to the control system 1a of the second embodiment. The control system 1b of this embodiment may also be implemented by multiple computer systems, for example, by a cloud system. Furthermore, as described above, the control model generation unit 2b and the robot control unit 3 may each be configured as separate devices, and in this case as well, each of the separate devices may be implemented by multiple computer systems.

[0104] As described above, the cooperative operation system 100b of this embodiment performs the operations described in Embodiment 2 and updates the control model based on the action results, which are the result of the cooperative operation between the robot 7 and the user 5. Therefore, it achieves the same effects as Embodiment 2 and allows the robot 7 to perform operations that are more suitable for the user 5.

[0105] In the example described above, a control model update function was added to the cooperative operation system 100a of Embodiment 2. However, the system is not limited to this, and a control model update function may also be added to the cooperative operation system 100 of Embodiment 1. For example, the control model may be updated in the same way as in the example described above by adding an action result acquisition unit 26 to the control model generation unit 2 of the cooperative operation system 100 and providing the learning unit 22 with a control model update function similar to that of the learning unit 22b.

[0106] Embodiment 4. Figure 15 shows an example of the configuration of the cooperative operation system according to Embodiment 4. The cooperative operation system 100c of this embodiment is the same as the cooperative operation system 100b of Embodiment 3, except that it includes a control system 1c instead of a control system 1b. Components having the same functions as in Embodiment 3 are denoted by the same reference numerals as in Embodiment 3, and redundant descriptions are omitted. The differences from Embodiment 3 will be mainly described below.

[0107] The control system 1c is the same as the control system 1b of Embodiment 3, except that it includes a control model generation unit 2c instead of a control model generation unit 2b. The control model generation unit 2c has an additional model selection receiving unit 27 and includes a basic model storage unit 21a instead of a basic model storage unit 21, but otherwise it is the same as the control model generation unit 2b of Embodiment 3. In this embodiment as well, the control model generation unit 2c and the robot control unit 3 may be provided as separate devices.

[0108] The basic model memory unit 21a stores in advance multiple, or multiple types of, basic control models. These multiple types of basic control models can be said to be typical patterns that characterize the cooperative action target 6. Typical patterns that characterize the cooperative action target 6 are, for example, patterns that correspond to actions (behaviors) based on typical human personality traits, such as being impatient, easygoing, or meticulous, if the basic control model is modeled after a person. In other words, for example, if the cooperative action includes conversation, three basic control models—a moderate model, a steady model, and a leading model—are stored in the basic model memory unit 21a. Each of the moderate model, steady model, and leading model differs in at least one of the following: the content of the conversation, the speed of the conversation, the frequency of speaking, or the genre of topics offered.

[0109] Furthermore, for example, in the case of a basic control model applied to industrial machinery, the basic model storage unit 21a stores basic control models that perform actions (behaviors) based on typical work content that requires consideration for the person working together, such as collaborative design work with engineering tools, collaborative precision work such as medical procedures (surgery), or collaborative long-duration work such as the installation of large equipment.

[0110] Furthermore, basic control models, or typical patterns, may be classified based on the typical characteristics of other objects that cooperate in operation. In this way, multiple control models each correspond to different characteristics (individual characteristics of operation). Note that the number of basic control models is not limited to this example, nor is it limited to three.

[0111] The model selection receiving unit 27 receives a selection result from the user 5 indicating the basic control model selected by the user 5 from among multiple basic control models. For example, the model selection receiving unit 27 may also receive input of the selection result from the user 5. Alternatively, the user 5 may input the selection result to another device, such as a user terminal (not shown), and the other device may transmit the selection result to the control model generation unit 2c, and the model selection receiving unit 27 may receive the selection result, thereby receiving the selection result of the basic control model. The user 5 selects a basic control model from among multiple basic control models according to their preferences and compatibility. For example, the user 5 may select a basic control model that matches the cooperative operation target 6 from among multiple basic control models. For example, if three basic control models, a moderate model, a solid model, and a traction model, are stored in the basic model storage unit 21a, and the cooperative operation target 6 that the user 5 is familiar with has a moderate personality, the user 5 may select the moderate model. Note that instead of the user 5 making the selection, the control system 1c or the operator of the control system 1c may select a basic control model suitable for the cooperative operation target 6.

[0112] The model selection receiving unit 27 reads the basic control model corresponding to the received selection result from the basic model storage unit 21a and outputs the read basic control model to the learning unit 22b. The learning unit 22b uses the basic control model received from the model selection receiving unit 27, i.e., the basic control model indicated by the selection result, and the individual action data to generate a control model, similar to Embodiment 3, and outputs the generated control model to the robot control unit 3. The learning unit 22b also updates the control model using the action results, similar to Embodiment 3. The operation of this embodiment other than that described above is the same as in Embodiment 3.

[0113] The control system 1c of this embodiment is implemented by a computer system, similar to the control system 1b of Embodiment 3. The control system 1c of this embodiment may also be implemented by multiple computer systems, for example, by a cloud system. Furthermore, as described above, the control model generation unit 2c and the robot control unit 3 may each be configured as separate devices, and in this case as well, each of the separate devices may be implemented by multiple computer systems.

[0114] As described above, the cooperative operation system 100c of this embodiment generates a control model using a basic control model selected from a plurality of basic control models with different operational characteristics, and individual behavior data. Furthermore, the cooperative operation system 100c of this embodiment updates the control model based on the behavior results, which are the result of the cooperative operation between the robot 7 and the user 5. Therefore, it achieves the same effects as in Embodiment 3, and allows the robot 7 to perform actions that are more in line with the user 5's preferences and compatibility.

[0115] In the above example, the cooperative operation system 100b of Embodiment 3 was given a function to generate a control model using a basic control model selected from a plurality of basic control models. However, the system is not limited to this, and the cooperative operation system 100 of Embodiment 1 may also be given a function to generate a control model using a basic control model selected from a plurality of basic control models. For example, by adding a model selection receiving unit 27 to the control model generation unit 2 of the cooperative operation system 100 and replacing the basic model storage unit 21 with a basic model storage unit 21a, a control model may be generated using a basic control model selected from a plurality of basic control models, similar to the above example. Alternatively, the cooperative operation system 100a of Embodiment 2 may be given a function to generate a control model using a basic control model selected from a plurality of basic control models. For example, the cooperative operation system 100aBy adding a model selection receiving unit 27 to the control model generation unit 2a and replacing the basic model storage unit 21 with a basic model storage unit 21a, a control model may be generated using a basic control model selected from multiple basic control models, similar to the example described above.

[0116] Embodiment 5. Figure 16 shows an example of the configuration of a cooperative operation system according to Embodiment 5. The cooperative operation system 100d of this embodiment includes a control system 1d, a detection device 4, and a situation detection device 8a. The control system 1d generates a virtual space such as a metaverse and transmits virtual space information to a terminal device 94 to allow the user 5 to perceive the virtual space. Based on the virtual space information received from the control system 1d, the terminal device 94 outputs video in the virtual space to a video display device 95 and audio in the virtual space to an audio display device 96. The user 5 performs cooperative actions with a virtual character 902 via their own avatar 901 in the virtual space.

[0117] In Embodiments 1 to 4, a robot 7 was used as an example of a humanoid that works in cooperation with the user 5. However, in this embodiment, we will describe an example where the humanoid that works in cooperation with the user 5 is a virtual character 902 in a virtual space. In this embodiment, when the user 5 works in cooperation with the virtual character 902, the virtual character 902 is controlled using a control model generated based on the user 5's personal behavior data acquired in advance while working in cooperation with the person to be coordinated 6, similar to Embodiment 1. Components having the same function as in Embodiment 1 are denoted by the same reference numerals as in Embodiment 1, and redundant explanations are omitted. The following mainly describes the differences from Embodiment 1.

[0118] Figure 16 shows an example in which the terminal device 94 transmits video and audio to the video display device 95 and audio display device 96 via wireless communication, but one or more of the video and audio may be transmitted via wired communication. The terminal device 94 may be included in the control system 1d, or the terminal device 94, video display device 95, and audio display device 96 may be included in the control system 1d.

[0119] Furthermore, while Figure 16 shows that user 5 uses a video display device 95 and an audio display device 96 as means for perceiving the virtual space, means capable of perceiving one or more of the senses of force, touch, smell, and taste may also be used. The sense of force and touch may include information perceived by the skin, such as temperature, in addition to stress. Also, while Figure 16 shows the use of a video display device 95 and an audio display device 96, neither of these may be used. Also, while Figure 16 shows an example where a VR goggles or head-mounted display is used as the video display device 95 and headphones are used as the audio display device 96, it is not limited to these examples. For example, the video display device 95 may be a display or monitor, and the audio display device 96 may be a speaker, and specific examples of the video display device 95 and audio display device 96 are not limited to the examples shown in Figure 16. In addition, two or more of the terminal device 94, video display device 95, and audio display device 96 may be integrated. For example, the display of the terminal device 94 may be used as the video display device 95. Furthermore, for example, a head-mounted display having the functions of both a terminal device 94 and a video display device 95 may be used, or a head-mounted display with headphones may be used.

[0120] The situation detection device 8a acquires the user 5's status during cooperative operation between the user 5 and the virtual character 902. For example, the situation detection device 8a detects the user 5's voice, movements, etc., and transmits the detection results to the terminal device 94. The terminal device 94 transmits the detection results received from the situation detection device 8a to the control system 1d. There may be multiple situation detection devices 8a. Also, for example, by using a headset as the voice presenter 96, the voice presenter 96 and the situation detection device 8a may be integrated. Also, the terminal device 94 may be equipped with a situation detection device 8a. Furthermore, the situation detection device 8a may be worn by the user 5, or it may be provided around the user 5, such as a camera that photographs the user 5 from the outside.

[0121] The control system 1d includes a control model generation unit 2 and a virtual space control unit 9, similar to those in Embodiment 1. The control model generation unit 2 and the virtual space control unit 9 may each be provided as separate devices. The configuration and operation of the control model generation unit 2 are the same as in Embodiment 1, but the control model generated by the control model generation unit 2 is a control model for controlling the operation of the virtual character 902, and the basic control model stored in the basic model storage unit 21 is also a basic control model for controlling the operation of the virtual character 902.

[0122] The virtual space control unit 9 comprises a transmitting / receiving unit 91, a virtual space generation unit 92, and a virtual character control unit 93. The transmitting / receiving unit 91 communicates with the terminal device 94 and exchanges information with the terminal device 94. For example, the transmitting / receiving unit 91 acquires status data from the terminal device 94 indicating the status of user 5 in cooperative operation, and outputs the acquired status data to the virtual space generation unit 92 and the control instruction generation unit 33. The transmitting / receiving unit 91 may also receive status data from the status detection device 8a. In addition, the transmitting / receiving unit 91 transmits, for example, the virtual space information received from the virtual space generation unit 92 (described later) to the terminal device 94.

[0123] The virtual space generation unit 92 generates a virtual space, generates virtual space information to allow the user 5 to perceive the generated virtual space, and outputs the generated virtual space information to the transmission / reception unit 91. The virtual space information includes data representing images (image data) and data representing sounds (sound data). Depending on the cooperative operation and the content of the virtual space, the virtual space information does not necessarily have to include data representing sounds. It may also include at least one of the information that the user 5 can detect by force and touch, smell, and taste. Furthermore, the virtual space generation unit 92 generates virtual space information so that the user 5's avatar 901 in the virtual space performs actions based on the situation data received from the transmission / reception unit 91. In addition, when the virtual space generation unit 92 receives a control instruction from the virtual character control unit 93 (described later), it generates virtual space information so that the virtual character 902 performs actions based on the control instruction.

[0124] The virtual character control unit 93 is an example of an motion control unit (motion control device) for controlling a humanoid. The virtual character control unit 93 includes a control instruction generation unit 33 and a control model storage unit 34. The control model storage unit 34 stores the control model generated by the control model generation unit 2, similar to the first embodiment. This control model is, as described above, a control model for controlling the movements of the virtual character 902. The control instruction generation unit 33 generates a control instruction to control the movements of the virtual character 902 using the status data indicating the user 5's status received from the transmitting / receiving unit 91 and the control model stored in the control model storage unit 34, and outputs the generated control instruction to the virtual space generation unit 92.

[0125] In this embodiment, the controlled object is a virtual character 902 instead of a robot 7. However, similar to Embodiment 1, before the cooperative action between the virtual character 902 and the user 5 takes place, a control model is generated using the user 5's personal behavior data, which includes cooperative action data acquired when the user 5 and the person being coordinated 6 performed a cooperative action. As a result, the virtual character 902 can perform the same or similar actions as the person being coordinated 6 from the start of the cooperative action, thereby reducing the stress on the user 5 caused by at least one of the actions and speech patterns of the virtual character 902 during the cooperative action.

[0126] Furthermore, in the example shown in Figure 16, the virtual character control unit 93 is provided within the virtual space control unit 9, but this is not limited to this. For example, the virtual space control unit 9 may be provided outside the control system 1d as a separate virtual space control device. In this case, the transmitting / receiving unit 91 and the virtual character control unit 93 are provided within the control system 1d, and the virtual space generation unit 92 is provided in the virtual space control device. The virtual space control device also includes a transmitting / receiving unit 91, and control instructions generated by the virtual character control unit 93 are transmitted to the virtual space control device via the transmitting / receiving unit 91 of the control system 1d, and the virtual space generation unit 92 of the virtual space control device receives the control instructions via the transmitting / receiving unit 91 of the virtual space control device. The virtual space generation unit 92 of the virtual space control device transmits the generated virtual space information to the terminal device 94 via the transmitting / receiving unit 91 of the virtual space control device. Alternatively, the transmitting / receiving unit 91 may be provided within the virtual character control unit 93. In this case as well, the virtual character control unit 93 and the control model generation unit 2 may be provided as separate devices.

[0127] The control system 1d of this embodiment is implemented by a computer system, similar to the control system 1 of the first embodiment. The control system 1d of this embodiment may also be implemented by multiple computer systems, for example, by a cloud system. Furthermore, as described above, the control model generation unit 2 and the virtual space control unit 9 may each be configured as separate devices, and in this case as well, each of the separate devices may be implemented by multiple computer systems.

[0128] In the example described above, the control model generation unit 2 of Embodiment 1 generates a control model for controlling the virtual character 902, and the virtual character control unit 93 controls the virtual character 902 using the generated control model. However, when controlling the virtual character 902, an action result acquisition unit 26 may be provided as in Embodiment 3, and the control model may be updated using the action results, or a model selection receiving unit 27 may be used to select a basic control model to be used from a plurality of basic control models for controlling the virtual character 902, as in Embodiment 4. Furthermore, both updating the control model using action results and selecting a basic control model to be used from a plurality of basic control models may be performed.

[0129] Embodiment 6. Figure 17 shows an example configuration of a cooperative operation system according to Embodiment 6. The cooperative operation system 100e of this embodiment includes a control system 1e, a detection device 4a, and a situation detection device 8a. Similar to Embodiment 5, the control system 1e generates a virtual space and transmits virtual space information to the terminal device 94 to allow the user 5 to perceive the virtual space. The situation detection device 8a, terminal device 94, video display device 95, and audio display device 96 are the same as in Embodiment 5. Note that the terminal device 94 may be included in the control system 1e, or the terminal device 94, video display device 95, and audio display device 96 may be included in the control system 1e.

[0130] The control system 1e includes a control model generation unit 2a similar to that in Embodiment 2 and a virtual space control unit 9 similar to that in Embodiment 5. The control model generation unit 2a and the virtual space control unit 9 may each be provided as separate devices. The configuration and operation of the control model generation unit 2a are the same as in Embodiment 2, but the control model generated by the control model generation unit 2a is a control model for controlling the operation of the virtual character 902, and the basic control model stored in the basic model storage unit 21 is also a basic control model for controlling the operation of the virtual character 902. Components having the same functions as in Embodiment 2 or Embodiment 5 are denoted by the same reference numerals as in Embodiment 2 or Embodiment 5, and redundant descriptions are omitted. The following mainly describes the differences from Embodiment 2 or Embodiment 5.

[0131] In this embodiment, similar to Embodiment 2, the control model generation unit 2a generates a control model based on the personal behavior data of the cooperative operation target 6 acquired by the detection device 4a when cooperative operation is being performed with the user 5. This control model is a control model for controlling the operation of the virtual character 902. The virtual character control unit 93 of the virtual space control unit 9 uses the control model generated by the control model generation unit 2a to control the virtual character 902, similar to Embodiment 5.

[0132] Furthermore, as described in Embodiment 5, for example, the virtual space control unit 9 may be provided as a separate virtual space control device outside the control system 1e.

[0133] In this embodiment, the controlled object is a virtual character 902 instead of a robot 7. However, similar to the second embodiment, before the cooperative action between the virtual character 902 and the user 5 takes place, a control model is generated using the personal behavior data of the person to be coordinated, including the cooperative action data acquired when the user 5 and the person to be coordinated 6 performed a cooperative action. As a result, the virtual character 902 can perform the same or similar actions as the person to be coordinated 6 from the start of the cooperative action, thereby reducing the stress on the user 5 caused by at least one of the actions and speech of the virtual character 902 during the cooperative action.

[0134] The control system 1e of this embodiment is implemented by a computer system, similar to the control system 1a of the second embodiment. The control system 1e of this embodiment may also be implemented by multiple computer systems, for example, by a cloud system. Furthermore, as described above, the control model generation unit 2a and the virtual space control unit 9 may each be configured as separate devices, and in this case as well, each of the separate devices may be implemented by multiple computer systems.

[0135] In the example above, we described an example where the virtual character 902 performs cooperative actions and the individual action data includes cooperative action data. However, the applicable actions are not limited to cooperative actions; a control model for the virtual character 902 can be generated using individual action data that reflects the personality of a specific person. In other words, individual action data, which is data relating to the actions of a specific person, is stored in the data storage unit 23, and the learning unit 22a uses the individual action data to generate a control model for the virtual character 902 in the virtual space that reflects the personality of the specific person. For example, if user 5 sets a specific person, and a control model for the virtual character 902 is generated using the individual action data of that person, a control model that reflects the personality of the specific person in line with user 5's wishes is generated. This reduces user 5's stress when conversing with the virtual character 902 or viewing the virtual character 902's words and actions. Furthermore, the method of setting a specific person is not limited to this example. By generating a control model for virtual character 902 based on the personal behavioral data of a specific individual, it is possible to generate a control model that reflects at least one of the behaviors and speech patterns of that specific individual, thereby allowing virtual character 902 to reflect at least one of the individual's personality traits.

[0136] In the example described above, the control model generation unit 2a of Embodiment 2 generates a control model for controlling the virtual character 902, and the virtual character control unit 93 controls the virtual character 902 using the generated control model. However, when controlling the virtual character 902, an action result acquisition unit 26 may be provided as in Embodiment 3, and the control model may be updated using the action results, or a basic control model to be used may be selected from a plurality of basic control models for controlling the virtual character 902 using a model selection reception unit 27 as in Embodiment 4. Furthermore, both updating the control model using action results and selecting a basic control model to be used from a plurality of basic control models may be performed.

[0137] The configurations shown in the above embodiments are merely examples, and it is possible to combine them with other known technologies, combine different embodiments, and omit or modify parts of the configuration without departing from the gist of the invention. [Explanation of Symbols]

[0138] 1,1a,1b,1c,1d,1e Control system, 2,2a,2b,2c Control model generation unit, 3 Robot control unit, 4,4a Detection device, 5 User, 6 Cooperative action target, 7 Robot, 8,8a Situation detection device, 9 Virtual space control unit, 21,21a Basic model storage unit, 22,22a,22b Learning unit, 23 Data storage unit, 24 Data acquisition unit, 25 Correction information storage unit, 26 Action result acquisition unit, 27 Model selection reception unit, 31 Instruction transmission unit, 32 Situation acquisition unit, 33 Control instruction generation unit, 34 Control model storage unit, 91 Transmit / receive unit, 92 Virtual space generation unit, 93 Virtual character control unit, 94 Terminal device, 95 Video presentation device, 96 Audio presentation device, 100,100a,100b,100c,100d,100e Cooperative action system.

Claims

1. A data storage unit that stores personal behavior data, which is data related to a person's actions, A learning unit generates a control model for controlling the movements of a virtual character in a virtual space, which reflects the individuality of the person, using the personal behavior data stored in the data storage unit. Equipped with, The aforementioned operation includes the coordinated operation of the virtual character with the avatar of the person in the virtual space. A control model generation device characterized in that the personality of the person includes at least one of the person's behavior and manner of speaking.

2. The control model generation device according to claim 1, characterized in that the personality of the person includes at least one of the trajectory and speed of movement of the person.

3. The control model generation apparatus according to claim 1, characterized in that the personality of the person includes at least one of the person's speaking speed, intonation, and dialect.

4. Control model generation unit, Operation control unit and Equipped with, The control model generation unit, A data storage unit that stores personal behavior data, which is data related to a person's actions, A learning unit generates a control model for controlling the movements of a virtual character in a virtual space, which reflects the individuality of the person, using the personal behavior data stored in the data storage unit. Equipped with, The aforementioned operation includes the coordinated operation of the virtual character with the avatar of the person in the virtual space. The personality of the person includes at least one of the person's behavior and manner of speaking. The control system is characterized in that the motion control unit controls the movement of the virtual character in the virtual space using the control model.

5. A method for generating a control model in a control model generation device, An accumulation step to accumulate personal behavior data, which is data related to a person's actions, A generation step, using the personal behavior data accumulated in the accumulation step, generates a control model for controlling the movements of a virtual character in a virtual space, the control model reflecting the individual's personality. Includes, The aforementioned operation includes the coordinated operation of the virtual character with the avatar of the person in the virtual space. A method for generating a control model, characterized in that the personality of the person includes at least one of the person's behavior and manner of speaking.

6. In a computer system, An accumulation step to accumulate personal behavior data, which is data related to a person's actions, A generation step, using the personal behavior data accumulated in the accumulation step, generates a control model for controlling the movements of a virtual character in a virtual space, the control model reflecting the individual's personality. A program that executes, The aforementioned operation includes the coordinated operation of the virtual character with the avatar of the person in the virtual space. The personality of the person is characterized by including at least one of the person's behavior and manner of speaking.

Images