Action processing device, action processing method and action processing program

Abstract

To more appropriately evaluate an action about communication.SOLUTION: In an action processing device for evaluating an action about communication of a group taking the communication among action main bodies, the action processing device 1 includes a group state index calculation part 104 for calculating a second group state index about a state of communication by a second group on the basis of second sensing information of the second group of an evaluation object, a vector information generation part 105 for generating visible information on the basis of the second group state index, vectorizing the visible information and generating second vector information, and an action evaluation part for evaluating an action of the second group on the basis of a first group state index associated with first vector information that satisfies a condition with a comparison result with the second vector information determined in advance in the first vector information to be reference of a knowledge database.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technique for evaluating behavior related to communication between actors. [Background technology]

[0002] Communication assessments are now being conducted in a variety of fields. For example, medical professionals are being tested on interpersonal communication. In other two-way communication, such as between counselor and client, sales staff and customers, or leader and follower, training in interview techniques is essential for the leading party to effectively influence the following party. For example, counselors receive training in interview techniques during their education courses through video instruction on good counseling and role-playing.

[0003] To conduct good communication, including in interviews, it is essential to master not only verbal responses such as speech content, but also nonverbal response techniques, which are a type of nonverbal behavior. For example, it is known that in counseling sessions that are evaluated as good, there is high synchronization between verbal response latencies and physical behavior between the counselor and client. However, compared to verbal responses, the quality of nonverbal response techniques cannot be adequately acquired through training such as watching videos or role-playing; rather, they are highly individual skills that must be acquired through trial and error.

[0004] Therefore, Patent Document 1 has been proposed as a technology for "appropriately supporting a person to be scored when scoring interpersonal communication that requires dialogue skills." Patent Document 1 discloses "an information processing device that includes a processing unit that scores a dialogue between a first speaker in a first space and a second speaker in a second space different from the first space based on reference information that serves as a standard for scoring the dialogue, and presents scoring information related to the scoring of the dialogue to the first speaker in real time." [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2022 / 102432 Summary of the Invention [Problem to be solved by the invention]

[0006] Patent Document 1 is biased toward a scoring method for verbal response techniques in fields such as doctors and pharmacists, where it is easy to clearly state and provide examples of what the person being scored should say. As a result, the evaluation of non-verbal response techniques using sensing information is insufficient. In particular, when rapport building is immature in the early stages of counseling, the client's response to the counselor's (the person being scored) utterances may contradict their actual thoughts, making it difficult to make a more appropriate evaluation.

[0007] For this reason, there is a risk that intervention support for communication-related behavior will be insufficient in Patent Document 1. Therefore, an object of the present invention is to perform a more appropriate evaluation of communication-related behavior. [Means for solving the problem]

[0008] In order to achieve this object, the present invention employs a behavior processing device for evaluating communication-related behavior of a group of actors communicating with each other, the behavior processing device having: a memory unit that stores a knowledge database that associates a first group state index related to the state of communication in a first group with first vector information generated based on the first group state index and indicating characteristics of the behavior; an input unit that accepts second sensing information that indicates communication-related behavior in a second group; a group state index calculation unit that calculates a second group state index related to the state of communication in the second group based on the second sensing information; a vector information generation unit that generates visual information that indicates the state of communication in the second group based on the second group state index and vectorizes the visual information to generate second vector information; and a behavior evaluation unit that evaluates the behavior of the second group based on the first group state index associated with first vector information, among the first vector information included in the knowledge database, whose comparison result with the second vector information satisfies a predetermined condition.

[0009] The present invention also includes a behavior processing method executed by the behavior processing device, a behavior processing program for causing the behavior processing device to function as a computer, and a storage medium storing such a program.Furthermore, the present invention also includes a behavior processing system including the behavior processing device. [Effects of the Invention]

[0010] According to the present invention, it is possible to more appropriately evaluate communication-related behavior. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a functional block diagram of a behavior processing device 1 according to an embodiment of the present invention. [Figure 2] FIG. 1 is a configuration diagram illustrating an implementation example of a behavior processing system according to a first embodiment. [Figure 3A]FIG. 8 is a diagram showing sensing information 81 used in the first embodiment. [Figure 3B] FIG. 10 is a diagram illustrating a sensing feature amount 82 used in the first embodiment. [Figure 3C] FIG. 10 is a diagram showing a group state index 83 used in the first embodiment. [Figure 3D] FIG. 8 is a diagram showing additional information 84 used in the first embodiment. [Figure 3E] FIG. 2 is a diagram showing a knowledge database 85 used in the first embodiment. [Figure 3F] FIG. 10 is a diagram showing the actor characteristic information 87 used in the first embodiment. [Figure 3G] FIG. 1 is a diagram showing an intervention determination method 90 used in the first embodiment. [Figure 4] 10 is a flowchart illustrating a sensing information collection process according to the first embodiment. [Figure 5] 10 is a flowchart showing a process of building a knowledge DB 85 in the first embodiment. [Figure 6] 10 is a flowchart showing a process for generating explanatory text and vector information in the first embodiment. [Figure 7] 10 is a flowchart showing a behavior evaluation process in the first embodiment. [Figure 8] 10 is a flowchart showing details of step S74 of the behavior evaluation process in the first embodiment. [Figure 9] FIG. 10 is a diagram showing a display screen of an intervention action in the first embodiment. [Figure 10] FIG. 10 is a diagram showing a display screen of another intervention action in the first embodiment. [Figure 11] FIG. 10 is a configuration diagram showing an implementation example of a behavior processing system according to a second embodiment. [Figure 12] 10 is a flowchart showing a behavior evaluation process in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] In this embodiment, behavior related to communication between agents in a group composed of multiple agents is evaluated. These agents have relationships such as counselor-client, sales staff-customer, and leader-follower. The agents include not only so-called natural persons but also autonomously acting virtual humans (hereinafter referred to as virtual humans) such as robots, avatars, and chatbots (automated conversation programs). Communication includes not only one-to-one communication but also N-to-M communication. Note that this communication also includes communication between three or more units such as one-to-one-to-one, for example, facilitation of a conversation at a round table. For this reason, the group targeted in this embodiment is composed of two or more agents.

[0013] The behavior processing device 1 that executes the processing in this embodiment will be described below. Figure 1 is a functional block diagram of the behavior processing device 1 in this embodiment. The behavior processing device 1 has an input unit 101, an output unit 102, a preprocessing unit 103, a group state index calculation unit 104, a vector information generation unit 105, a behavior evaluation unit 106, an intervention behavior generation unit 107, a control command unit 108, and a memory unit 109.

[0014] First, the input unit 101 receives sensing information indicating the behavior of the subject. The sensing information includes non-verbal information indicating non-verbal behavior as behavior. The non-verbal behavior indicated by the non-verbal information includes the subject's posture, movements such as nodding, tone and volume of voice, heart rate, body temperature, blinking, eye movement, pupils, electromyography, and brain waves. Therefore, the non-verbal information includes not only conscious behavior but also reactions. Furthermore, if the subject is a virtual human, it also includes internal states such as the robot's joint angles and the virtual movements of the avatar. Furthermore, audio-related information, including the content of the voice, may be selectively used as sensing information.

[0015] Furthermore, the sensing information may include language information. Furthermore, the sensing information is not limited to information detected by a sensor. For example, information indicating that a button is pressed by an agent when a specific condition is met is also included in the sensing information. The input unit 101 can be realized by an input device such as a keyboard or a communication device. Furthermore, the input unit 101 can be realized by a program for operating the input device or communication device, or a combination of the program and modules that constitute the input device or communication device.

[0016] Furthermore, the output unit 102 outputs the evaluation results from the action evaluation unit 106 and the intervention action generated by the intervention action generation unit 107. For this reason, the output unit 102 can be realized by a display device such as a monitor or a communication device. Furthermore, the output unit 102 can be realized by a program for operating the display device or communication device or a combination with modules constituting the program.

[0017] The preprocessing unit 103 also performs preprocessing on the received sensing information to enable subsequent processing. This preprocessing includes noise reduction, outlier processing, and feature extraction. The group state index calculation unit 104 also calculates, based on the sensing information, a group state index that evaluates the state of communication in the group from which the sensing information was detected (hereinafter simply referred to as a group state index related to the state of communication). Here, the group state index can be at least one of a synchronization index and an information flow index of the subject of behavior. For this purpose, the group state index calculation unit 104 preferably calculates the group state index from the sensing feature.

[0018] Furthermore, the vector information generation unit 105 generates visual information indicating the state of communication in the group based on the sensing information and the group state index, and vectorizes the visual information to generate vector information. Here, the visual information is information that visualizes the state of communication in the group, and at least one of explanatory text and video information can be used. Here, video information is image information having time information (a time element) and is not limited to a single unit (e.g., a file) but may be realized by multiple still image information. Furthermore, the visual information may be editable in response to user operations. Note that it is preferable that the vector information generation unit 105 generates the visual information using sensing features and the group state index generated from the sensing information.

[0019] Furthermore, the behavior evaluation unit 106 evaluates the behavior of the group of one of the vector information based on the comparison result of the two pieces of vector information. As an example, a knowledge database 85 is used in advance. The knowledge database 85 associates first sensing information indicating behavior related to communication in the first group, a first group state index related to the state of communication in the first group, and first vector information. The first vector information indicates the characteristics of the behavior of the subject of the behavior in the first group, and is generated based on the first sensing information and the first group state index.

[0020] Then, the behavior evaluation unit 106 evaluates the behavior of the second group. To this end, the behavior evaluation unit 106 performs the evaluation based on a first group state index associated with first vector information in the knowledge database 85, the difference between which is within a predetermined threshold value from the second vector information received by the input unit 101. Furthermore, the behavior evaluation unit 106 may identify the behavior of the group of one of the vector information based on a comparison result of the two vector information.

[0021] The intervention action generation unit 107 generates an intervention action for the acting subject based on at least one of the evaluation result and the action by the action evaluation unit 106. The intervention action includes advice regarding the action. The control instruction unit 108 generates a control instruction for the virtual human in accordance with the intervention action generated by the intervention action generation unit 107 or the action identified by the action evaluation unit 106. The control instruction unit 108 can be omitted.

[0022] The storage unit 109 also stores sensing information 81, sensing feature amounts 82, group state indices 83, additional information 84, a knowledge database 85 (knowledge DB 85), intervention candidate plans 86, agent characteristic information 87, intervention determination method 90, explanation support model 91, and processing model 92. These will be described in the examples below. This concludes the description of this embodiment, and examples 1 and 2 will be described below, which specifically illustrate this embodiment. [Example]

[0023] In Example 1, a counselor and a client are used as the main actors (also referred to as actors) that make up a group. Here, communication such as consultation and counseling about worries and the like takes place between the counselor and the client. When the client becomes nervous in response to worries, nonverbal behavior such as an increased heart rate and a higher-pitched voice occurs. Furthermore, such nonverbal behavior may occur during communication such as consultation or counseling, or may occur before or after communication. For this reason, in Example 1, nonverbal behavior before and after communication can also be used.

[0024] Moreover, Example 1 is an example in which the behavior of a group is evaluated by a behavior processing system including a behavior processing device 1. Below, the hardware configuration of the behavior processing system will be described with reference to FIG. 2. FIG. 2 is a configuration diagram showing an implementation example of the behavior processing system in Example 1. In the behavior processing system, a behavior processing device 1 is connected via a network 9 to a counselor terminal 7-1 and a client terminal 7-2 used by behavior subjects. Then, the behavior processing device 1 evaluates the behavior of a group made up of counselors and clients who use the counselor terminal 7-1 and the client terminal 7-2, respectively.

[0025] First, the behavior processing device 1 can be realized by a computer called a server, and is installed in the same location as the counselor terminal 7-1 and the client terminal 7-2, or in a data center, etc. The behavior processing device 1 has a processor 2, a memory 3, a storage device 4, an input / output device 5, and a communication device 6, which are connected to each other via a communication path such as a bus.

[0026] The processor 2 is also called an arithmetic unit or a processing unit, and executes the processing of each unit in FIG. 1 according to various programs described below. The memory 3 is also called a main storage unit, and for processing by the processor 2, programs stored in a storage medium such as a storage device 4 and information used in the processing by these programs are deployed in the memory 3. That is, as shown in FIG. 2, a behavior processing program 30 is deployed in the memory 3.

[0027] The behavior processing program 30 has a reception module 31, a preprocessing module 32, a group state index calculation module 33, a vector information generation module 34, a behavior evaluation module 35, an intervention behavior generation module 36, and a notification module 37. These modules cause the processor 2 to execute the functions of each module in FIG. 1, and the correspondence between them is as follows: Input unit 101: Reception module 31 Output unit 102: Notification module 37 Preprocessing unit 103: Preprocessing module 32 Population state index calculation unit 104: Population state index calculation module 33 Vector information generation unit 105: Vector information generation module 34 Behavior Evaluation Section 106: Behavior Evaluation Module 35 Intervention behavior generation unit 107: Intervention behavior generation module 36 Therefore, the processor 2's behavior processing program 30 executes the processing of an input unit 101, an output unit 102, a preprocessing unit 103, a group state index calculation unit 104, a vector information generation unit 105, a behavior evaluation unit 106, an intervention behavior generation unit 107, and a control command unit 108. Note that with regard to the input unit 101 and the output unit 102, the reception module 31 and the notification module 37 cause the input / output device 5 and the communication device 6 to realize their functions. Furthermore, each of these modules may be configured as an independent program, or may be realized as a program combining parts of them.

[0028] The behavior processing program 30 is distributed via the network 9 or stored in a storage medium and installed in the behavior processing device 1. The behavior processing device 1 may be realized as a business support device that supports the work of a counselor. In this case, the behavior processing program 30 may be realized as one function of the business support program.

[0029] The storage device 4 is also referred to as a secondary storage device, and can be realized by a storage such as a hard disk drive, and stores the behavior processing program 30 and various information (sensing information 81, etc.). Thus, the storage device 4, together with the memory 3, corresponds to the storage unit 109 in FIG. 1. Here, the storage device 4 stores the following information as information used in the first embodiment: sensing information 81, sensing feature amount 82, group state index 83, additional information 84, knowledge database 85 (knowledge DB 85), intervention candidate plan 86, agent characteristic information 87, intervention determination method 90, explanation support model 91, and processing model 92. At least one of the various information in the storage device 4 may be stored in a database system or file device in a housing separate from the behavior processing device 1.

[0030] The input / output device 5 is an input device and a display device, and may be configured as separate devices or may be integrated into one device such as a touch panel. In the example of FIG. 2, the behavior processing device 1 is a computer that can be used by a user such as a supervisor, and therefore the input / output device 5 is provided. However, if the behavior processing device 1 is implemented as a large computer installed in a data center, the input / output device 5 can be omitted. The input / output device 5 may also be implemented as a terminal device (a computer such as a tablet) in a separate housing from the behavior processing device 1. The input / output device 5 cooperates with the reception module 31 and the notification module 37 to execute the functions of the input unit 101 and the output unit 102 in FIG. 1.

[0031] The communication device 6 also has an interface function for connecting to the network 9 and communicates with the counselor terminal 7-1 and the client terminal 7-2. Therefore, the communication device 6 also cooperates with the reception module 31 and the notification module 37 to execute the functions of the input unit 101 and the output unit 102 in FIG.

[0032] Next, the counselor terminal 7-1 is a terminal device used by the counselor, who is the actor 1, as a user, and can be realized by a computer such as a PC, smartphone, tablet terminal, etc. The counselor terminal 7-1 has an actor measuring instrument 11-1, a sensing device 12-1, an input / output device 13-1, a communication device 14-1, and a notification / announcement device 15-1, which are connected to each other via a communication path.

[0033] First, the actor measuring device 11-1 measures the behavior of the counselor, who is actor 1. For this purpose, the actor measuring device 11-1 can be realized by a biosensor 21-1 that measures the counselor's pulse, etc., a camera 22-1, etc. The camera 22-1 includes a camera that photographs the counselor's appearance and a so-called thermograph, etc., and can measure the counselor's physical movements (including head movement) and body temperature.

[0034] Furthermore, the sensing device 12-1 creates sensing information indicating the behavior of the actor 1 according to the measurement results of the actor measuring device 11-1. The input / output device 13-1 is an input device and a display device for the counselor, and may be configured as separate devices or may be configured as an integrated device such as a touch panel.

[0035] The communication device 14-1 has an interface function for connecting to the network 9 and communicates with the behavior processing device 1 and the client terminal 7-2. The notification / alert device 15-1 outputs processing details such as behavior evaluation results in the behavior processing device 1 and sensing information created by the sensing device 12-1.

[0036] Next, the client terminal 7-2 is a terminal device used by the client (actor 2) as a user, and can be realized by a computer such as a PC, smartphone, tablet terminal, or wearable computer. The client terminal 7-2 has an actor measurement device 11-2, a sensing device 12-2, an input / output device 13-2, and a communication device 14-2, which are connected to each other via a communication path. The actor measurement device 11-2, the sensing device 12-2, the input / output device 13-2, and the communication device 14-2 have the same functions as the actor measurement device 11-1, the sensing device 12-1, the input / output device 13-1, and the communication device 14-1 of the counselor terminal 7-1. However, the measurement target of the actor measurement device 11-2 is the client.

[0037] The actor measuring device 11-1 and the actor measuring device 11-2 may each be realized in a separate housing from the counselor terminal 7-1 and the client terminal 7-2. Furthermore, with regard to configurations other than the actor measuring device 11, the counselor terminal 7-1 and the client terminal 7-2 may each be configured in multiple housings, including a main body and a wearable computer. Furthermore, the input / output device 13-1 and the input / output device 13-2 may function as the actor measuring device 11-1 and the actor measuring device 11-2. For example, when a counselor or a client feels a specific emotion, the behavior of the counselor or the client can be measured by pressing a specific button.

[0038] The network 9 connects the behavior processing device 1, the counselor terminal 7-1, and the client terminal 7-2. The network 9 may be configured as either a wide area network such as the Internet or a local area network such as a LAN, or may be configured as multiple networks.

[0039] This concludes the explanation of the configuration of the first embodiment, and the following describes the information used in the first embodiment, that is, the information stored in the storage device 4 and the processing flow. In this case, the processing content will be mentioned, and the processing subject basically uses the configuration of Fig. 1, but it is clear from the above explanation of the relationship between Fig. 1 and Fig. 2 that the processing can also be performed with the configuration of Fig. 2.

[0040] 3A is a diagram illustrating sensing information 81 used in the first embodiment. In the first embodiment, sensing information 81-1 indicating a vector quantity and sensing information 81-2 indicating a scalar quantity are used as the sensing information 81. Either one of these may be used, or these may be managed as one piece of sensing information 81.

[0041] First, the sensing information 81-1 is information that indicates the behavior of each acting subject (actor) as a vector quantity. For this reason, the sensing information 81-1 has the following items: ID, Session ID, User ID, Data Type, Datetime, F1, F2 (hereafter, up to Fn). The ID is an item for identifying the sensing information 81-1, and a unique identification code (e.g., a number) is assigned to each record. Furthermore, the Session ID indicates the measurement unit for the acting subject's behavior. For example, the measurement unit can be a unit of communication such as one counseling session (one session).

[0042] Furthermore, User ID indicates the subject (actor) of the measured behavior, and in the case of Example 1, identification information indicating either the counselor or the client is recorded. Furthermore, Data Type indicates the sensing information 81-1, that is, the type and target of the measured behavior. In the example of FIG. 3A, the coordinates of the face "landmark" are used. Furthermore, Datetime is time information related to the measurement, and the time of measurement, the time when the sensing information 81-1 was acquired, etc. can be used.

[0043] Furthermore, F1 to Fn indicate coordinates measured for each part of the face. For example, coordinates for each distinctive part are used, such as F1 for the nose and F2 for the eyes. Then, as time passes, records with ID=1, 2, 3, etc. are recorded for the landmarks of agent A. As a result, the sensing information 81-1 shows the behavior of agents A and B over time. Note that there is no limit to the number of F1 to Fn, as long as it is 1 or greater.

[0044] The sensing information 81-2 is information that indicates the behavior of each actor using scalar quantities. Similarly to the sensing information 81-1, the sensing information 81-2 has the following fields: ID, Session ID, User ID, Data Type, Datetime, F1, and F2 (hereafter, Fn). Differences from the sensing information 81-1 are explained below. Data Type indicates the sensing information 81-2, which is indicated by a scalar quantity, i.e., the type and target of the measured behavior. In the example of FIG. 3A, "HR," or heart rate, is used. The heart rate (e.g., per minute), which is an example of a measured scalar quantity, is recorded in F1. When the sensing information 81-1 and the sensing information 81-2 are treated as a single piece of information, it is desirable to record information that distinguishes between vector quantities and scalar quantities in the Data Type, etc.

[0045] 3B is a diagram showing the sensing feature 82 used in the first embodiment. The sensing feature 82 is generated from the sensing information 81 by the preprocessing unit 103 and is information indicating the characteristics of a behavior. Therefore, the sensing feature 82 has the following items: ID, Session ID, User ID, Data Type, Datetime, and Value 1 (hereafter, up to Value n).

[0046] The ID is an item for identifying the sensing feature 82, and a unique identification code (e.g., a number) is assigned to each record. The session ID indicates the unit of measurement for the behavior of the subject, and the session ID of the sensing information 81 from which the sensing feature 82 was generated is used.

[0047] Similarly to the sensing information 81, the User ID indicates the subject (actor) of the measured action. The Data Type indicates the type of the sensing feature 82. In the example of FIG. 3B, the face tilt "TILT deg" is used. This is because the face coordinates are used as the sensing information 81-1 of FIG. 3A, and the tilt can be calculated from these. That is, the pre-processing unit 103 calculates the sensing feature 82 (face tilt) from the sensing information 81-1 (face coordinates). The Datetime is time information related to the measurement, and the Datetime in the sensing information 81-1 is used.

[0048] Furthermore, Value 1 to Value n each indicate a sensing feature, for example, the inclination (angle) of the face. Note that the number of Values ​​1 to n is not limited as long as it is 1 or more. Furthermore, when multiple Values ​​are used, each item can be used for each direction (x, y, z axes, etc.).

[0049] 3C is a diagram showing the group status index 83 used in Example 1. The group status index 83 is an index related to the state of communication between the group, i.e., between the acting entities such as the counselor and the client. Therefore, the group status index 83 has the following items: ID, Session ID, UID 1, DID 1, UID 2, DID 2, Data Type, Datetime, Value, and Event.

[0050] First, the ID is an item for identifying the group state index 83, and a unique identification code (e.g., a number) is assigned to each record. The Session ID indicates the unit of measurement for the behavior of the subject, and the Session ID of the sensing information 81 or sensing feature 82 from which the group state index 83 was generated is used.

[0051] Furthermore, UID 1 indicates the actor that is communicating and taking action. For this reason, UID 1 uses the User ID of the sensing information 81 or sensing feature 82 that is the measured actor. DID 1 identifies the data of the actor of UID 1. In other words, the ID of the sensing information 81 or sensing feature 82 is used.

[0052] Furthermore, UID 2 indicates the other actor that is communicating with UID 1 and taking action. For this reason, UID 2 also uses the User ID of the sensing information 81 or sensing feature 82, which is the measured actor. DID 2 identifies the data of the actor of UID 2. In other words, the ID of the sensing information 81 or sensing feature 82 is used. Note that DID 1 and DID 2 can be associated with each other because the counselor terminal 7-1 and the client terminal 7-2 are time-synchronized. As a result, each UID (User ID) and Session ID can also be identified from the ID corresponding to the DID.

[0053] Furthermore, Data Type indicates the type of group state index 83. In the example of FIG. 3C, "TILT_IPC_VLF", i.e., face tilt synchronicity, is used. Furthermore, Datetime is time information related to measurement, and the Datetime in the sensing information 81-1 or the sensing feature amount 82 is used. As described above, the time related to measurement is synchronized.

[0054] Furthermore, Value indicates the numerical value of TILT_IPC_VLF (synchronization of face tilt). This is calculated by the group state index calculation unit 104 from the face tilt of each of the counselor (A) and client (B) in the sensing feature 82. For this purpose, the group state index calculation unit 104 uses techniques such as frequency analysis. This makes it possible to evaluate the relationship between the sensing feature 82 or sensing information 81 of each counselor and client, that is, the communication therebetween. In the example of FIG. 3C, it can be seen that the numerical value for ID=1 is the lowest, and the degree of synchronization tends to improve over time (ID=4 is the highest).

[0055] In addition, in the Event, at least a part of additional information 84 (data information) relating to the evaluation result, intervention action according to the evaluation result, and the evaluation is recorded. These will be described later.

[0056] Next, FIG. 3D is a diagram showing additional information 84 used in Example 1. The additional information 84 is information about communication between a group, i.e., between actors such as a counselor and a client, and can be used as reference data or learning data for evaluation. Therefore, the additional information 84 has the following items: ID, Session ID, UID 1, DID 1, UID 2, DID 2, Key 1, Value 1, Key 2, Value 2, and Description. The ID, Session ID, UID 1, DID 1, UID 2, and DID 2 are the same as the group status index 83.

[0057] Furthermore, Key 1 and Value 1, and Key 2 and Value 2, respectively, indicate the relationship between the behavior of the actors. In FIG. 3D, when ID=1, Key 1 and Value 1 indicate that the group, i.e., counselor (A) and client (B), has a synchrony of 1 and a mimicry of 0. These values ​​are recorded below for each ID. In Example 1, UID 1, DID 1, UID 2, and DID 2 are recorded because the additional information 84 is created using measurement results of actual communication. When additional information 84 is created manually, these may be omitted, or the role (counselor, client, etc.) may be recorded as UID 1 or UID 2. In addition, an explanation of the corresponding record is recorded in Description.

[0058] Next, FIG. 3E is a diagram showing a knowledge database 85 (knowledge DB85) used in the first embodiment. In the knowledge DB85, at least sensing information, population state indexes, and vector information are associated with each other. In the first embodiment, explanatory text, which is an example of visual information for generating vector information, is also associated with each other. In FIG. 3E, multiple pieces of knowledge data (each index in the figure) are shown. Then, "Vector:" indicates vector information. Also, "text:" indicates explanatory text for generating the corresponding vector information. Furthermore, "metadata:" indicates corresponding additional information. Also, "features:" indicates corresponding sensing features. These correspond to one record in the knowledge DB85. Note that these are merely examples, and the knowledge DB85 may also include, for example, sensing information.

[0059] Next, the intervention candidate plan 86 is information indicating the intervention candidate plan generated in Example 1. Next, FIG. 3F is a diagram showing the actor characteristic information 87 used in Example 1. The actor characteristic information 87 is information about the actor. Therefore, the actor characteristic information 87 has the following items: UID, registration date, last update date, age, gender, role, and personality (neuroticism). The UID identifies the actor. Note that the actor is not limited to a natural person, but also includes a virtual person. The registration date and last update date indicate the dates when the corresponding record was registered or last updated, respectively. The age, gender, role, and personality (neuroticism) are items that indicate the attributes and characteristics of the corresponding actor. Note that the role indicates the role of each actor. Note that the role of UID=C indicates a virtual human and a virtual person, but the role played by the virtual human (e.g., product explainer) may also be recorded.

[0060] 3G is a diagram showing an intervention determination method 90 used in Example 1. The intervention determination method 90 records rules for determining whether to intervene depending on the behavior. For example, the rules specify conditions such as when the face tilt is xx degrees or more, when the synchronization between the behavioral agents is yy or more, when the group state index 83 is a predetermined value, etc.

[0061] Next, the explanation support model 91 is a model for generating intervention candidates from visual information such as explanatory text, vector information, additional information, etc. The intervention candidates indicate intervention actions performed by actors such as virtual humans (such as product presenters) or counselors. It is desirable that intervention candidates for natural people be expressed in natural language, and it is desirable that intervention candidates for virtual humans be created in a format that allows control over them. Next, the processing model 92 is a model used to generate visual information such as explanatory text and video image information from the group state index 83.

[0062] This concludes the description of the information used in the first embodiment, and next, a description will be given of the processing flow of the first embodiment. First, FIG. 4 is a flowchart showing the sensing information collection processing in the first embodiment. In step S41 of FIG. 4, the input unit 101 receives sensing information 81 from the counselor terminal 7-1 or the client terminal 7-2. Then, the preprocessing unit 103 stores this as the sensing information 81 in the storage unit 109. At this time, the preprocessing unit 103 may access the actor characteristic information 87 and identify the actor corresponding to the received sensing information 81. In step S41, the processor 2 receives the sensing information 81 via the communication device 6 in accordance with the receiving module 31 of FIG. 2. The receiving of the sensing information 81 may be performed in either a pull type or a push type.

[0063] 5 is a flowchart showing the construction process of the knowledge DB 85 in the first embodiment. In step S51, the preprocessing unit 103 reads the target sensing information 81 from the storage unit 109. In step S51, the sensing information 81 measured by the actor measuring device 11 (various sensors), received from the counselor terminal 7-1 and the client terminal 7-2, and stored in the storage unit 109 is read. The following describes a case where a group consists of two people, a counselor and a client. In this case, video recording the communication of the entire group and information measured by a depth sensor are read as the sensing information 81.

[0064] Furthermore, for each of the counselor and the client, video recording of head movements and biometric information such as heart rate, electrodermal activity, and acceleration recorded by biometric sensors worn by each are read as sensing information 81. The sensing information 81 is acquired by a smartphone or a PC, which are examples of the counselor terminal 7-1 and the client terminal 7-2.

[0065] Furthermore, when handling video for the purpose of recording head and facial behavior, the video data as well as head and facial feature point data extracted from the video data are treated as sensing information 81. Such head and facial feature point data can be created by the sensing device 12 of the counselor terminal 7-1 or the client terminal 7-2. Note that the biosensor 21 of the actor measurement device 11 that measures these is not limited to the above, and other sensors that detect body temperature, blinking, eye movement, electromyography, or brain waves can also be used. The biosensor 21 can be a wearable computer (device) that can be worn by the subject of the action, or a system built into a smartphone that the subject of the action can carry around.

[0066] Furthermore, in step S52, the preprocessing unit 103 calculates sensing feature quantities 82 from the read sensing information 81. More preferably, the preprocessing unit 103 performs preprocessing such as noise reduction and outlier processing on the read sensing information 81, performs feature extraction processing, and generates sensing feature quantities 82 for each of the acting subjects (actors) constituting the group.

[0067] First, an example of preprocessing such as noise reduction will be described. Furthermore, the preprocessing unit 103 may perform normalization of the signal scale to be used or noise removal on the sensing information 81 for the purpose of correcting for individual differences among actors and reducing noise. For example, the following normalization processes may be performed for each actor or for each measurement date, for each measurement date for an actor, or between actors. Examples of this normalization include min-max normalization, which normalizes using the maximum and minimum values; z-score normalization, which normalizes using the mean and standard deviation of the signal; and quantile normalization, which normalizes using the quantile of the signal intensity distribution.

[0068] Furthermore, if it is assumed from the age of the agent characteristic information 87 that the signal strength will fluctuate due to aging or other reasons, the pre-processing unit 103 may perform a standard deviation process to normalize the signal strength for each age group.

[0069] Furthermore, the preprocessing unit 103 may perform the following noise removal processes: clipping or winsorizing to remove abnormal values ​​of the signal and place them within a certain range, moving average processing to suppress and smooth sudden fluctuations at a single time, or zero-order differentiation processing using a Savitzky-Golay filter.

[0070] Furthermore, the preprocessing unit 103 may perform upsampling or downsampling using an analog filter or a digital filter in order to reduce the calculation load in subsequent processing or to unify the time granularity between pieces of sensing information. This concludes the explanation of preprocessing such as noise reduction.

[0071] Then, the preprocessing unit 103 performs feature extraction processing on the preprocessed sensing information 81 to generate sensing feature amounts 82 related to each of the actors constituting the group. An example of the feature amount extraction processing will be described below.

[0072] It is also possible to perform feature extraction processing according to the biosignals in the biomeasurement data that is the source of the sensing information 81. For example, the following feature amounts can be extracted from heartbeat interval data acquired by a heart rate sensor, which is an example of the actor measuring instrument 11. Average heart rate over a given time window. - The Low Frequency component (LF), which is obtained by frequency domain analysis and is known to primarily reflect sympathetic nervous activity, and the High Frequency component (HF), which is known to primarily reflect parasympathetic nervous activity. -SDNN, RMSSD, and NN50 are used in time domain analysis. Use of features obtained using Lorenz plots used in nonlinear domain analysis, features obtained by detrended flux analysis, and features obtained by complex demodulation method.

[0073] Furthermore, skin conductance level (SCL) or skin conductance response (SCR) may be used for electrodermal activity data acquired by an electrodermal activity sensor, which is an example of the actor measuring instrument 11. In the case of triaxial acceleration data acquired by an acceleration sensor, which is an example of the actor measuring instrument 11, feature quantities can be extracted as follows. That is, the acceleration norm or the number of zero crossings, which is the number of times per predetermined time period that a signal processed by a band-pass filter for the acceleration norm passes through a threshold of ±0.01 G when the gravitational acceleration is 1 G, can be used.

[0074] Furthermore, when facial feature point data is measured, three-dimensional Euler angles of head behavior may be calculated using the feature point coordinates, and the obtained Euler angles, such as NOD / SHAKE / TILT, may be used as feature quantities. In this case, for example, the three-dimensional head Euler angle in the tilt direction is calculated as TILT_deg. In this way, the preprocessing unit 103 calculates, from the sensing information 81, sensing feature quantities 82 indicating the state of each subject constituting the group. The preprocessing unit 103 then stores these in the memory unit 109.

[0075] In step S53, the group state index calculation unit 104 calculates a group state index 83 from the sensing feature amount 82 generated from the sensing information 81. An example of this will be described below. There are various known group state indices for evaluating the quality of communication and various display methods using the indices.

[0076] For example, in bilateral counseling, typical counseling strategies and phenomena reflecting communication states, such as synchrony, echoing, mimicry, mirroring, and reflection, are known. In the first embodiment, a state index is calculated based on these. That is, the group state index calculation unit 104 calculates a group state evaluation index that reflects these states from the sensing feature amount 82. Below, an example of calculating a mind-body synchronization index that reflects synchrony as a group state index will be described using bilateral counseling as an example.

[0077] It is known that there are several types of mind-body synchronization indices. Therefore, below we will explain the interpersonal physiological coherence based on wavelet transform coherence, which measures synchrony in the time-frequency domain, and transfer entropy, which measures the state of information transfer. In other words, we will explain how the group state index calculation unit 104 calculates the group state index using these indices.

[0078] First, we will explain Wavelet Transform Coherence (WTC), which is the first example of a mind-body synchronization index. WTC is a method for quantifying frequency synchrony in time-frequency space based on wavelet transform. When a signal xn is given a frequency scale s and a time shift n, its continuous wavelet transform can be expressed as follows (Equation 1).

[0079]

number

[0080] In this case, * in (Equation 1) indicates a complex conjugate, and φ0 indicates a wavelet function. Here, (Equation 2), which is a Morlet wavelet with ω0 = 6.0, is used as a representative wavelet function.

[0081]

number

[0082] Furthermore, when Wx(s, n) and Wy(s, n) are calculated for the two signals xn and yn, respectively, and smoothed in the time-frequency domain, WTC can be calculated using Equation 3.

[0083]

number

[0084] Furthermore, WTC includes a region known as the Cone of Influence (COI), where the analysis reliability is low. For this reason, in this analysis, the WTC in the region not corresponding to the COI can be averaged over a predetermined time-frequency range to obtain Interpersonal Physiological Coherence (IPC), which can be used as a mind-body synchronization index. For example, the group state index calculation unit 104 may use a time window with a fixed time interval of 30 seconds and perform the following processing. That is, the 0.004 to 0.04 Hz band may be defined as the VLF band, the 0.04 to 0.15 Hz band as the LF band, and the 0.15 to 0.40 Hz band as the HF band, and the five conditions of the LF band, HF band, TP (= LF + HF) band, VLF band, and all bands may be averaged in the time-frequency direction.

[0085] Then, the group state index calculation unit 104 can use TILT_IPC_VLF, which is obtained by calculating the IPC in the VLF band for, for example, the head Euler angle TILT, as the group state index 83. In this case, for example, a situation in which body synchronization is high is interpreted as a state in which the synchrony strategy is being highly executed and reflects a good communication state. For this reason, when the IPC is used as the group state index 83, state evaluation and generation of intervention actions, which will be described later, are performed based on the level and changing state of the IPC. As described above, in the first example, the group state index calculation unit 104 calculates the group state index using (Equation 1) to (Equation 3).

[0086] Next, a second example of a mind-body synchronization index is transfer entropy (TE). TE is an information flow index that examines the direction of information transmission (causality) from the amount of information. The mutual information I(X;Y) between signals X and Y can be expressed by Equation 4, where P(xi) is the probability of the i-th element xi of signal X, and P(xi, yj) is the probability of the connection between xi and the j-th element yj of signal Y.

[0087]

number

[0088] Here, mutual information is symmetrical with respect to X and Y and cannot take into account the direction of causality, so TE is an index that extends this. When the conditional probability of X conditioned on Y is P(X|Y), TE TY→X, which indicates the amount of information transmitted from signal Y to signal X, is calculated using (Equation 5).

[0089]

number

[0090] Here, TE TY→X quantifies whether knowing the local signal Xt at time t with a window width k and the remote signal Yt at time t with a window width l is meaningful in terms of information content for predicting the signal Xt+τ that is τ steps ahead of the local signal Xt(k). This can also be understood as an index that relaxes the assumptions of normality of the time series and linearity of causality in Granger causality analysis. For example, if there is no prior knowledge of the information propagation time in a task, for simplicity, the population state index calculation unit 104 can set k=l=τ=1 to measure whether the state of the previous step is effective for predicting the state of the next step.

[0091] This TE TY→X is difficult to use for inter-subject comparisons because the domain of definition changes depending on the difference in the amount of information contained in the signal. Therefore, several normalization strategies have been proposed to align the domain of definition. That is, TE can be rewritten as (Equation 6) and (Equation 7) from the perspective of entropy H.

[0092]

number

[0093]

number

[0094] For this reason, the population state index calculation unit 104 may perform normalization from a formulation based on entropy and use normalized TE and NTE shown in (Equation 8).

[0095]

number

[0096] Furthermore, for example, when the average heart rate or acceleration norm is used as the sensing feature, the signal is a continuous, non-negative signal whose domain changes depending on the subject. For this reason, the population state index calculation unit 104 may perform min-max normalization on the signal, set the number of bins n using Equation 9, which is Sturges' formula with the analytic signal length l, and perform binning at equal intervals.

[0097]

number

[0098] In this case, for example, when the analytic signal length of the sensing feature 82 is 720 points, the population state index calculation unit 104 may select binning with a bin width such that the number of bins is 11. Furthermore, when the population state index calculation unit 104 uses the number of zero crossings, which is a discrete signal, as the sensing feature 82, it may use the discrete value as the bin as it is.

[0099] Furthermore, when the population state index calculation unit 104 uses Euler angles (nod / shake / tilt) as the sensing feature 82, it may perform binning as follows: That is, based on the feature of continuous positive and negative signals with 0 at a position directly facing the screen, normalization is performed by the absolute maximum value centered around 0, and binning is performed at equal intervals so that the number of bins in Sturges' formula is 11.

[0100] From the above, binning can use TE or NTE as the population state index 83 based on binning. Note that, although binning is used to estimate the probability distribution between the target sensing features 82 in the above, the first embodiment is not limited to this example. For example, kernel density estimation or the like may be used instead of binning. In this way, in the second example, the population state index calculation unit 104 calculates the population state index using (Equation 4) to (Equation 9).

[0101] In this way, the group state index calculation unit 104 calculates a group state index 83 that reflects the communication state of a group, using the sensing feature amounts 82 of each of the actors that make up the group. Then, the group state index calculation unit 104 stores the calculated group state index 83 in the storage unit 109.

[0102] Furthermore, in step S54, the vector information generation unit 105 constructs the knowledge DB 85 based on the population state index 83. Note that the construction includes generating and updating the knowledge DB 85. For this purpose, the vector information generation unit 105 generates explanatory text and vector information for the sensing information 81 and / or the sensing feature 82, and the population state index 83. Then, the vector information generation unit 105 associates these with each other to construct the knowledge DB 85. Furthermore, in the first embodiment, more preferably, the vector information generation unit 105 constructs the knowledge DB 85 by associating additional information 84 with the sensing information 81 as needed. Note that the generation of explanatory text and vector information will be described later with reference to FIG. 6. Therefore, the additional information 84 will be described here. Note that in step S54, the vector information generation unit 105 may construct the knowledge DB 85 using at least one of the sensing information 81 and the population state index 83.

[0103] In the first embodiment, when the sensing information 81 and / or sensing feature quantities 82, group state index 83, explanatory text, and vector information constituting the knowledge DB 85 are used as knowledge, it is desirable to associate additional information 84 as meta information of the knowledge record. For example, when constructing the knowledge DB 85, ​​the vector information generation unit 105 may use information about a session that serves as a model for a good or bad communication state. In this case, meta information such as teacher information and explanations about what perspective the session serves as a model is used. This meta information (additional information 84) will be described later in step S544 of FIG. 6. In the first embodiment, in addition to the knowledge DB 85 of FIG. 5, the knowledge DB 85 can also be updated. This update includes adding records to the knowledge DB 85 and adding the knowledge DB itself. Furthermore, this update includes updating the explanatory text and vector information contained in the knowledge DB 85. Therefore, the update will be described later in FIG. 6, that is, when generating explanatory text and vector information. This concludes the explanation of FIG.

[0104] Next, the generation of the explanatory text and vector information in step S54 will be described. To this end, first, the vector information generation unit 105 generates the explanatory text based on the group state index 83. Furthermore, the vector information generation unit 105 may generate the explanatory text based on the sensing information 81. In this case, it is preferable to use the sensing feature 82 generated from the sensing information 81, but the sensing information 81 itself may also be used. Furthermore, the vector information generation unit 105 may generate the explanatory text based on both the sensing information 81 and the group state index 83. Then, the explanatory text is vectorized to generate vector information. Note that here, explanatory text is exemplified as an example of visible information, but the vector information generation unit 105 may generate video information as the visible information and generate vector information from the video information. Furthermore, the vector information generation unit 105 may generate explanatory text and video information and generate vector information from them. In this case, the vector information generation unit 105 may selectively use one of the generated explanatory text and video information to generate vector information, or may use both to generate vector information.

[0105] Details of this will be explained below with reference to Fig. 6. Fig. 6 is a flowchart showing the process of generating explanatory text and vector information in Example 1. In step S541, the vector information generation unit 105 reads raw data such as the sensing information 81, the sensing feature amount 82, and the group state index 83 from the storage unit 109. In this embodiment, the vector information generation unit 105 reads the group state index 83 according to the configuration of the knowledge DB 85 and processes it. However, the vector information generation unit 105 may also read the sensing information 81 and the sensing feature amount 82. In addition, other necessary data may also be read.

[0106] Furthermore, in step S542, the vector information generation unit 105 generates a description based on the read information, more preferably the group state index 83. Note that only the processing for the group state index 83 will be described below. However, since the processing for the sensing information 81 and the sensing feature amount 82 is similar, a description of these will be omitted. Furthermore, as described in step S54, the description is an example of visible information, and in step S542, video information may be generated, or both the description and video information may be generated.

[0107] First, the premise will be explained. A qualitative evaluation of nonverbal response techniques can be achieved by having an expert evaluate the communication state and the group state index 83. However, a method for quantifying nonverbal response techniques has not been disclosed, and evaluation has been difficult simply by mechanically viewing the numerical information of nonverbal response techniques. Therefore, in Example 1, instead of the expert's qualitative evaluation, the group state index 83 is verbalized through explanatory text, and the explanatory text is vectorized and handled, enabling mechanical evaluation. The explanatory text is an example of visible information, and video information, etc., can be used. In other words, it is sufficient if the group state index 83 can be visualized through visible information. It is also desirable that this visible information be editable. Furthermore, in Example 1, the behavior of the second group is evaluated based on sensing information, making it possible to more appropriately evaluate communication-related behavior, including nonverbal response techniques, in a group.

[0108] Then, the vector information generation unit 105 generates an explanatory text that explains the absolute value and time-series change of the population state index 83 from a predetermined verbalization perspective using a processing model 92 for the population state index 83. For example, the processing model 92 may be a large language model (LLM) and a template similar to prompts or setting information that are instructions for the LLM.

[0109] Furthermore, the vector information generation unit 105 generates an explanatory text by inputting the raw data or image data of the group state index 83 into the processing model 92. For example, when TILT_IPC_VLF is used as the group state index 83, the vector information generation unit 105 generates an explanatory text as follows: "In TILT_IPC_VLF, synchrony significantly increased from XX:XX to YY:YY, eventually reaching ZZ and stabilizing. ~~~~." This is a natural-sounding description that verbalizes the qualitative evaluation that was previously performed by an expert. This content is shown as "text" in FIG. 3E. In generating the description above, the description may be generated from one population state index 83, or may be generated from the result of combining multiple population state indexes 83. Furthermore, step S542 may be skipped, and vector information may be generated from the sensing information 81, etc., in step S543.

[0110] Furthermore, in step S543, the vector information generation unit 105 vectorizes the generated explanatory text to generate vector information. Note that the vector information generation unit 105 may vectorize the group state index 83. An example of step S543 will be described below.

[0111] The vector information generation unit 105 uses, for example, a processing model 92 consisting of an LLM, a prompt, and a template. In this case, the vector information generation unit 105 calculates an embedding vector for the generated explanatory text and generates this as vector information. This content is shown as "vector" in Fig. 3E.

[0112] Furthermore, when vectorizing the group state index 83 at the same time, the vector information generation unit 105 may perform vectorization using dimension reduction processing based on LLM or other known machine learning or statistical processing. By performing the above two processes, it is possible to obtain vector information as a feature that reflects the communication state while maintaining the information of the raw data to a certain extent and adding the qualitative evaluation viewpoint that has traditionally been performed by experts.

[0113] The processing model 92 used for generating the caption and vectorization is not limited to LLM, and may be used as needed. For example, if the required caption is simple, a known caption generation model that visualizes the population state index 83 and assigns a caption to the image can be used. Alternatively, the caption may be generated by extracting the time series changes or absolute values ​​of the population state index 83 under predetermined conditions and inputting them into a caption template.

[0114] Furthermore, vectorization only requires that natural language or time-series signals be vectorized into fixed-length vectors. Therefore, for example, a deep learning model or known algorithm capable of dimensionality reduction may be used. Examples of such deep learning models include Long Short-Term Memory (LSTM), Convolutional Neural Network (CNN), Transformer, and Auto Encoder (AE). Furthermore, known statistical or machine learning algorithms such as Principal Component Analysis (PCA) and Uniform Manifold Approximation and Projection (UMAP) may be used.

[0115] Although the explanatory text is used here as an example of visible information, as described above, video information may also be used. That is, in step S543, the vector information generating unit 105 can generate vector information by vectorizing at least one of the explanatory text and the video information.

[0116] Then, in step S544, the vector information generating unit 105 adds additional information 84, which is meta information. An example of this will be described below. The following is used as meta information in two-party communication. First, flag information (metadata{synchrony:1, mimicry:1, ~~}) is used to indicate whether the session should be used as reference for Synchrony, Echoing, Mimicry, Mirroring, or Reflection.

[0117] Additionally, definitions and precautions (metadata{advice:~~}) for adopting the strategy may be used as additional information 84. Also, the ID and storage information (metadata{src_data:~~}) of the data used to generate the explanatory text and vector information may be used. Also, raw data itself (metadata{features:{TILT_IPC_VLF:~~}}) such as sensing information 81, sensing feature amount 82, and population state index 83 may be used as additional information 84.

[0118] Furthermore, when generating intervention actions using the knowledge DB 85 (described later), the intervention action generation unit 107 uses the necessity of proposing intervention candidates as a state evaluation. At this time, known methods such as k-Nearest Neighbor (kNN) or approximate nearest neighbor (ANN) are used. That is, the intervention action generation unit 107 extracts vectorized information having similarity that meets predetermined conditions from the knowledge DB 85, ​​and determines the necessity of proposing intervention candidates based on the extraction results and a condition for determining whether generation of intervention actions is necessary (intervention determination method). If it is determined that proposing intervention candidates is necessary, the intervention action generation unit 107 generates intervention action candidates for the explanatory text, vectorized information, and additional information 84 of the extracted knowledge records. These intervention action candidates may be generated in natural language as intervention action candidates to be taken by a counselor, or may be generated as an explanation support model 91 for generating an operation plan for an actuator required for generating the behavior of a system or a virtual human. Note that the behavior generation of the virtual human is performed in the second embodiment. These generation processes are performed based on rules and patterns for generating predetermined intervention actions, using the explanatory text, vectorized information, and additional information 84 of the extracted knowledge record group as input.

[0119] In step S545, the vector information generation unit 105 adds the explanatory text, vector information, and additional information 84 generated in steps S542 to S544 to the knowledge DB 85, ​​and updates this record. In addition, steps S541 to S545 are repeatedly executed for the number of pieces of target raw data (sensing information 81, etc.). In the above, in the first embodiment, the knowledge DB 85 is constructed using the results of measuring behavior, but it may also be constructed manually.

[0120] Here, the update of the knowledge DB 85, ​​particularly the update of the explanatory text and vector information, will be described. The knowledge DB 85 is updated by reflecting the evaluation results of the behavior evaluation process (FIGS. 7 and 8) described below in the knowledge DB 85. At this time, the process is executed according to the flowchart shown in FIG. 5. In particular, the vector information generation unit 105 modifies the explanatory text, which is an example of visible information, according to the evaluation results of the behavior evaluation process, and generates vector information according to the modified explanatory text. This process is executed as shown in the flowchart of FIG. 6. As a result, the accuracy of information such as the group state index, explanatory text, and vector information stored in the knowledge DB 85 can be improved. As such, in the first embodiment, it is possible to expand the information in the knowledge DB 85 of the first group using not only the evaluation based on the information in the knowledge DB 85 of the second group that is the evaluation target in the behavior evaluation process, but also the processing results for the second group.

[0121] Next, a group behavior evaluation process using the knowledge DB 85 will be described. Fig. 7 is a flowchart showing the behavior evaluation process in the first embodiment. In steps S71 to S73 of Fig. 7, the same processes as steps S51 to S53 of Fig. 5 are executed. However, in steps S71 to S73, the processing is performed on the group to be evaluated. Hereinafter, the group to be handled in steps S51 to S53 will be referred to as the first group, and the group to be evaluated will be referred to as the second group.

[0122] Then, in step S74, the vector information generation unit 105 and the behavior evaluation unit 106 evaluate the behavior of the second group. As a result, if the evaluation is possible (possible), the process proceeds to step S75, and if the evaluation is not possible (impossible), the process returns to step S71. An example of the evaluation process in step S74 will be described below.

[0123] 8 is a flowchart showing details of step S74 of the behavior evaluation processing in the first embodiment. First, in steps S741 to S743, processing similar to steps S541 to S543 in FIG. 6 is executed. Note that, while steps S541 to S543 are executed using data of the first population, steps S741 to S743 are executed using data of the second population. As a result, second vector information, which is vector information of the second population, is generated in step S743.

[0124] Furthermore, in step S744, the vector information generation unit 105 reads first vector information, which is vector information of the first group, from the knowledge DB 85 in the storage unit 109. Furthermore, in step S745, the vector information generation unit 105 compares the first vector information read in step S744 with the second vector information generated in step S743.

[0125] Then, in step S746, vector information generating unit 105 determines whether the comparison result in step S745 satisfies a predetermined condition. At least one of the following (1) to (3) can be used as this condition. (1) The difference between the first vector information and the second vector information is within a predetermined threshold value. (2) The first vector information can improve communication or behavior in the second group by a certain amount or more. (3) Based on the first vector information, intervention actions that can improve the communication or behavior of the second group can be identified.

[0126] If the result of the determination in step S746 is that the condition is satisfied (for example, the difference is within the threshold value), the process proceeds to step S747. If the condition is not satisfied, the process returns to step S71. That is, it is determined in step S74 that it is impossible.

[0127] Furthermore, in step S747, the vector information generation unit 105 evaluates the behavior or communication of the second group based on the first group state index associated with the first vector information determined in step S746 to satisfy the condition. To this end, for example, the behavior evaluation unit 106 identifies the behavior of the second group based on the first sensing information and / or the first group state index corresponding to the first vector information that satisfies the condition. Note that the behavior and communication of the second group can be evaluated based on the sensing information and / or the first group state index themselves. Therefore, the act of identifying and outputting these can be considered as evaluation. This concludes the explanation of FIG. 8, and we return to FIG. 7 to continue the explanation of the behavior evaluation process.

[0128] Next, in step S75, the intervention action generation unit 107 generates an intervention action for the second group in accordance with the evaluation result in step S747. The details of this have been explained in step S544, so they will not be repeated here. Then, in step S76, the output unit 102 notifies the generated intervention action. For this purpose, for example, the processor 2 notifies the intervention action via the communication device 6 in accordance with the notification module 37. Alternatively, the processor 2 may display the intervention action on the input / output device 5 in accordance with the notification module 37.

[0129] This intervention action is preferably notified to the counselor terminal 7-1, but may also be notified to the client terminal 7-2. As a result, the intervention action can be displayed on the counselor terminal 7-1 and the client terminal 7-2. Furthermore, in step S76, the contents of the knowledge DB 85 for the second group may also be included. That is, the sensing information 81, sensing feature amount 82, group state index 83, explanatory text, and vector information of the second group may also be the subject of notification. This concludes the explanation of FIG. 7, and the display screen for the notification in step S76 will now be described.

[0130] Fig. 9 is a diagram showing a display screen of an intervention action in Example 1. Fig. 9 shows an example in which the notification destination is a counselor terminal 7-1, which is configured with a counselor terminal main body 7-11 and a wearable computer 7-12 that cooperate with each other. Fig. 9 also shows a wristwatch-type wearable computer 7-12, but the form is not limited to this.

[0131] First, when the notification in step S76 is made, the wearable computer 7-12 displays an alert indicating that an intervention action has been notified. This display is performed by the notification notification device 15-1 in FIG. 2. Then, when the counselor operates the counselor terminal main body 7-11, the notification content is displayed on the display screen. This display is also performed by the notification notification device 15-1 in FIG. 2. In the example in FIG. 9, the notification content suggests an action to improve synchronization.

[0132] 9 shows the contents of notifications and displays in real time processing during or immediately after communication each time an intervention action is generated in step S75. However, notification processing may also be performed for post-event review after the end of communication, for example, counseling. Next, this post-event review notification will be described.

[0133] FIG. 10 is a diagram showing a display screen of another intervention action in Example 1. That is, FIG. 109 shows the display content for post-event review. In FIG. 10, the notification destination is the counselor terminal 7-1, and the display is made on the notification notification device 15-1 of FIG. 2. In FIG. 10, the display screen 15-10 of the notification notification device 15-1 has a first display area 15-11, a second display area 15-12, and a third display area 15-13. A notification about the second group is displayed in these areas.

[0134] First, the first display area 15-11 displays "TILT_IPC_VLF," which is an example of the group state index 83, and image data, which is an example of the sensing information 81, for a predetermined Session ID (unit of measurement) = 1. In other words, the first display area 15-11 displays the above-mentioned raw data from which the knowledge DB 85 can be constructed.

[0135] On the other hand, the generated intervention action is displayed in the second display area 15-12 and the third display area 15-13. First, the second display area 15-12 displays the same items as the first display area 15-11, namely, "TILT_IPC_VLF" which is an example of the group state indicator 83 and image data which is an example of the sensing information 81. Furthermore, the third display area 15-13 displays notification content proposing an action to improve synchronization, as in FIG. 9. Note that the content shown in FIGS. 9 and 10 may be displayed on the input / output device 5 of the behavior processing device 1. This concludes the description of the first embodiment. [Example]

[0136] In Example 1, communication between real people is evaluated, but in Example 2, an example is shown in which one of the people is a virtual person. First, Fig. 11 is a configuration diagram showing an implementation example of a behavior processing system in Example 2. Below, Fig. 11 will be explained, focusing on the differences from Fig. 2.

[0137] In Fig. 11, a virtual human terminal 7-3 is used instead of the counselor terminal 7-1 in Fig. 2. The virtual human terminal 7-3 has an actor measuring device 11-3, a sensing device 12-3, an input / output device 13-3, a communication device 14-3, and a behavior control device 16, which are connected to each other via communication paths. Of these, the sensing device 12-3, the input / output device 13-3, and the communication device 14-3 can be configured in the same way as the counselor terminal 7-1.

[0138] However, the actor measuring instrument 11-3 is provided with an internal state sensor 23 instead of a biosensor 21-1. The internal state sensor 23 is a modified version of the biosensor 21-1 for virtual humans, such as robots and avatars. In other words, it has a sensor for actuator joint angles and a function for identifying the avatar's physical behavior over time, for virtual humans. Therefore, the internal state sensor 23 is not limited to a sensor in the narrow sense that detects physical quantities, but has a function for identifying the internal state of software, etc. Furthermore, the behavior control device 16 is a control device that controls the virtual human. Note that if the virtual human is a robot that physically behaves, To this behavior control device 16, an actuator that performs this behavior is connected.

[0139] The virtual human terminal 7-3 may execute the functions of the virtual human, that is, the actions related to communication. Also, the functions of the virtual human may be executed by a device separate from the virtual human terminal 7-3.

[0140] Furthermore, the behavior processing device 1 has a control command module 38 of the behavior processing program 30 added to the behavior processing device 1 of Example 1. This module is for executing the function of the control command unit 108 in FIG. 1 , and the processor 2 creates a control command for the virtual human from the intervention action in accordance with the control command module 38. Then, the processor 2 notifies the behavior control device 16 of the control command via the communication device 6 in accordance with the control command module 38. In response to this, the behavior control device 16 controls the virtual human. For example, the behavior control device 16 operates the robot's actuators in accordance with the control command. Note that the control command module 38 (control command unit 108) may also be omitted in Example 2. In this case, the behavior control device 16 of the virtual human terminal 7-3 creates a control command in accordance with the intervention action notified from the behavior processing device 1. As a result, the actuators of the virtual human operate, i.e., are controlled, based on the created control command.

[0141] The information used in the second embodiment is the same as that in the first embodiment, and therefore a description thereof will be omitted. Next, a description will be given of the processing flow in the second embodiment. The sensing information collection process, the knowledge DB 85 construction process, and the explanatory text and vector information generation process in the second embodiment are the same as those in the first embodiment. These have already been explained in FIGS. 4 to 6, and therefore a detailed explanation thereof will be omitted. Naturally, the target of the sensing information collection process, the knowledge DB 85 construction process, and the explanatory text and vector information generation process in the second embodiment is not a counselor but a virtual human.

[0142] Next, a description will be given of the behavior evaluation process of Example 2. Fig. 12 is a flowchart showing the behavior evaluation process in Example 2. In Fig. 12, steps S71 to S74 are the same as those in Fig. 7 of Example 1.

[0143] Then, in step S75-1, the intervention action generation unit 107 generates an intervention action for the second group, i.e., an intervention action for the e-intention, in accordance with the evaluation result in step S747. Specifically, the intervention action generation unit 107 extracts vectorized information having similarity that meets a predetermined condition from the knowledge DB 85, ​​and determines the necessity of an intervention candidate proposal based on the extraction result and a condition for determining whether the generation of an intervention action is necessary (an intervention determination method). As a result, if it is determined that an intervention candidate proposal is necessary, the intervention action generation unit 107 generates an intervention action candidate proposal for the explanatory text, vectorized information, and additional information 84 of the extracted knowledge record group. In the second embodiment, the intervention action generation unit 107 uses the explanation support model 91 to generate an intervention candidate proposal that generates an operation plan for actuators necessary for generating the behavior of the virtual human. Then, the control command unit 108 generates a control command that embodies the intervention candidate proposal. Note that the intervention candidate proposal itself may be treated as a control command. In this case, the control command unit 108 can be omitted.

[0144] Then, in step S77, the intervention action generated in step S75 is executed by the virtual human. To this end, the output unit 102 notifies the virtual human terminal 7-3 of the generated control command. To this end, for example, the processor 2 notifies the control command or intervention action via the communication device 6 in accordance with the notification module 37. Alternatively, the processor 2 may display the intervention action on the input / output device 5 in accordance with the notification module 37.

[0145] In addition, in the virtual human terminal 7-3, when the communication device 14-3 receives the notified control command or intervention action, the behavior control device 16 controls the behavior of the virtual human in accordance with the control command or intervention action. Note that in the second embodiment, an intervention action is generated for the virtual human, but it may also be performed for a client (natural person) in the same manner as in the first embodiment. This concludes the description of each embodiment, but the present invention is not limited to these. For example, embodiment 1 and embodiment 2 may be combined. In this case, a group of three or more actors is evaluated, including a mixture of natural and virtual humans. In this case, it is possible to both control the e-people and notify them of the evaluation results. Furthermore, an administrator terminal used by a user (administrator) such as a supervisor may be connected to the behavior processing device 1, and the evaluation results may be notified to this terminal. In this case, the administrator can evaluate the counselor's communication, such as counseling, and the counselor himself. [Explanation of symbols]

[0146] 1...behavior processing device, 101...input unit, 102...output unit, 103...preprocessing unit, 104...group state index calculation unit, 105...vector information generation unit, 106...behavior evaluation unit, 107...intervention action generation unit, 108...control command unit, 109...memory unit, 81...sensing information, 82...sensing feature amount, 83...group state index, 84...additional information, 85...knowledge DB, 86...intervention candidate plan, 87...actor characteristic information, 90...intervention judgment method, 91...explanation support model, 92...processing model

Claims

1. A behavior processing device for evaluating communication-related behavior of a group of subjects communicating with each other, comprising: a storage unit that stores a knowledge database in which a first group state index relating to a state of communication in a first group and first vector information that is generated based on the first group state index and indicates characteristics of the behavior are associated with the first group state index; an input unit that receives second sensing information indicating behavior related to communication in a second group; a group state index calculation unit that calculates a second group state index related to a state of communication in the second group based on the second sensing information; a vector information generation unit that generates visual information indicating a state of communication of the second group based on the second group state index, and vectorizes the visual information to generate second vector information; A behavior processing device having a behavior evaluation unit that evaluates the behavior of the second group based on a first group state index associated with first vector information included in the knowledge database, the first vector information being compared with the second vector information and satisfying a predetermined condition.

2. The behavior processing device according to claim 1 , The vector information generating unit is a behavior processing device that generates, as visual information, explanatory text or moving image information that indicates a state related to communication in a group.

3. 3. The behavior processing device according to claim 2, the group state index calculation unit calculates a first group state index related to a state of communication in the first group from first sensing information related to communication in the first group; the vector information generation unit generates a first description or first video information indicating a state of communication of the first group based on the first sensing information or the first group state index, and vectorizes the first description or the first video information to generate first vector information; The memory unit stores the generated first vector information.

4. The behavior processing device according to claim 1 , the behavior evaluation unit identifies a behavior of the second group based on first sensing information related to communication of the first group corresponding to first vector information that satisfies the condition and the first group state index; The behavior processing device further comprises an output unit that outputs the behavior of the second group.

5. 5. The behavior processing device according to claim 4, The behavior processing device includes an output unit that outputs the evaluation result of the behavior evaluation unit.

6. The behavior processing device according to claim 1 , The behavior processing device, wherein the first sensing information and the second sensing information regarding communication of the first group are non-verbal information indicating non-verbal behavior.

7. The behavior processing device according to claim 1 A behavior processing device, wherein the first group state index and the second group state index include at least one of a synchronization index and an information flow index of the subject of the behavior.

8. The behavior processing device according to claim 1 , The behavior processing device, wherein the knowledge database further includes additional information regarding non-verbal response techniques in communication situations of the first group.

9. 5. The behavior processing device according to claim 4, The behavior processing device further comprises an intervention behavior generation unit that generates an intervention behavior for the subject of the second group in accordance with the behavior identified by the behavior evaluation unit.

10. The behavior processing device according to claim 9, The actors of the second group include a virtual human; The behavior processing device further comprises a control command unit that generates a control command for controlling the virtual human in accordance with the intervention behavior.

11. The behavior processing device according to claim 1 , The behavior processing device includes a behavior evaluation unit that further uses first sensing information associated with first vector information within the threshold to evaluate the behavior of the second group.

12. The behavior processing device according to claim 1 , A behavior processing device in which the conditions are at least one of: the difference between the first vector information and the second vector information is within a predetermined threshold; the first vector information can improve the communication by a certain value or more; and intervention behavior that can improve the communication can be identified based on the first vector information.

13. The behavior processing device according to claim 10, An actuator is controlled based on the generated control command.

14. A behavior processing method executed by a behavior processing device for evaluating communication-related behavior of a group of subjects communicating with each other, comprising: a memory unit stores a knowledge database in which a first group state index relating to a state of communication in a first group and first vector information which is generated based on the first group state index and indicates characteristics of the behavior are associated with each other; an input unit receiving second sensing information indicating behavior related to communication in a second group; a group state index calculation unit calculates a second group state index related to a state of communication in the second group based on the second sensing information; a vector information generation unit generates visual information indicating a state of communication of the second group based on the second group state index, and vectorizes the visual information to generate second vector information; A behavior processing method in which a behavior evaluation unit evaluates the behavior of the second group based on a first group state index associated with first vector information included in the knowledge database, the first vector information being compared with the second vector information and satisfying a predetermined condition.

15. a behavior processing device that is a computer for evaluating behavior related to communication of a group of agents communicating with each other, a storage unit that stores a knowledge database in which a first group state index relating to a state of communication in a first group and first vector information that is generated based on the first group state index and indicates characteristics of the behavior are associated with the first group state index; an input unit that receives second sensing information indicating behavior related to communication in a second group; a group state index calculation unit that calculates a second group state index related to a state of communication in the second group based on the second sensing information; a vector information generation unit that generates visual information indicating a state of communication of the second group based on the second group state index, and vectorizes the visual information to generate second vector information; A behavior processing program for functioning as a behavior evaluation unit that evaluates the behavior of the second group based on a first group state index associated with first vector information included in the knowledge database, the first vector information whose comparison result with the second vector information satisfies a predetermined condition.

Images