Presentation device and presentation method

Abstract

In a presentation device (10), a storage unit (14) stores information related to a risk of trouble for each expression constituting a communication. A calculation unit (15c) references the risk-related information stored in the storage unit (14), and calculates a risk value representing the magnitude of the risk of an expression constituting a communication. A determination unit (15d) uses the information of a conversation counterpart in the communication to determine a comprehension status of the conversation counterpart with respect to the expression. An adjustment unit (15e) adjusts the calculated risk value for the expression according to the determined comprehension status.

Description

Presentation device and presentation method

[0001] The present invention relates to a presentation device and a presentation method.

[0002] In language-based communication, expressions that may hinder mutual respect and understanding, such as ambiguous expressions, overbearing expressions, expressions based on stereotypes, etc. Conventionally, a technology has been proposed that encourages the correction of ambiguous expressions (see Non-Patent Document 1).

[0003] Ryohei Saijo and three others, "Research and Development of Web Conferencing Tools to Support Mutual Respect - Aiming for a Society that Respects Diversity," 2023 Institute of Electronics, Information and Communication Engineers General Conference, Proceedings of the IEICE Communications Conference, 2023, pp. SS-84-SS-85

[0004] However, with conventional techniques, it is difficult to evaluate the risks inherent in expressions used in communication, taking into account the understanding state of the other party.

[0005] In order to solve the above-mentioned problems and achieve the objectives, the presentation device of the present invention is characterized by having a memory unit that stores information regarding the risk of trouble for each expression that constitutes communication, a calculation unit that refers to the risk information stored in the memory unit and calculates a risk value that represents the magnitude of risk of the expression that constitutes communication, a discrimination unit that uses information about the dialogue partner in the communication to determine the dialogue partner's understanding of the expression, and an adjustment unit that adjusts the calculated risk value of the expression in accordance with the determined understanding state.

[0006] According to the present invention, it is possible to evaluate the risk inherent in an expression used in communication, taking into account the understanding state of the other party.

[0007] FIG. 1 is a schematic diagram illustrating the overall configuration of the presentation device of the first embodiment. FIG. 2 is a diagram illustrating the data structure of a master dictionary. FIG. 3 is a diagram illustrating the data structure of an individual dictionary. FIG. 4 is a diagram illustrating the data structure of a risk expression dictionary. FIG. 5 is a diagram for explaining risk types. FIG. 6 is a diagram illustrating the data structure of a non-verbal risk expression DB. FIG. 7 is a diagram illustrating the data structure of a relationship DB. FIG. 8 is a diagram illustrating the data structure of a social attribute DB. FIG. 9 is a diagram illustrating the data structure of a personal characteristic DB. FIG. 10 is a diagram illustrating the data structure of a communication history DB. FIG. 11 is a diagram illustrating the data structure of an understanding status list. FIG. 12 is a diagram illustrating the data structure of an understanding status history. FIG. 13 is a diagram illustrating the data structure of a risk adjustment DB. FIG. 14 is a diagram for explaining the processing of an individual dictionary generation unit. FIG. 15 is a diagram for explaining the processing of an individual dictionary generation unit. FIG. 16 is a diagram for explaining the processing of a language risk calculation unit. FIG. 17 is a diagram for explaining the presentation process of the first embodiment. FIG. 18 is a diagram for explaining intervention information. FIG. 19 is a flowchart showing a presentation process procedure. FIG. 20 is a diagram for explaining the second embodiment. FIG. 21 is a diagram for explaining the second embodiment. FIG. 22 is a schematic diagram illustrating a schematic configuration of a presentation device of the third embodiment. FIG. 23 is a diagram for explaining the third embodiment. FIG. 24 is a diagram for explaining the third embodiment. FIG. 25 is a diagram for explaining the third embodiment. FIG. 26 is a diagram for explaining the third embodiment. FIG. 27 is a diagram illustrating an example of a computer that executes a presentation program.

[0008] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the description of the drawings, the same parts are designated by the same reference numerals.

[0009] [First embodiment] [Outline of presentation device] A presentation device of this embodiment calculates a risk value representing the magnitude of risk of an expression that constitutes communication each time a user makes an utterance or other expression. The presentation device also adjusts the calculated risk value according to information indicating the level of understanding of the conversation partner (understanding state), which is expressed by non-verbal information such as nodding, smiling, and frowning.

[0010] If the risk value is equal to or greater than a predetermined threshold, the presentation device intervenes to notify the user of the risk and prompts the user to correct the risk expression.

[0011] Here, intervention means presenting information about expressions that contain risks (hereinafter also referred to as risk expressions or ambiguous expressions). Furthermore, hereinafter, information presented during intervention will be referred to as intervention information.

[0012] [Configuration of Presentation Device] Fig. 1 is a schematic diagram illustrating the overall configuration of a presentation device according to the first embodiment. As illustrated in Fig. 1, a presentation device 10 according to the present embodiment is realized by a general-purpose computer such as a personal computer, and includes an input unit 11, an output unit 12, a communication control unit 13, a storage unit 14, and a control unit 15.

[0013] The input unit 11 is realized using input devices such as a keyboard, mouse, camera, and microphone, and inputs various instruction information such as a command to start processing to the control unit 15 in response to an input operation by an operator. The output unit 12 is realized by a display device such as a liquid crystal display, a printing device such as a printer, etc. For example, the output unit 12 displays the results of a presentation process described below.

[0014] The communication control unit 13 is realized by a NIC (Network Interface Card) or the like, and controls communication between the control unit 15 and an external device via a telecommunication line such as a LAN (Local Area Network) or the Internet. For example, the communication control unit 13 controls communication between the control unit 15 and an external management device that manages various types of information.

[0015] The storage unit 14 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 14 stores in advance the processing program that operates the presentation device 10 and data used during execution of the processing program, or temporarily stores the data each time processing is performed. The storage unit 14 may be configured to communicate with the control unit 15 via the communication control unit 13.

[0016] In the present embodiment, the storage unit 14 stores information regarding the risk of trouble for each expression constituting communication. For example, the storage unit 14 stores, as the risk-related information, information regarding risk corresponding to the usage situation. For example, the storage unit 14 stores, as the risk-related information, a master dictionary common to the entire device and an individual dictionary corresponding to the usage situation. For example, the storage unit 14 stores, as the risk-related information, information regarding the risk of trouble caused by words included in language and information regarding the risk of trouble caused by non-verbal expressions. For example, the storage unit 14 stores, as the risk-related information, information regarding the risk of trouble caused by any one or more of speech speed, speech volume, number of characters, pitch, volume, amount of movement, movement speed, and amount of change in facial expression.

[0017] For example, the storage unit 14 stores a risk expression dictionary 14a. The risk expression dictionary 14a includes information about the risk of language trouble for each word constituting the dialogue. Specifically, the risk expression dictionary 14a includes information about one or more of the risks of the part of speech of each word, the number of meanings indicated by each word, and the ambiguity of the meaning indicated by each word. This information is collected via the input unit 11 or from a management device that manages various information, etc., prior to or during the presentation process described below, and stored in the storage unit 14. The risk expression dictionary 14a is composed of a combination of a master dictionary 141 and individual dictionaries 142 subordinate to it.

[0018] The master dictionary 141 contains information about risks that are common to all situations, regardless of individual usage situations. The individual dictionary 142 contains information about risks set for each individual domain. The individual dictionary 142 is not limited to being set for each domain, but may also be set for each individual. The risk expression dictionary 14a is generated by the individual dictionary generation unit 15b, which will be described later, based on the master dictionary 141 and the individual dictionary 142.

[0019] Here, Fig. 2 is a diagram illustrating an example of the data structure of the master dictionary. Fig. 3 is a diagram illustrating an example of the data structure of the individual dictionary. Fig. 4 is a diagram illustrating an example of the data structure of the risk expression dictionary. Also, Fig. 5 is a diagram for explaining risk types. First, as illustrated in Figs. 2 to 4, the master dictionary 141, the individual dictionary 142, and the risk expression dictionary 14a include information items such as part of speech, meaning, number of meanings, risk level, risk type, etc. for each word.

[0020] The number of meanings is the number of different meanings for the same word. In addition, the number of homonyms may be added. The risk level is a value set by the user for each word, taking into account, for example, the part of speech, risk type, relevance to business, etc. Alternatively, the risk level may be a value based on a predefined classification table of ambiguous words, etc.

[0021] The risk type is, for example, a value ranging from 1 to 7 that indicates each class of ambiguity classification. Here, Fig. 5 shows an example of seven ambiguity classes. For example, a risk type of 1 is set for a word that has multiple meanings.

[0022] The master dictionary 141, the individual dictionary 142, and the risk expression dictionary 14a may include information items that change dynamically during use, such as the importance of each word in a sentence. For example, for words with high risk values ​​calculated in the presentation process described below, if the importance is not high, the risk value may be individually set to be calculated as low.

[0023] Furthermore, the master dictionary 141, the individual dictionary 142, and the risk expression dictionary 14a are not limited to information items for each word, but may be information items for each group of expressions consisting of multiple words. In this case, the expressions may include, for example, expressions using words for facial expressions such as "smiling face" or "angry face."

[0024] The storage unit 14 also stores a non-verbal risk expression DB (Data Base) 14b, a relationship DB 14c, a social attribute DB 14d, a personal characteristic DB 14e, and a communication history DB 14f. The non-verbal risk expression DB 14b stores information regarding the risk of trouble in non-verbal expressions that constitute communication.

[0025] Here, an example of the data structure of the non-verbal risk expression DB 14b will be described using Figure 6. Figure 6 is a diagram illustrating an example of the data structure of the non-verbal risk expression DB. As illustrated in Figure 6, the non-verbal risk expression DB 14b is a DB that holds information on non-verbal risks related to each modal for each user. Note that the non-verbal risk expression DB 14b may be a single DB overall, or may be created and used for each individual user, or may be configured to have a master dictionary and individual dictionaries, similar to the risk expression dictionary, with individual adjustments made using the individual dictionaries. An example created for each individual user will be shown below.

[0026] The non-verbal risk expression DB 14b includes "risk elements," "calculation methods," "risk thresholds," and "risk importance" as non-verbal risk information. "Risk elements" indicate information elements that may pose a risk to the user in each modal. "Calculation methods" indicate the calculation methods used by the non-verbal risk calculation unit 152, which will be described later, to quantify risk elements and calculate risk values.

[0027] The "risk threshold" is a threshold used to determine whether a value calculated based on a calculation method poses a high risk to the user. Multiple "risk thresholds" (e.g., upper and lower limits) may be set for each row of the DB. In the above example, the "risk threshold" indicates a value in units corresponding to the method (e.g., Hz), but a normalized value (0.0 to 1.0) or standardized value may be used using information obtained in advance (e.g., the upper and lower limits of the user's hearing range). In this case, the calculation results of each calculation method are also normalized or standardized values. Furthermore, the "risk threshold" may be either the risk threshold as a receiver when the user is receiving an expression, or the risk threshold as a sender when the user is making an expression, or both. The "risk importance" indicates the importance of the risk element in each row for each user; the higher the value, the more likely that element will lead to miscommunication.

[0028] The relationship DB 14c includes information about the relationships between speakers. The social attribute DB 14d includes information representing the social attributes of the speaker and the interlocutor. The individual characteristic DB 14e includes information representing the speaker's characteristics related to utterances. The communication history DB 14f includes the dialogue history. This information is collected via the input unit 11 or from a management device that manages various information, prior to or during the presentation process described below, and stored in the memory unit 14.

[0029] 7 is a diagram illustrating an example of the data configuration of the relationship DB. As illustrated in FIG. 7, the relationship DB 14c includes, for each user, information items representing the relationship with a specific other user, such as a relationship score, relationship information, and non-work interactions. FIG. 7 illustrates the relationship DB 14c for a user with a user ID of 002.

[0030] The relationship score ranges from 0.0 to 1.0, with values ​​closer to 1 indicating a closer relationship and values ​​closer to 0 indicating a more distant relationship. For example, since a user with whom a user frequently interacts has a closer relationship, the relationship score may be calculated using the frequency of interactions in the communication history DB 14f (described below). Alternatively, the relationship score may be set with reference to job titles in the social attribute DB 14d (described below), such that 0.5 is used for the same job title and 0.1 is used for jobs with greater distance between them, such as between the president and an ordinary employee. Alternatively, the relationship score may be calculated comprehensively with reference to information items in the communication history DB 14f, social attribute DB 14d, and personal characteristic DB 14e (described below), or may be calculated based on compatibility using evaluation values ​​of the company's corporate culture, the individual's personality, such as the Big 5, etc.

[0031] The relationship information indicates a business relationship between a user and a specific other user, such as a relationship with a superior in the same department, etc. The non-work-related interaction indicates whether or not there is interaction outside of work.

[0032] Fig. 8 is a diagram illustrating an example of the data configuration of the social attribute DB. As illustrated in Fig. 8, the social attribute DB 14d includes information items representing the social attributes, i.e., social positions, of the speaker and interlocutor, such as affiliation, position, and years of employment. The social attribute DB 14d is set for each topic or situation of the conversation. Fig. 8 illustrates the social attribute DB 14d for business topics.

[0033] 9 is a diagram illustrating an example of the data configuration of the individual characteristic DB. As illustrated in Fig. 9, the individual characteristic DB 14e includes, for each user, information items representing the speaker's characteristics regarding each utterance, such as frequency, degree of agreement with usage status, frequently occurring topics, and evaluation of risk value.

[0034] The frequency is a value ranging from 0.0 to 1.0 that indicates how often the statement occurs. The consistency of usage is a value ranging from 0.0 to 1.0 that indicates the consistency of the topics in which the statement occurs. The evaluation of the risk value is a value ranging from 0.0 to 1.0 that indicates the user's self-evaluation of the history of risk values ​​calculated for the word, with a higher value indicating a higher evaluation that the risk value is appropriate.

[0035] 10 is a diagram illustrating an example of the data configuration of the communication history DB. As illustrated in Fig. 10, the communication history DB 14f represents a dialogue history, and includes, for each utterance, information items such as a utterance ID, a user ID, a time, the content of the utterance, a risk value, a topic, an interlocutor ID, and a miscommunication record.

[0036] The risk value is a risk value previously calculated by the presentation process described below. The topic is a label indicating an outline of the conversation in which the statement was made, and an appropriate social attribute DB 14d is selected based on this. The miscommunication record indicates whether or not there is a record of any trouble caused by the statement.

[0037] The communication history DB 14f may also include non-verbal communication expressions such as facial expressions based on video data or the like.

[0038] The storage unit 14 also stores an understanding state list 14g, an understanding state history 14h, and a risk adjustment DB 14i.

[0039] 11 is a diagram illustrating an example of the data configuration of the understanding state list. The understanding state list 14g represents a list of understanding states expressed by a conversation partner. Here, the understanding state is a classification type of information that indicates the user's understanding state, such as nodding, smiling, frowning, etc., expressed by non-verbal information. In the example shown in FIG. 11, the understanding states are classified into a smiling expression, a frowning expression, a nodding gesture, and a tilting of the head gesture.

[0040] Fig. 12 is a diagram illustrating an example of the data structure of the understanding situation history. The understanding situation history 14h is a history of expressed understanding situations and their speakers, and includes, for example, a timestamp, a speaker ID, and an understanding situation, as illustrated in Fig. 12. Alternatively, in the understanding situation history 14h, the understanding situation of the dialogue partner may be associated with a sentence using, for example, the ID of the spoken sentence (expression) instead of the timestamp.

[0041] The understanding status history 14h also includes the speaker (interlocutor) ID and the understanding status for a predetermined period of time in the past, such as the past 30 seconds, etc. The understanding status history 14h may also include the start time / end time of each understanding status expression.

[0042] 13 is a diagram illustrating an example of the data configuration of the risk adjustment DB. The risk adjustment DB 14i stores information that associates the conditions of the conversation partner's understanding state of the expression to be processed with the method of adjusting the risk value. In the example shown in FIG. 13, for example, if the understanding state of a person other than the speaker includes "facial expression - frown" and "motion - tilting head" within the past 30 seconds, the risk value r ｉ It is specified that the ratio should be 1.5.

[0043] In addition, if the understanding status of someone other than the speaker, such as "facial expression - smiling" or "action - nodding," is included within the past 30 seconds, the risk value r ｉ It is specified that the ratio should be 0.5.

[0044] In addition, if the expression history of the understanding status of the speaker other than the speaker is not included within the past 30 seconds, the risk value r ｉ is not to be adjusted.

[0045] The method for adjusting the risk values ​​in the risk adjustment DB 14i is not limited to the above, and a function that changes the magnitude of the risk value, such as addition or subtraction, may be applied. Alternatively, instead of adjusting the risk value, the risk adjustment DB 14i may specify a change in the association with the presentation method. For example, instead of presenting high risk values ​​in red and medium risk values ​​in orange, a change in the presentation method may be specified such that high risk values ​​are presented in orange and medium risk values ​​are presented in red.

[0046] Furthermore, the understanding states targeted by the risk value adjustment method are not limited to understanding states within a predetermined time, such as within the past 30 seconds. For example, the most recently displayed understanding state may be targeted by the risk value adjustment method. Alternatively, the risk value adjustment method may be specified by weighting multiple understanding states according to their temporal proximity. In this case, for example, understanding states whose weighted values ​​are equal to or greater than a predetermined threshold may be targeted by the risk value adjustment method, or the adjustment method may be changed to further increase the risk value.

[0047] Alternatively, instead of the understanding state within a predetermined time, only the understanding state expressed during the utterance of the sentence to be processed may be the subject of the risk value adjustment method.

[0048] Returning to the description of FIG. 1 , the control unit 15 is realized using a CPU (Central Processing Unit) or the like, and executes a processing program stored in memory. As a result, the control unit 15 functions as an acquisition unit 15a, an individual dictionary generation unit 15b, a calculation unit 15c, a determination unit 15d, an adjustment unit 15e, and a presentation unit 15f, as exemplified in FIG. 1 , and executes the presentation process. Note that these functional units may each be implemented in different hardware, or some of them may be implemented in different hardware. The control unit 15 may also include other functional units.

[0049] The acquisition unit 15a acquires expressions constituting the communication to be processed. Here, "expressions constituting the communication to be processed" refer to expressions constituting verbal or non-verbal communication. For example, the acquisition unit 15a acquires words constituting the dialogue to be processed as expressions constituting verbal communication. Specifically, the acquisition unit 15a accepts input of the dialogue to be processed via the input unit 11 or the communication control unit 13, and performs morphological analysis to extract words constituting the dialogue.

[0050] Furthermore, for example, the acquisition unit 15a acquires, as expressions constituting non-verbal communication, audio data including the speech of the processing target or image data or video data including the body movements and facial expressions of the processing target using a built-in camera, microphone, keyboard, etc. At that time, the acquisition unit 15a may also acquire the content of the speech and the start time of the speech sentence.

[0051] The acquiring unit 15a may store the acquired expressions in the storage unit 14 prior to the presentation process described below. Alternatively, the acquiring unit 15a may immediately transfer the information to the calculating unit 15c. The acquiring unit 15a may also acquire an individual usage situation corresponding to the dialogue situation.

[0052] The acquisition unit 15a also acquires information about the conversation partner in communication. For example, the acquisition unit 15a acquires video information of the conversation. The acquisition unit 15a may acquire a signal from an acceleration sensor of the conversation partner, audio information such as the volume of the voice, text of the speech, etc.

[0053] The individual dictionary generation unit 15b generates information on risks according to the usage situation as information on risks, and stores the information on risks in the storage unit 14. For example, the individual dictionary generation unit 15b obtains necessary information from the user or from each DB in the storage unit 14 to generate the individual dictionary 142. Note that although the individual dictionary generation unit 15b is shown as part of the control unit 15 in FIG. 1 , it can also be held in the storage unit 14, in which case the input from the user required for generation is obtained from the user via the control unit 15, and information from each DB is obtained directly within the storage unit 14.

[0054] The individual dictionary generation unit 15b can generate the individual dictionary 142 based on the set values, by manually setting the necessary values ​​for the master dictionary 141 by an individual or a representative of a group who requires individual adjustments.

[0055] In addition, the individual dictionary generation unit 15b may acquire feedback from the user at any time and automatically update the individual dictionary, such as by having the user provide feedback on the risk assessment results after the end of communication (for example, after the end of a speech or a meeting) and reflecting the results in the individual dictionary, or by acquiring the user's feedback results to the personal characteristic DB 14e and updating the individual dictionary.

[0056] Furthermore, when generating an individual dictionary 142 specialized for between specific members (or within a work domain), the individual dictionary generation unit 15b may automatically generate the individual dictionary 142 using group trends acquired from the relationship DB 14c or the communication history DB 14f. For example, the individual dictionary generation unit 15b may generate an individual dictionary so that the risk value of the "specific word" is lowered when a "specific word" appears frequently in the communication history DB 14f between specific members and the relationship score between the members is high (i.e., a close relationship).

[0057] Furthermore, the individual dictionary generation unit 15b may automatically generate the individual dictionary 142 by using the social attribute DB 14d and the personal characteristic DB 14e of the members participating in communication so as to take into account the risk of expressions used when communicating with people of a specific age group or position. For example, when there are members (veterans) with years of service of 15 years or more and members (young people) with years of service of three years or less, the individual dictionary generation unit 15b may assume that the members with years of service of three years or less are younger and generate individual dictionaries tailored to both users that increase the risk of expressions frequently used by young members (youth slang) and casual expressions frequently used by veterans (expressions that may be perceived as a little rough).

[0058] Furthermore, the individual dictionary generation unit 15b may create the risk expression dictionary 14a by integrating the master dictionary 141 and the individual dictionary 142. For example, the individual dictionary generation unit 15b compares the master dictionary 141 with the individual dictionary 142, and if a word is included in both the master dictionary 141 and the individual dictionary 142 and the risk level of the word is lower in the individual dictionary 142 than in the master dictionary 141, the individual dictionary generation unit 15b may delete the word from the risk expression dictionary 14a. Furthermore, if there is a word that is not in the master dictionary 141 but is in the individual dictionary 142, the individual dictionary generation unit 15b may add the word to the risk expression dictionary 14a.

[0059] 14 and 15 are diagrams for explaining the processing of the individual dictionary generation unit. For example, as illustrated in FIG. 14, the individual dictionary generation unit 15b compares the master dictionary 141 with the individual dictionary 142, and if the word "kill" is included in both the master dictionary 141 and the individual dictionary 142 and the risk level of the word "kill" is lower in the individual dictionary 142 than in the master dictionary 141, the individual dictionary generation unit 15b deletes the word from the risk expression dictionary 14a. Also, as illustrated in FIG. 14, if there is a word "probably" that is not in the master dictionary 141 but is in the individual dictionary 142, the individual dictionary generation unit 15b adds the word "probably" to the risk expression dictionary 14a as a word missing from the master dictionary 141.

[0060] Furthermore, the individual dictionary generation unit 15b may compare the master dictionary 141 with the individual dictionary 142, and if the values ​​(number of meanings, danger level, risk type, etc.) differ, may apply the value of the individual dictionary 142. Furthermore, if there is a word that is not in the master dictionary 141 but is in the individual dictionary 142, the individual dictionary generation unit 15b may add the word to the risk expression dictionary 14a.

[0061] For example, as illustrated in Fig. 15, the individual dictionary generation unit 15b compares the master dictionary 141 with the individual dictionary 142, and if the "risk levels" of the word "kill" differ, applies the risk level of "2" in the individual dictionary 142. Also, as illustrated in Fig. 15, if there is a word "probably" that is not in the master dictionary 141 but is in the individual dictionary 142, the individual dictionary generation unit 15b adds the word "probably" to the risk expression dictionary 14a as a word that is missing from the master dictionary 141.

[0062] Furthermore, when there are multiple individual dictionaries 142, the individual dictionary generation unit 15b can weight the individual dictionaries 142 and integrate them so that the setting values ​​of the individual dictionaries 142 with higher weights are applied preferentially to the risk expression dictionary 14a. Furthermore, the individual dictionary generation unit 15b may store the generated risk expression dictionary 14a in the storage unit 14 or the control unit 15 and reuse it when communication is carried out in the same or similar situation, or may generate it each time communication starts. Note that the integration process of the risk expression dictionary 14a may be performed on the storage unit 14 side.

[0063] The calculation unit 15c refers to the information on risk stored in the storage unit 14 and calculates a risk value that indicates the magnitude of risk of an expression that constitutes the input communication. For example, the calculation unit 15c acquires information on risk corresponding to the usage situation of the user who made the expression from the storage unit 14, refers to the information on risk, and calculates the risk value. For example, the calculation unit 15c generates a risk expression dictionary 14a by integrating the master dictionary 141 and the individual dictionary 142 according to the usage situation, and calculates the risk value by referring to the risk expression dictionary 14a. The calculation unit 15c has a linguistic risk calculation unit 151 and a non-linguistic risk calculation unit 152.

[0064] The language risk calculation unit 151 acquires from the storage unit 14 the risk expression dictionary 14a corresponding to the usage situation of the user who made the expression, and calculates a risk value by referring to information related to the risk. That is, the language risk calculation unit 151 refers to the risk expression dictionary 14a stored in the storage unit 14 that corresponds to the usage situation of the input communication, and calculates a risk value that indicates the magnitude of risk for each word that makes up the input dialogue. Here, the risk value indicates an estimated value of the magnitude of the risk of language trouble, i.e., miscommunication, occurring, for each word that appears in the sentence, or for the entire sentence. The language risk calculation unit 151 calculates the risk value using risk variables derived from the information related to risk contained in the risk expression dictionary 14a.

[0065] Here, the risk variable is derived based on, for example, the part of speech or the risk level of the word. Alternatively, the risk variable is derived based on the number of meanings of the word. Alternatively, the risk variable is derived based on the risk type. Note that, for example, risk variables related to context based on information about the relationship between speakers are not used.

[0066] Furthermore, the language risk calculation unit 151 may calculate a risk value for each group of expressions made up of multiple words, which is a unit of information in the risk expression dictionary 14 a. In this case, expressions may include, for example, expressions using words that express facial expressions, such as a “smiley face” or an “angry face.”

[0067] The language risk calculation unit 151 calculates a risk value using the product of multiple pieces of information on risks that have been weighted in a predetermined manner. Specifically, the language risk calculation unit 151 calculates the risk value r of the i-th word in a sentence using the following formula (1): ｉ Calculate.

[0068]

[0069] Here, the risk variable p ｉ１takes on three levels of value, from 1 to 3, with the higher the value, the higher the risk. For example, if the part of speech is a modifier such as an adjective or adverb, it is ambiguous and risky, so it is assigned a value of 3, if it is a demonstrative it is assigned a value of 2, and otherwise it is assigned a value of 1. Alternatively, the risk level may be applied as is, or a value based on the part of speech may be used in combination with the risk level value.

[0070] Also, the risk variable p based on the number of meanings of a word ｉ２ For example, the value registered in the risk expression dictionary 14a may be used as is, or the risk variable p ｉ１ Similarly, the values ​​may be classified into three levels.

[0071] In addition, a risk variable p based on the risk type of a word ｉ３ For example, the value may be set by a user who is familiar with the business situation. In addition, the risk type may be defined in seven stages, for example, as shown in Figure 5, and a risk variable having values ​​in multiple stages may be set taking into account the importance of these.

[0072] In addition, the risk value r ｉ The weight variable w is set to a value greater than 0.0. １ ~w ３ and the risk variable p ｉ１ ~p ｉ３ is adjusted.

[0073] Here, Fig. 16 is a diagram for explaining the processing of the language risk calculation unit. Fig. 16(1) shows a risk variable p ｉ３ The risk level is defined as three levels, 1 to 3, as shown in Figure 16(2). The risk level shown in Figure 16(2) is the misunderstanding risk, and the misunderstanding risk is an example of a variable that becomes more risky as the level of importance increases, 1, 3, and 4 in Figure 5. ｉ２ is exemplified.

[0074] FIG. 16(3) shows the part of speech p of the i-th word. ｉ１ , meaning number p ｉ２ , risk of misunderstanding ｉ３ Risk value r using ｉIn the example shown in FIG. 16(3), the part of speech p ｉ１ Weight variable 1 (w １ ) is set to 0, and the part of speech p ｉ１ By suppressing the influence of the semantic number p ｉ２ and risk of misunderstanding ｉ３ The risk value r ｉ has been calculated.

[0075] In the example shown in FIG. 16(3), the risk value r ｉ To prevent this from becoming 0, the part of speech p ｉ１ × weight variable 1 (w １ ), semantic number p ｉ２ × Weight variable 2 (w ２ ), risk of misunderstanding p ｉ３ × Weight variable 3 (w ３ ) are each multiplied by 1.

[0076] Returning to the explanation of Fig. 1, the language risk calculation unit 151 may calculate a risk value using the sum of multiple pieces of information related to risks that have been weighted in a predetermined manner. For example, the language risk calculation unit 151 calculates the risk value r of the i-th word in a sentence using the following formula (2): ｉ Calculate.

[0077]

[0078] In this case, each risk variable p ｉ１ ~p ｉ３ is a normalized value between 0.0 and 1.0, and the weight variable w １ ~w ３ is a value in the range of 0.0 to 1.0. The calculated risk value r ｉ The value ranges from 0.0 to 1.0, making it easy to compare risk values.

[0079] The language risk calculation unit 151 may further calculate the risk value using one or more of information about the relationship between speakers, information representing the social attributes of the speaker and interlocutor, information representing the speaker's characteristics, or dialogue history. For example, the language risk calculation unit 151 calculates the risk value by referring to the relationship DB 14 c, the social attribute DB 14 d, the individual characteristic DB 14 e, or the communication history DB 14 f. This makes it possible to calculate a risk value that takes context into consideration.

[0080] In this case, the language risk calculation unit 151 calculates the risk value r of the i-th word in the sentence using, for example, the following equation (3): ｉ Calculate.

[0081]

[0082] Here, the risk variable p ｉ１ ~p ｉ３ is the same as the above equation (1). １ ~w ７ is a value in the range of 0.0 to 1.0. The calculated risk value r ｉ The weight variable w １ ~w ７ and the risk variable p ｉ１ ~p ｉ７ is adjusted.

[0083] Risk variable p based on the relationship between users ｉ４ For example, the risk variable p may be calculated by subtracting the relationship score in the relationship DB 14c from 1, or may be calculated by multiplying the relationship score by 1, where 1 is the value of the relationship score if there is non-work-related interaction and 2 is the value of the relationship score if there is no non-work-related interaction. ｉ４ The weaker the relationship, the larger the value becomes, and the greater the risk value becomes.

[0084] Risk variable p based on social attributes ｉ５ For example, the risk variable p may be the difference in the number of years of employment between users in the social attribute DB 14d, or may be the difference in the quantified job titles between users who are interacting, with the higher the job title, the larger the value. ｉ５The greater the difference in position within the organization, such as years of employment or position, the larger the value becomes, and the greater the risk value becomes.

[0085] Alternatively, the risk variable p calculated as above ｉ４ and the risk variable p ｉ５ The product or sum of the risk variable p ｉ４ and the risk variable p ｉ５ It may be substituted as:

[0086] Risk variable p based on individual characteristics ｉ６ For example, the risk variable p may be set to a value obtained by subtracting the degree of agreement of the usage status of the individual characteristic DB 14e from 1, or may be set to the reciprocal of the degree of agreement of the usage status. ｉ６ may be the frequency in the individual characteristic DB 14e, or may be the product of the frequency and a value obtained by subtracting the degree of agreement of the usage situation from 1.

[0087] Risk variable p based on dialogue history ｉ７ For example, the risk variable p may be a risk value in the communication history DB 14f, or may be a product of the risk value and a numerical value of the miscommunication record (2 if there is one, 1 if there is none). Alternatively, assuming that the closer the relationship to the business, the greater the risk of miscommunication, the risk variable p ｉ７ may be set as a numerical value that quantifies the relationship between the topic and work, and the deeper the relationship with work, the larger the value. For example, confidential information related to corporate strategy may be set to 1, general internal company information may be set to 0.5, and public information related to after-work recreational events may be set to 0.1.

[0088] Alternatively, the language risk calculation unit 151 calculates the risk value r of the i-th word in the sentence using the following formula (4): ｉ may be calculated.

[0089]

[0090] Here, the risk variable p ｉ１ ~p ｉ７ is a value normalized to 0.0 to 1.0, similar to the above formula (3), and the weight variable w １ ~w ７ is a value in the range of 0.0 to 1.0.

[0091] In addition, the risk variable p ｉ１ ~p ｉ７ The values ​​are not limited to those described above. For example, by additionally registering a predetermined information item in each piece of information (14a to 14f) in the storage unit 14, values ​​such as those exemplified below may be applied.

[0092] For example, the risk variable p ｉ１ ~p ｉ３ The degree of occurrence of miscommunication due to the word, the degree of danger when miscommunication occurs due to the word, the frequency of occurrence of the word in general conversation, etc. may be applied as the criterion.

[0093] In addition, the risk variable p ｉ４ or a risk variable p using the social attribute DB 14d ｉ５ As examples, the common background of the speakers, the degree of strength or overlap of attributes, the proportion of common understanding between the speakers, the personal relationships between the speakers, etc. may be applied.

[0094] In addition, the risk variable p ｉ６ As the risk assessment, the speaker's background, attributes, and an evaluation of the risk value calculated for past statements may be applied.

[0095] In addition, the risk variable p ｉ７ As the target, words that appear frequently in the current dialogue, words that appear frequently in past dialogues, words that have caused miscommunication in past dialogues, words that have gained common understanding, etc. may be applied.

[0096] Furthermore, the method for calculating the risk value is not limited to using the information (14a to 14f) in the storage unit 14. For example, the risk value may be calculated by using the following information that can be collected during a conversation:

[0097] For example, the risk value may be calculated by combining the analysis results of biological signals such as facial expressions, electroencephalograms, and gaze, with the content of statements stored in the communication history DB 14 f. Furthermore, an evaluation value of medical or objective changes in the physical condition of the person may be additionally applied as a risk variable.

[0098] Alternatively, the seriousness of a problem that occurs after a dialogue may be fed back and reflected in the risk or importance in the risk expression dictionary 14a or the risk value in the communication history DB 14f. The seriousness of this problem may be subjectively evaluated by the user and reflected in the risk or importance in the risk expression dictionary 14a. These pieces of information may also be used in combination.

[0099] Also, the weight variable w １ ~w ７ is a variable that adjusts the influence of each risk variable, and is set according to the topic and participants of the conversation that is the target of the risk value calculation process. For example, if there is a participant who is not good at guessing the meaning of ambiguous statements, the weight variable w １ ~w ３ By increasing the value of , the value in the risk expression dictionary 14a is more strongly reflected in the risk value.

[0100] In addition, in a case where participants in a frequently held conference have a deep understanding of the background of each other's statements, the weight variable w ７ This allows the value of risk variables based on expressions that appear frequently or expressions that are closely related to business to be strongly reflected in the risk value.

[0101] In addition, when there is a specific relationship such as a one-on-one conversation between a superior and a subordinate, the weight variable w ４ , w ５ By increasing the value of the risk variable p ｉ４ , risk variable p based on the social attribute DB 14d ｉ５ The value of will be strongly reflected in the risk value.

[0102] In addition, when a conversation between speakers with specific superior skills contains utterances or expressions that are unique to that individual, the weight variable w ４ , w ６ By increasing the value of the risk variable p ｉ４ , risk variable p based on the personal characteristics DB 14e ｉ６ The value of will be strongly reflected in the risk value.

[0103] Also, the weight variable w１ ~w ７ may be dynamically adjusted based on a judgment of the quality of consensus reached in the most recent dialogue or discussion, measurements of satisfaction with past dialogues, measurements of proficiency with the topic of the conversation, etc.

[0104] The risk value of each word calculated by the above formulas (1) to (4) can also be used to calculate the risk of an entire sentence or a group of expressions (for example, a phrase or clause) made up of multiple words. In this case, the risk value r calculated for the words included in each sentence is ｉ The average value of the risk value r １ ~r Ｎ The risk value of the sentence may be calculated by weighting each value of the risk expression dictionary 14a based on the risk level or importance or a preset value. ｉ The sum, product, etc. can also be used.

[0105] The non-verbal risk calculation unit 152 calculates a risk value that indicates the magnitude of risk of data related to the input non-verbal expression, by referring to information related to risk contained in the non-verbal risk expression DB 14b of the storage unit 14. For example, the non-verbal risk calculation unit 152 obtains information related to the risk corresponding to the usage situation of the user who made the expression from the non-verbal risk expression DB 14b of the storage unit 14, and calculates the risk value by referring to the information related to the risk.

[0106] Here, the risk value calculation method by the non-verbal risk calculation unit 152 will be explained. First, the non-verbal risk calculation unit 152 obtains the calculation results of each calculation method using the following formula (5). P ｉｊ is the calculation result of the calculation method j for the i-th expression in the series of expressions. ｊ (x ｍ ) is a function that performs calculations according to the calculation method j. Also, i is a subscript that indicates the number of an expression in a series of expressions (for example, in a conversation, the number of turns, the number of words, etc.). Also, j is a subscript that indicates the calculation method. ｍ is the P for risk element m ｉｊ is a variable for calculating f ｊIt may consist of multiple values ​​depending on the risk. m is a subscript indicating the risk factor.

[0107]

[0108] The calculation method described above differs depending on the risk element of each expression. For example, the non-verbal risk calculation unit 152 calculates a value for the risk element "speech rate" of voice data using the calculation method "number of characters / speech time." For example, the non-verbal risk calculation unit 152 calculates a value for the risk element "speech volume" of voice data using the calculation method "number of characters per utterance." For example, the non-verbal risk calculation unit 152 calculates a risk element "pitch" of voice data using the calculation method "average value of frequency of uttered voice" or "variance (range of change) of frequency of uttered voice." For example, the non-verbal risk calculation unit 152 calculates a risk element "volume" of voice data using the calculation method "average value of volume of uttered voice (amplitude of voice waveform)" or "maximum value of volume of uttered voice (amplitude of voice waveform)."

[0109] Furthermore, for example, the non-verbal risk calculation unit 152 calculates the "number of characters" risk factor for text data as the "number of characters in one message" calculation method. For example, the non-verbal risk calculation unit 152 calculates the "amount of movement" risk factor for video data related to the user's physical movements as the "amount of movement of peripheral parts such as the hands and head" calculation method. For example, the non-verbal risk calculation unit 152 calculates the "speed" risk factor for video data related to the user's physical movements as the "speed of movement of peripheral parts such as the hands and head" calculation method. For example, the non-verbal risk calculation unit 152 calculates the "average amount of movement of feature points from a neutral expression" or the "maximum amount of movement of feature points from a neutral expression" calculation method for the "amount of change" risk factor for image data related to the user's facial expression. Here, for facial expressions, emotion information obtained from commercially available technology (emotion estimation from facial expressions) is used (for example, the presence or absence of negative emotions and their frequency are considered as risk influences, etc.).

[0110] For example, the non-verbal risk calculation unit 152 calculates the speaking rate and speech volume using text obtained by performing speech recognition on speech acquired by a microphone. Furthermore, for example, the non-verbal risk calculation unit 152 calculates the number of characters from the number of characters in a message that is input and sent from an input device such as a keyboard. Furthermore, for example, the non-verbal risk calculation unit 152 calculates the pitch and volume by analyzing the waveform of speech acquired by a microphone.

[0111] Furthermore, for example, the non-verbal risk calculation unit 152 can calculate specific risk expressions such as sighs and clicking of the tongue from speech waveforms using calculation models generated in advance, and may utilize a calculation method to derive whether or not these expressions occur and how often they occur (for example, to detect their occurrence, pattern matching of characteristic waveforms may be performed, or an estimation model that has undergone machine learning in advance may be used).

[0112] Additionally, for example, the nonverbal risk calculation unit 152 may also calculate speech intonation (such as suddenly raising or raising the voice, age-specific intonation, etc.) using a model previously trained as a specific speech pattern, and utilize a method for deriving whether or not such intonation occurs, its frequency, etc. For example, the nonverbal risk calculation unit 152 may detect sudden loud or high-pitched speech as an emotional expression and use the occurrence of such speech as a nonverbal risk value, or may detect intonation specific to young people and calculate the difference from the intonation of an older listener (such as the similarity of waveforms) to use as a nonverbal risk value for the older listener.

[0113] Furthermore, for example, the non-verbal risk calculation unit 152 may calculate movements and facial expressions using feature points (coordinates of the fingers or head, or coordinates of any feature points on the face) calculated using an existing feature point extraction method using video captured by a camera. Note that while the description up to this point has been based on the assumption that an input device is built into a laptop PC, any dedicated device (for example, a motion capture device or a facial tracker) connected (whether wired or wireless) to a user terminal may also be used.

[0114] Then, the non-verbal risk calculation unit 152 determines whether the value calculated by each calculation method is within the range of the risk threshold using the following formula (6), and calculates the non-verbal risk value r' ｉｊ The non-verbal risk value r' ｉｊ is the non-linguistic risk value calculated for the i-th expression in the series of expressions using calculation method j. ｊ , β ｊ is a risk threshold corresponding to calculation method j. Note that there may be one or more risk thresholds.

[0115]

[0116] The non-verbal risk calculation unit 152 calculates the non-verbal risk value r' ｉｊ To calculate the non-verbal risk value r', the absolute value of the difference between the value calculated by each calculation method and the risk threshold is taken using the following formula (7). ｉｊ It may also be possible to use the following.

[0117]

[0118] The non-verbal risk calculation unit 152 calculates the non-verbal risk value r' ｉｊ To calculate the non-verbal risk value r', the value calculated by each calculation method is divided by the risk threshold using the following formula (8), and the value obtained by dividing the quotient by the risk threshold so that the non-verbal risk value increases as the quotient becomes farther from 1.0 is used. ｉｊ One or more of the non-verbal risk values ​​calculated by any of the formulas (6) to (8) may be used as the overall evaluation value R of the non-verbal risk. ｉ The overall evaluation value R ｉ is the overall assessment value of non-linguistic risk for the i-th expression in the series of expressions.

[0119]

[0120] For example, the non-verbal risk calculation unit 152 calculates the total non-verbal risk evaluation value R by summing the weighted values ​​calculated by each calculation method using the following formula (9): ｉ Here, w ｊ is a weight variable that adjusts the influence of each risk variable, and N is the number of calculation methods used to calculate the non-verbal risk value.

[0121]

[0122] Furthermore, for example, the non-verbal risk calculation unit 152 calculates the overall non-verbal risk evaluation value R by multiplying the values ​​calculated by each calculation method by the weights using the following formula (10): ｉ Here, i, j, m, and N are integers greater than 0.

[0123]

[0124] Furthermore, for example, the non-verbal risk calculation unit 152 calculates the overall evaluation value R ｉ may be combined with the risk value calculated by the language risk calculation unit 151, or may be used as one of the terms in the calculation formula to calculate a new risk value that includes risks from both the language and non-language aspects.

[0125] Furthermore, for example, when the non-verbal risk calculation unit 152 calculates the overall evaluation of the non-verbal risk value using the above-mentioned formulas (6) to (8), it uses the non-verbal risk value r' that has a high risk importance (for example, above a certain threshold, or in the top three, etc.) as the overall evaluation value. ｉｊ In addition, for example, the non-verbal risk calculation unit 152 may select the weight coefficient w ｊ The risk importance corresponding to each calculation method can be used.

[0126] Furthermore, for example, the non-verbal risk calculation unit 152 calculates a weighting coefficient w based on the values ​​of the relationship DB 14c and the social attribute DB 14d. ｊ For example, when the relationship is not close, or when there is a large difference in the length of time that the person has been in the organization or in their position, the non-verbal risk calculation unit 152 may adjust the weighting coefficient w ｊ The value is adjusted to be larger, so that the calculated overall evaluation of the non-verbal risk value becomes larger.

[0127] Furthermore, for example, the non-verbal risk calculation unit 152 quantitatively evaluates the difference in risk importance between the users participating in the dialogue, and calculates a weighting coefficient w ｊFor example, the non-verbal risk calculation unit 152 may calculate the average of the risk importance levels according to the calculation methods among users and use this value as the weighting coefficient w ｊ and may be adjusted so that the weighting coefficients of the calculation method used for the overall evaluation of non-verbal risk scores are uniform across all participants.

[0128] The calculation method is not limited to the above, and for example, the non-verbal risk calculation unit 152 may use an existing method to detect characteristic movements such as nodding, and calculate the non-verbal risk value using the frequency, amount, speed, etc. In this case, for example, the non-verbal risk calculation unit 152 may calculate the frequency of the sender's nodding, and calculate the risk that frequent nodding will bother the receiver.

[0129] Furthermore, the non-verbal risk calculation unit 152 may detect facial expressions using an existing method, quantitatively calculate the proportion of specific emotions (facial expressions) included, and calculate the non-verbal risk value. For example, the non-verbal risk calculation unit 152 may calculate the risk that the recipient will become intimidated if a certain level of anger is included. In addition, the non-verbal risk calculation unit 152 may measure gaze information using an eye tracker and use the results to calculate the non-verbal risk value. For example, the non-verbal risk calculation unit 152 may calculate the risk that the recipient will feel glared at or intimidated when gazing at an area where a specific person is present for a certain period of time or more.

[0130] Returning to the description of Fig. 1 , the determination unit 15d determines the dialogue partner's understanding state of the expression using information about the dialogue partner. Specifically, the determination unit 15d detects the dialogue partner's head movement from video information of the dialogue partner acquired in advance prior to processing by the adjustment unit 15e, which will be described later, to determine which of the understanding states illustrated in Fig. 11 applies. Alternatively, the determination unit 15d determines the dialogue partner's understanding state by detecting the dialogue partner's head movement from the acquired acceleration sensor.

[0131] 12, the determination unit 15d stores the determined understanding state of the conversation partner in an understanding state history 14h of the storage unit 14. The understanding state history 14h may include time information such as the start time and end time of the understanding state expression.

[0132] The adjustment unit 15e adjusts the calculated risk value of an expression constituting communication in accordance with the determined understanding state of the dialogue partner. For example, the adjustment unit 15e refers to the risk adjustment DB 14i illustrated in Fig. 13 and adjusts the calculated risk value of the expression to be processed using a risk value adjustment method that matches the condition of the determined understanding state of the dialogue partner.

[0133] Here, Fig. 17 is a diagram for explaining the presentation process of the first embodiment. Fig. 17 illustrates an example of the presentation process for the dialogue shown in Fig. 17(a). That is, when Speaker 1 expresses "It's about that particular matter," and Speaker 2, the dialogue partner, tilts his or her head in response to this expression, the determination unit 15d determines in advance from the video information the understanding state expressed by Speaker 2, "motion - tilt of head." Then, the determination unit 15d stores this in the understanding state history 14h together with a timestamp, as illustrated in Fig. 17(b).

[0134] The acquisition unit 15a then acquires the speech information of the expression of the speaker 1 that constitutes the dialogue, "It's about that particular matter," and the calculation unit 15c calculates the risk value of the risk expression for each element that constitutes this expression, as illustrated in FIG.

[0135] Next, the adjustment unit 15e refers to the risk adjustment DB 14i illustrated in Fig. 13 and adjusts the risk value of the risk expression as illustrated in Fig. 17(d). For example, in the example illustrated in Fig. 17(d), since "motion-head tilt" of a person other than the speaker is included within -30 seconds from the current time, the adjustment unit 15e multiplies the calculated risk value of "2.0" by 1.5 to adjust it to an adjusted risk value of "3.0".

[0136] Returning to the description of Figure 1, the presentation unit 15f presents to the user information generated based on the calculated risk value or the adjusted risk value. For example, the presentation unit 15f presents to the user, via the output unit 12, as intervention information for the risk expression constituting the communication, the risk value calculated by the calculation unit 15c or the risk value adjusted by the adjustment unit 15e, or information such as a message, notification sound, or vibration generated based on the risk value.

[0137] For example, the presenting unit 15f presents information using the calculated language risk value, the comprehensive evaluation value of non-verbal risk, or the risk value combined with language risk. Additionally, for example, the presenting unit 15f may present the results to the user separately as values ​​indicating communication risk, such as the non-verbal risk value using each calculation method, the risk value based on language expression, or the risk value calculated by combining both.

[0138] Furthermore, when presenting information, the presenting unit 15f may calculate at least one of the verbal and non-verbal risk values ​​of the recipient of the conversation partner, and present information to the speaker according to the risk on the recipient side.

[0139] Furthermore, the presenting unit 15f may calculate a risk value for at least one of the language and non-language of both the sender and the recipient, and present information according to the difference between the calculated risk values. For example, when the difference between the risk values ​​is equal to or greater than a threshold value or when the ratio of the risk values ​​is equal to or greater than a threshold value, the presenting unit 15f presents the risk values ​​or presentation information such as a message, notification sound, or vibration generated based on the risk values.

[0140] In addition, the presentation unit 15f may, for example, calculate a risk value for at least one of the verbal and non-verbal expressions of each participant as a recipient, and if the average value exceeds a certain level, detect the expression as being risky for the conversation as a whole and present the information to all participants.

[0141] In addition, although the example has been described in which the presentation unit 15f calculates and uses the communication risk (such as difficulty in understanding) when the recipient receives the non-verbal expression of the sender by utilizing the risk threshold of the receiver, when calculating the non-verbal risk of the sender, the presentation unit 15f may set a threshold from the perspective of the ease of expression of the sender himself / herself and present the information. For example, when the amount of speech is less than the risk threshold or the speech rate is slower than the risk threshold, the presentation unit 15f presents a message encouraging reconsideration or improvement, indicating that there is a risk that the explanation will be difficult.

[0142] FIG. 18 is a diagram illustrating intervention information. In the example shown in FIG. 18, information on calculated risk values ​​is presented on an online conference tool as intervention information notifying a risk. For example, the presentation unit 15f highlights text containing words with high risk values. For example, the presentation unit 15f changes the color of the text containing words with high risk values, underlines them, makes the font larger, makes them bold, or highlights them.

[0143] Alternatively, the presentation unit 15f may display text with a high risk value in a format that attracts the user's attention, such as by displaying the text as a pop-up. Furthermore, the presentation unit 15f may display words with a high risk value in a position that is easily noticeable to the user.

[0144] Furthermore, instead of or in addition to displaying text, the presentation unit 15f may present words with a high risk value using audio information or tactile information. For example, the presentation unit 15f may increase the volume of the notification sound for words with a high risk value, use a sound that is likely to make the user feel the need to take action, such as a sound for an incorrect answer, increase the vibration, or use a frequency or pattern of vibration that is likely to attract attention. Conversely, the presentation unit 15f may use the sound of a correct answer on a quiz show when the risk value is low.

[0145] The presenting unit 15f may also display a CG or image agent on the screen and present words with high risk values ​​in the form of advice from the agent.

[0146] In the online conference tool illustrated in Fig. 18, the presentation unit 15f may present the same screen to user 1 and user 2, or may present different screens. For example, when the presentation unit 15f presents the same screen, both users are more likely to feel equal and mutual understanding is promoted. On the other hand, when the presentation unit 15f presents different screens, although the promotion of mutual understanding is reduced, users with better communication skills can more flexibly engage in dialogue, thereby improving the efficiency of communication.

[0147] Furthermore, the presentation unit 15f may adjust the presentation method, cancel the presentation itself, etc. For example, when the risk value is lower than a predetermined threshold, or in response to a user operation or user characteristics, the presentation unit 15f may present the information in a different format, such as by reducing the level of highlighting or turning off the highlighting for a user who is bothered by the highlighting, or by changing visual information to sound information. Furthermore, the user characteristics may be reflected differently depending on the composition of the conference participants.

[0148] Furthermore, when the risk value is adjusted by the processing of the discrimination unit 15d and the adjustment unit 15e, the presentation unit 15f presents intervention information based on the adjusted risk value, as illustrated in FIG. 17(e).

[0149] As described above, when the risk adjustment DB 14i specifies a change in association with the presentation method instead of adjusting the risk value, the presentation unit 15f may present the intervention information using the presentation method changed by the adjustment unit 15e. As a result, for example, the presentation method may be changed from presenting high risk values ​​in red and medium risk values ​​in orange to presenting high risk values ​​in orange and medium risk values ​​in red.

[0150] [Presentation Process] Next, an example of the presentation process by the presentation device 10 according to the present embodiment will be described with reference to Fig. 19. Fig. 19 is a flowchart showing the procedure of the presentation process. The flowchart in Fig. 19 starts, for example, when the user performs an operation input to instruct the start of the process.

[0151] First, the acquisition unit 15a acquires an individual usage situation corresponding to the dialogue situation (step S1). For example, the acquisition unit 15a acquires information about the user engaging in the dialogue or information about the work domain. Then, the acquisition unit 15a acquires expressions constituting the dialogue (step S2). For example, the acquisition unit 15a acquires words constituting the dialogue sentence to be processed as expressions constituting the dialogue. The acquisition unit 15a also acquires the speech rate, speech volume, and pitch, as well as image data, video data, or chat message text, identified from the voice data constituting the dialogue to be processed.

[0152] Next, the calculation unit 15c calculates a risk value for the trouble by referring to the risk expression dictionary 14a and the non-verbal risk expression DB 14b according to the individual usage situation (step S3). For example, the calculation unit 15c calculates the risk value by using the product of multiple pieces of information in the risk expression dictionary 14a that have been weighted in a predetermined manner. Alternatively, the calculation unit 15c calculates the risk value by using the sum of multiple pieces of information in the risk expression dictionary 14a that have been weighted in a predetermined manner. Furthermore, for example, the calculation unit 15c obtains information on the risk corresponding to the usage situation of the user who made the expression from the non-verbal risk expression DB 14b in the storage unit 14, and calculates the risk value by referring to the information on the risk.

[0153] Furthermore, the calculation unit 15c may calculate the risk value using one or more pieces of information from the relationship DB 14c, the social attribute DB 14d, the personal characteristic DB 14e, and the communication history DB 14f.

[0154] The adjustment unit 15e adjusts the calculated risk value of the expression that constitutes the communication, depending on the determined understanding state of the dialogue partner (step S4).

[0155] Specifically, the determination unit 15d determines the understanding state of the dialogue partner's expression in advance using acquired video information of the dialogue partner in communication, and stores the determined understanding state in the understanding state history 14h.

[0156] Then, the adjustment unit 15e refers to the understanding state history 14h and the risk adjustment DB 14i, and adjusts the risk value calculated for the expression using a risk value adjustment method that matches the conditions of the understanding state of the dialogue partner determined for the expression to be processed.

[0157] The presentation unit 15f presents to the user information generated based on the calculated risk value or the adjusted risk value (step S5). For example, the presentation unit 15f presents, as intervention information for the risk expression constituting the communication, the risk value calculated by the calculation unit 15c or the risk value adjusted by the adjustment unit 15e, or intervention information such as a message, notification sound, or vibration generated based on the risk value, to the user via the output unit 12. This completes the series of presentation processes.

[0158] Second Embodiment The processing of the presentation device 10 is not limited to the above embodiment. In the following, only differences from the presentation processing of the presentation device 10 of the above embodiment will be described, and a description of common points will be omitted.

[0159] For example, the presentation unit 15 f may present the adjusted risk value and the determined understanding state of the dialogue partner in association with each other. That is, when presenting intervention information generated based on the adjusted risk value to the user, the presentation unit 15 f may also present the determined understanding state of the dialogue partner.

[0160] The risk adjustment method of the risk adjustment DB 14i may be determined based on the understanding state of the dialogue partner from the start time of the utterance to the present. In this case, when acquiring the voice information of the dialogue to be processed, the acquiring unit 15a also acquires the content of the utterance and the start time of the dialogue sentence.

[0161] 20 and 21 are diagrams for explaining the presentation process of the second embodiment. An example of the data structure of the risk adjustment DB in this case is shown in Fig. 20. In the example shown in Fig. 20, the condition for expressing the understanding state specifies a method for adjusting the risk value in accordance with the understanding state of entities other than the speaker from the start of the utterance to the present.

[0162] 21 illustrates an example of a presentation process for the dialogue shown in FIG. 21(a). In other words, when Speaker 1 expresses "It's about that particular matter," and Speaker 2, the dialogue partner, tilts his or her head in response to this expression, the determination unit 15d first determines the understanding state expressed by Speaker 2, "motion - tilt of head," from the video information. Then, as illustrated in FIG. 21(b), the determination unit 15d stores the determined understanding state together with a timestamp in the understanding state history 14h.

[0163] Next, the acquisition unit 15a acquires the speech information of the expression of the speaker 1 that constitutes the dialogue, "It's about that particular matter," and the start time of the utterance. Furthermore, the calculation unit 15c calculates the risk value of the risk expression for each element that constitutes this expression, as illustrated in FIG. 21(c).

[0164] Next, the adjuster 15e refers to the risk adjustment DB 14i illustrated in FIG. 20 and adjusts the risk value of the risk expression as illustrated in FIG. 21(d).

[0165] Then, the presentation unit 15f, when presenting the intervention information of the risk expression (ambiguous expression), also displays the understanding status of the dialogue partner, as shown in Fig. 21(e) . In the example shown in Fig. 21(e) , the understanding status of the dialogue partner is presented as "Understanding status: ×" and "The person tilted their head during the utterance."

[0166] The presentation unit 15f may present a dialogue in which an understanding state is expressed regardless of whether a risk expression is included, or may present a dialogue in which both a risk expression and an understanding state are included.

[0167] [Third Embodiment] The adjustment unit 15e may adjust the calculated risk value based on an appropriate level of each conversation partner's understanding state, which is derived using the history of each conversation partner's understanding state. That is, the adjustment unit 15e may adjust the risk value according to the tendency of each user's average expression of understanding state, for example, by determining that the understanding state is abnormal if the tendency is significantly different from the average. This makes it possible, for example, to adjust the risk higher for a conversation partner who frequently nods and smiles, if the conversation partner rarely nods or smiles.

[0168] In this case, the understanding state history 14h includes the start time and end time of each understanding state expression. The storage unit 14 also stores an individual appropriate amount DB 14k. The individual appropriate amount DB 14k indicates the appropriate amount of each user's understanding state expression using the average understanding state of each user calculated from the understanding state history 14h of each user.

[0169] Here, Fig. 22 is a schematic diagram illustrating the general configuration of a presentation device according to the third embodiment. Figs. 23 to 26 are diagrams for explaining the third embodiment. In this case, as illustrated in Fig. 22, the storage unit 14 stores a personal optimum amount DB 14k.

[0170] In the personal appropriate amount DB 14k, for example, the average frequency of expression of the understanding state is calculated as the tendency of each user to express their average understanding state from each user's understanding state history 14h for the past 100 seconds, and the appropriate amount of each user's understanding state is calculated using the calculated average frequency of expression of the understanding state.

[0171] The individual optimum amount may be calculated from the average value and standard deviation for each user.Facial expressions are defined by the number of seconds, and actions are defined by the number of times, but are not limited to this.

[0172] Furthermore, the individual optimum level is not limited to calculation from the expression of each user's understanding state over a predetermined period of time, such as the past 100 seconds, but may be calculated from the expression of the understanding state in a predetermined number of utterances.

[0173] Here, Fig. 23 illustrates an example of the data configuration of the individual optimum amount DB. Specifically, the individual optimum amount DB 14k includes, for each speaker in a conversation, the average frequency of each understanding state and the individual optimum amount, as illustrated in Fig. 23. In the example illustrated in Fig. 23, the individual optimum amount is specified as outside the optimum range (positive) and outside the optimum range (negative).

[0174] 24 shows an example of the data structure of the understanding situation history 14h in this case. As shown in FIG. 24, the understanding situation history 14h in this case includes the start time and end time of each understanding situation expression.

[0175] 25 shows an example of the data configuration of the risk adjustment DB in this case. In the risk adjustment DB 14i, a method of adjusting the risk value is defined for each understanding state, taking into consideration the possibility that the state is "not understood or difficult for the other person to ask" when it differs from the individual's tendency.

[0176] For example, if the number of "smiling expressions" is extremely higher than average, it is judged that the situation is unlikely to be "not understanding or difficult for the other person to ask." On the other hand, if the number of "smiling expressions" is extremely lower than average, it is judged that the situation is likely to be "not understanding or difficult for the other person to ask."

[0177] Also, if there are significantly more "frowning facial expressions" than average, it is judged that there is a high possibility that the situation is "not understanding or difficult for the other person to ask." On the other hand, if there are significantly fewer "frowning facial expressions" than average, it is judged that there is a low possibility that the situation is "not understanding or difficult for the other person to ask."

[0178] Also, if the number of "actions - nodding" is extremely higher than average, it is judged that the situation is unlikely to be "not understanding or difficult for the other person to ask." On the other hand, if the number of "actions - nodding" is extremely lower than average, it is judged that the situation is likely to be "not understanding or difficult for the other person to ask."

[0179] Also, if the number of "movements - tilting head" is extremely higher than the average, it is judged that there is a high possibility that the situation is "not understanding or difficult for the other person to ask." On the other hand, if the number of "movements - tilting head" is extremely lower than the average, it is judged that there is a low possibility that the situation is "not understanding or difficult for the other person to ask."

[0180] In the example shown in Figure 25, a method of adjusting the risk value is specified depending on whether each understanding state is outside the appropriate range (positive) or outside the appropriate range (negative). For example, if "Expression - Smiling" is outside the appropriate range (positive), it is specified that the risk value should be adjusted to the calculated risk value (ri) x 0.75. On the other hand, if "Expression - Smiling" is outside the appropriate range (negative), it is specified that the risk value should be adjusted to the calculated risk value (ri) x 1.25.

[0181] 26 illustrates an example of a presentation process for the dialogue shown in Fig. 26(a). That is, when Speaker 2, the dialogue partner, smiles and nods in response to Speaker 1's expression "Take lots of pictures of giraffes," the determination unit 15d first determines the understanding state expressed by Speaker 2 from the video information. Then, the determination unit 15d stores the state of understanding expressed by Speaker 2 in the understanding state history 14h together with a timestamp, as illustrated in Fig. 26(b).

[0182] Next, the acquisition unit 15a acquires speech information of an expression of the speaker 1 that constitutes the dialogue, "Take lots of pictures of giraffes." Furthermore, the calculation unit 15c calculates a risk value of the risk expression for each element that constitutes this expression, as illustrated in FIG. 26(c).

[0183] Next, the adjuster 15e refers to the risk adjustment DB 14i illustrated in FIG. 25 and adjusts the risk value of the risk expression as illustrated in FIG. 26(d).

[0184] 26(d), the risk value is adjusted to "0.9" which is smaller than the calculated risk value "1.25" and is excluded from presentation. Therefore, the presentation unit 15f does not present the intervention information.

[0185] Other Embodiments In the presentation device 10 of the above embodiment, risk values ​​based on ambiguous expressions are calculated, but this is not limiting. For example, the presentation device 10 can calculate risk values ​​for stereotyped expressions instead of or in addition to risk values ​​based on ambiguous expressions. In this case, the calculation unit 15c calculates the risk values ​​using information on risks based on stereotypes.

[0186] For example, the risk variable p based on the risk expression dictionary 14a ｉ１ ~p ｉ３ As the risk level, a risk type set from the perspective of stereotypes, ambiguous expressions, or the like may be applied.

[0187] In addition, the risk variable p ｉ４ or a risk variable p using the social attribute DB 14dｉ５ As the criteria, the background common to the speakers, the degree of strength of attributes, the degree of overlap, etc. may be applied.

[0188] In addition, the risk variable p ｉ６ The speaker's background, attributes, etc. may be applied as the context.

[0189] Specifically, for example, when the degree of overlap between the speakers' common backgrounds is low, it is assumed that there is a high possibility that the speakers will misinterpret the language, and the risk variable p ｉ４ , risk variable p ｉ６ The value of the risk variable p is set to a large value. In addition, the background of the speaker, such as gender and age, is compared with the background of the parties in past trouble cases, and the risk value in past trouble cases is used to set the risk variable p ｉ６ This allows the calculation unit 15c to calculate a risk value based on a stereotype.

[0190] [Effects] As described above, in the presentation device 10 of this embodiment, the memory unit 14 stores information regarding the risk of trouble for each expression that constitutes communication. The calculation unit 15c refers to the risk information stored in the memory unit 14 and calculates a risk value that represents the magnitude of risk for the expression that constitutes communication. The determination unit 15d uses information about the dialogue partner in communication to determine the dialogue partner's understanding of the expression. The adjustment unit 15e adjusts the calculated risk value of the expression depending on the determined understanding state.

[0191] This makes it possible to evaluate the risks inherent in the expressions used in communication, taking into account the understanding of the other party.

[0192] The presenting unit 15f presents the adjusted risk value and the determined understanding state of the dialogue partner in association with each other, thereby making it possible to notify the user of the understanding state of the dialogue partner regarding the risk expression.

[0193] Furthermore, the adjustment unit 15e adjusts the calculated risk value based on the appropriate level of the understanding state of each dialogue partner, which is derived using the history of the understanding state of each dialogue partner, thereby making it possible to evaluate risk expressions in consideration of the average reactions of the dialogue partners.

[0194] [Program] A program written in a computer-executable language may be created to execute the processes executed by the presentation device 10 according to the above embodiment. In one embodiment, the presentation device 10 can be implemented by installing a presentation program that executes the above presentation process as package software or online software on a desired computer. For example, by executing the presentation program on an information processing device, the information processing device can function as the presentation device 10. The information processing device referred to here includes desktop and notebook personal computers. Other information processing devices also include mobile communication devices such as smartphones, mobile phones, and PHS (Personal Handyphone System) phones, as well as slate devices such as PDAs (Personal Digital Assistants), and xR and metaverse-related devices. The functions of the presentation device 10 may also be implemented on a cloud server.

[0195] 27 is a diagram showing an example of a computer that executes a presentation program. The computer 1000 includes, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0196] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1031. The disk drive interface 1040 is connected to a disk drive 1041. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1041. The serial port interface 1050 is connected to a mouse 1051 and a keyboard 1052, for example. The video adapter 1060 is connected to a display 1061, for example.

[0197] Here, the hard disk drive 1031 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. The various pieces of information described in the above embodiments are stored in the hard disk drive 1031 or memory 1010, for example.

[0198] The presentation program is stored in the hard disk drive 1031 as, for example, a program module 1093 in which instructions to be executed by the computer 1000 are written. Specifically, the program module 1093 in which each process executed by the presentation device 10 described in the above embodiment is written is stored in the hard disk drive 1031.

[0199] Data used for information processing by the presentation program is stored as program data 1094, for example, in the hard disk drive 1031. Then, the CPU 1020 reads the program module 1093 and the program data 1094 stored in the hard disk drive 1031 into the RAM 1012 as necessary, and executes each of the above-described procedures.

[0200] The program module 1093 and program data 1094 related to the presentation program are not limited to being stored in the hard disk drive 1031, and may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1041. Alternatively, the program module 1093 and program data 1094 related to the presentation program may be stored in another computer connected via a network such as a LAN or a WAN (Wide Area Network), and read by the CPU 1020 via the network interface 1070.

[0201] Although the present invention has been described above as an embodiment, the present invention is not limited to the description and drawings that form part of the disclosure of the present invention. In other words, other embodiments, examples, and operational techniques that can be made by those skilled in the art based on the present invention are all included in the scope of the present invention.

[0202] 10 Presentation device 11 Input unit 12 Output unit 13 Communication control unit 14 Memory unit 14a Risk expression dictionary 141 Master dictionary 142 Individual dictionary 14b Non-verbal risk expression DB 14c Relationship DB 14d Social attribute DB 14e Personal characteristic DB 14f Communication history DB 14g Understanding status list 14h Understanding status history 14i Risk adjustment DB 14k Personal appropriate amount DB 15 Control unit 15a Acquisition unit 15b Individual dictionary generation unit 15c Calculation unit 151 Verbal risk calculation unit 152 Non-verbal risk calculation unit 15d Discrimination unit 15e Adjustment unit 15f Presentation unit

Claims

1. A presentation device characterized by having: a memory unit that stores information regarding the risk of trouble for each expression that constitutes communication; a calculation unit that refers to the information regarding risk stored in the memory unit and calculates a risk value that represents the magnitude of risk of the expression that constitutes communication; a discrimination unit that uses information about the dialogue partner in the communication to determine the dialogue partner's understanding of the expression; and an adjustment unit that adjusts the calculated risk value of the expression in accordance with the determined understanding state.

2. The presentation device according to claim 1, further comprising a presentation unit that presents the adjusted risk value in association with the determined understanding state of the conversation partner.

3. The presentation device described in claim 1, characterized in that the adjustment unit adjusts the risk value based on an appropriate amount of understanding status of each dialogue partner derived using the history of understanding status of each dialogue partner.

4. A presentation method executed by a presentation device, wherein the presentation device has a memory unit that stores information regarding the risk of trouble for each expression that constitutes communication, and the presentation method includes: a calculation step of calculating a risk value that represents the magnitude of risk of an expression that constitutes communication by referring to the information regarding risk stored in the memory unit; a determination step of determining the level of understanding of the expression of a dialogue partner in the communication by using information about the dialogue partner; and an adjustment step of adjusting the calculated risk value of the expression in accordance with the determined level of understanding.

Images