Inference device, inference method, and inference program
The estimation system uses AI to analyze dialogue information and estimate personal characteristics, addressing the challenge of accurately assessing interlocutor relationships by employing deep learning for precise relationship estimation and recommending suitable interactions.
Patent Information
- Application Number
- PCT/JP2023/020384
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2024-12-05
AI Technical Summary
Conventional technologies face challenges in accurately estimating the relationship between interlocutors during conversations, often focusing solely on communication skills and failing to provide precise relationship assessments.
An estimation system that acquires dialogue information, including multimodal data, to estimate personal characteristics of interlocutors using AI technologies like deep learning, enabling accurate relationship estimation between multiple individuals.
The system achieves highly accurate estimation of relationships between interlocutors, improving personal characteristic estimation through multitask learning and recommending smooth human relationships such as romantic and friendship connections.
Smart Images

Figure JP2023020384_05122024_PF_FP_ABST
Abstract
Description
Estimation device, estimation method, and estimation program
[0001] The present invention relates to an estimation device, an estimation method, and an estimation program.
[0002] There is known a technique for estimating communication skills of interlocutors from non-verbal information during a conversation and presenting information for promoting smooth communication (see, for example, Patent Document 1).
[0003] Japanese Patent Application Laid-Open No. 2017-116716
[0004] However, conventional techniques may not be able to accurately estimate the relationship between interlocutors. For example, conventional techniques simply estimate the communication skills of speakers and present information that will improve those communication skills, so they may not be able to accurately estimate the relationship between interlocutors (such as favor, indifference, or dislike). The present invention aims to solve this problem and enable highly accurate estimation of the relationship between interlocutors.
[0005] In order to solve the above-mentioned problems, the present invention is characterized by having an acquisition unit that acquires dialogue information regarding a dialogue between multiple interlocutors, a personal characteristic estimation unit that estimates personal characteristics of each of the multiple interlocutors based on the dialogue information acquired by the acquisition unit, and a relationship estimation unit that estimates the relationship between the multiple interlocutors based on the personal characteristics estimated by the personal characteristic estimation unit.
[0006] According to the present invention, it is possible to estimate the relationship between interlocutors with high accuracy.
[0007] FIG. 1 is a diagram illustrating an example of an estimation system. FIG. 2 is a diagram illustrating an example of a configuration of an estimation device. FIG. 3 is a diagram illustrating an example of an individual characteristic. FIG. 4 is a diagram illustrating an example of an individual characteristic. FIG. 5 is a diagram illustrating an example of an individual characteristic. FIG. 6 is a diagram illustrating an example of an individual characteristic. FIG. 7 is a diagram illustrating an example of an individual characteristic. FIG. 8 is a diagram illustrating an example of an individual characteristic. FIG. 9 is a diagram illustrating an example of an individual characteristic. FIG. 10 is a diagram illustrating an example of an individual characteristic. FIG. 11 is a diagram illustrating an example of an individual characteristic. FIG. 12 is a diagram illustrating an example of an individual characteristic. FIG. 13 is a diagram illustrating an example of an estimation result of an individual characteristic. FIG. 14 is a diagram illustrating an example of an estimation result of a relationship. FIG. 15 is a flowchart illustrating an example of a processing procedure executed by the estimation device. FIG. 16 is a diagram illustrating an example of a configuration of a computer that executes an estimation program.
[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings, but the present invention is not limited to the embodiment.
[0009] [Overview] An overview of an estimation system 1 will be described using Fig. 1. Fig. 1 is a diagram for explaining an example of an estimation system. The estimation system 1 acquires dialogue information using an estimation device 10 (described later), estimates personal characteristics of each of multiple interlocutors based on the dialogue information, and estimates relationships between the multiple interlocutors based on the personal characteristics.
[0010] Here, dialogue information refers to information about a dialogue between multiple interlocutors. Examples of dialogue information include multimodal information of each of the two parties in a two-party dialogue (image information such as facial images, audio information such as audio signals, and linguistic information such as speech transcripts). Personal characteristics refer to the characteristics of each of the multiple interlocutors. Examples of personal characteristics include multiple scales such as the Big 5 personality traits, self-esteem scales, and self-consciousness scales. The relationship between interlocutors refers to the relationship each of the multiple interlocutors has with the other. Examples of the relationship between interlocutors include romantic relationships, friendships, and family relationships. For example, romantic relationships include the degree of romantic relationship, such as whether one would like to give one's contact information to the other, and a love-liking scale for the other. Friendship relationships include the degree of intimacy one feels as a friend. Family relationships include the degree of intimacy one feels as a family.
[0011] For example, the estimation system 1 acquires dialogue information including multimodal information such as facial images, audio signals (audio waveforms), and speech transcripts of each person in a speed dating dialogue between a man and a woman using an estimation device 10, estimates personal characteristics of each person based on the dialogue information, and estimates the romantic relationship between each person and the other based on the personal characteristics. In this case, the estimation system 1 may perform the above-mentioned estimation using an estimation device 10 that utilizes AI (Artificial Intelligence) technology such as deep learning technology.
[0012] Thus, unlike conventional technologies, the estimation system 1 does not simply estimate the communication skills of interlocutors, but rather estimates personal characteristics including communication skills and the like using the estimation device 10, and then performs multitask estimation to estimate the relationships between multiple interlocutors from the estimated personal characteristics. As a result, the estimation system 1 enables highly accurate estimation of the relationships between interlocutors using the estimation device 10, such as hierarchical estimation of the relationships between interlocutors, which has not been possible with conventional technologies.
[0013] Furthermore, according to the estimation system 1, by collectively estimating various personal characteristics using a single estimation device 10, the estimation performance of personal characteristics can be improved due to the complementarity between the personal characteristics compared to when multiple estimation devices 10 estimate personal characteristics one by one. According to the estimation system 1, by estimating the relationships between multiple interlocutors based on the personal characteristics of each of the multiple interlocutors estimated by the estimation device 10, it is possible to highly accurately estimate the relationships between multiple interlocutors while taking into account the individual characteristics, such as the personality, of each of the multiple interlocutors. Furthermore, unlike matching systems based on the hobbies and interests of each of the multiple interlocutors, the estimation system 1 can recommend smooth human relationships, such as romantic relationships or friendships, based on the personal characteristics and relationships of each of the multiple interlocutors estimated by the estimation device 10. Therefore, according to the estimation system 1, the estimation system 1 can be applied to a system for recommending smooth human relationships, such as romantic relationships or friendships.
[0014] [Example of Configuration of Estimation Device] An example of the configuration of the estimation device 10 will be described using FIG. 2. FIG. 2 is a diagram showing an example of the configuration of the estimation device. The estimation device 10 acquires dialogue information, estimates personal characteristics of each of multiple interlocutors based on the dialogue information, and estimates relationships between the multiple interlocutors based on the personal characteristics. Examples of the estimation device 10 include a stationary computer such as a PC (Personal Computer), a mobile communication terminal such as a smartphone, a mobile phone, or a PHS (Personal Handy Phone System), and a terminal such as a PDA (Personal Digital Assistant). In the example shown in FIG. 2, the estimation device 10 includes an acquisition unit 11, a control unit 12, a presentation unit 13, and a storage unit 14.
[0015] [Acquisition Unit] The acquisition unit 11 acquires dialogue information related to a dialogue between multiple interlocutors. The acquisition unit 11 may acquire dialogue information including at least one of linguistic information and voice information related to the utterances of each of the multiple interlocutors, image information related to images of each of the multiple interlocutors, location information related to the location of the utterances, and attribute information related to the attributes of each of the multiple interlocutors. The acquisition unit 11 acquires dialogue information including at least one of linguistic information, voice information, image information, location information, and attribute information, allowing the estimation device 10 to estimate personal characteristics with higher accuracy. In the example shown in FIG. 2 , the acquisition unit 11 includes a language information acquisition unit 111, a voice information acquisition unit 112, an image information acquisition unit 113, a location information acquisition unit 114, and an attribute information acquisition unit 115.
[0016] The language information acquisition unit 111 acquires language information. An example of the language information acquisition unit 111 is a microphone. The audio information acquisition unit 112 acquires audio information. An example of the audio information acquisition unit 112 is a microphone. The image information acquisition unit 113 acquires image information. An example of the image information acquisition unit 113 is a camera. The position information acquisition unit 114 acquires position information. An example of the position information acquisition unit 114 is a calculation unit that calculates the relative position or order of an utterance in a dialogue. The attribute information acquisition unit 115 acquires attribute information. An example of the attribute information acquisition unit 115 is an input reception unit that receives input of attribute information via a UI (User Interface) or the like.
[0017] An example of acquisition by the acquisition unit 11 will be described below. The acquisition unit 11 acquires conversation information by recording conversation information between two interlocutors who are gathered in the same space using microphones (language information acquisition unit 111 and voice information acquisition unit 112) and a camera (image information acquisition unit 113) placed in front of each of the two interlocutors. Here, the same space may include not only a physical space but also a virtual space constructed on a computer. Furthermore, at least one of the two interlocutors may not only be a human being but also a humanoid such as an avatar or robot operating in a virtual space. The microphone records the speech of each of the multiple interlocutors, including their respective language information and voice information. The camera records video of each of the multiple interlocutors' faces, including their respective image information.
[0018] [Control Unit] The control unit 12 controls the entire estimation device 10. For example, the control unit 12 is configured with one or more processors having an internal memory storing programs defining various processing procedures and control information, and the processor executes each process using the programs and the internal memory. Examples of the control unit 12 include electronic circuits such as a central processing unit (CPU), a micro processing unit (MPU), and a graphics processing unit (GPU), and integrated circuits such as an application specific integrated circuit (ASIC) and a field programmable gate array (FPGA). In the example shown in FIG. 2 , the control unit 12 includes a model learning unit 121, an individual characteristic estimation unit 122, and a relationship estimation unit 123.
[0019] [Model Learning Unit] The model learning unit 121 uses the dialogue information acquired by the acquisition unit 11 and personal characteristic information regarding the personal characteristics of each of the multiple interlocutors to generate the trained model 141 by learning in which the dialogue information is input and the personal characteristic information is output. The model learning unit 121 may convert (encode) each piece of information included in the dialogue information acquired by the acquisition unit 11 into a vector, and use the dialogue information in which the vectors are integrated for each interlocutor and the personal characteristic information to generate the trained model 141 in which the dialogue information in which the vectors are integrated for each interlocutor is input and the personal characteristic information is output. For example, the model learning unit 121 deep-learns training data including the dialogue information, which is input information in which the vectors are integrated for each interlocutor, and the personal characteristic information, which is output information, and adjusts model parameters to generate the trained model 141 in which the dialogue information in which the vectors are integrated for each interlocutor is input and the personal characteristic information is output.
[0020] The model learning unit 121 may further use relationship information regarding the relationships between multiple interlocutors to generate the trained model 141, which takes dialogue information as input and outputs personal characteristic information and relationship information. For example, the model learning unit 121 generates the trained model 141 from training data including dialogue information, personal characteristic information, and relationship information by deep learning that trains the association between the dialogue information, which is input information, and the personal characteristic information and relationship information of each of the multiple interlocutors, which is output information, in an end-to-end manner. When the model learning unit 121 further uses relationship information to generate the trained model 141, the model learning unit 121 may generate the trained model 141 using relationship information regarding at least one of the relationship, the degree of the relationship, and the possibility of development of the relationship estimated by the relationship estimation unit 123 described below. In the example shown in Figure 1, the model learning unit 121 performs deep learning on learning data that includes relationship information regarding the possibility of development of a romantic relationship, such as "whether the man will give his contact information to the woman" and "whether the woman will give her contact information to the man," which is output information estimated by the relationship estimation unit 123, as well as personal characteristic information, and dialogue information, which is input information, to generate a learned model 141.
[0021] Here, all model parameters of the network of the trained model 141 are updatable values. Therefore, by training by the model training unit 121 (training through n epochs), all model parameters of the network of the trained model 141 (including model parameters for estimating the individual characteristics of each of the multiple interlocutors and model parameters for estimating the relationships between the multiple interlocutors) can be updated. In the example shown in FIG. 2 , the model training unit 121 has a model input unit 1211, an encoding unit 1212, an integration unit 1213, and a fusion unit 1214.
[0022] [Model Input Unit] The model input unit 1211 extracts input information to be input to the trained model 141 from the dialogue information acquired by the acquisition unit 11. The model input unit 1211 may extract the input information to be input to the trained model 141 from the dialogue information including at least one of language information, audio information, image information, position information, and attribute information acquired by the acquisition unit 11. In the example shown in (1) of Figure 1, the model input unit 1211 extracts the following information as input information from n consecutive utterances (n is any natural number equal to or greater than 1) in a dialogue between two people:・Speaker's language information: Transcription of the speaker's voice during that utterance ・Speaker's audio information: The speaker's audio signal during that utterance ・Speaker's image information: Video of the speaker's face during that utterance ・Receiver's audio information: The audio signal of participants other than the speaker during that utterance ・Receiver's image information: Video of the face of participants other than the speaker during that utterance ・Utterance position information: The position of the utterance in the dialogue, assigned to each utterance ・Speaker's attribute information: A binary value assigned to each utterance, indicating whether the speaker is the generator (speaker) of the utterance
[0023] In this case, the model input unit 1211 extracts the linguistic information of the speaker in that utterance section by transcribing as text the speaker's voice in that utterance section acquired by the voice information acquisition unit 112 such as a microphone. The model input unit 1211 extracts the speaker's voice information and the listener's voice information by converting the speaker's voice and the listener's voice in that utterance section acquired by the voice information acquisition unit 112 such as a microphone into the speaker's voice signal and the listener's voice signal in that utterance section.
[0024] The model input unit 1211 extracts image information of the speaker and the receiver by extracting video of the speaker's face and video of the receiver's face during the utterance section from the video of the speaker and the receiver during the utterance section acquired by the image information acquisition unit 113, such as a camera. The model input unit 1211 extracts position information of the speaker by assigning a variable related to the relative position or order to each utterance in the dialogue, such as 0 for a certain utterance, 1 for the utterance in the immediately preceding history, and 2 for the utterance in the history immediately before that. The model input unit 1211 extracts attribute information of the speaker by assigning a binary value indicating whether the speaker is male or female when analyzing a dialogue between a man and a woman, or a binary value indicating whether the speaker is a parent or a child when analyzing a dialogue between a parent and a child. For example, when analyzing a dialogue between men, the model input unit 1211 extracts attribute information of the speaker by assigning a variable indicating 0 for male and 1 for female to each utterance.
[0025] [Encoding Unit] Returning to the description of FIG. 2 , the encoding unit 1212 encodes the input information extracted by the model input unit 1211 into a vector. In the example shown in (2) of FIG. 1 , the encoding unit 1212 encodes the speaker's linguistic information related to the transcription of the speaker's speech in that utterance section using the Japanese version of BERT (Bidirectional Encoder Representations from Transformers). In this case, the encoding unit 1212 converts the speaker's linguistic information related to the transcription of the speaker's speech in that utterance section into a 768-dimensional vector for the Classification (CLS) token, and then converts it into a 200-dimensional speaker text vector through a fully connected layer with an activation function of ReLU (Rectified Linear Unit).
[0026] The encoding unit 1212 converts speech information related to the speaker's speech signal in the utterance interval into a 512-dimensional vector by using an x-vector, which is a pre-trained model for speaker recognition trained from VoxCeleb1 and VoxCeleb2. The encoding unit 1212 then converts the 512-dimensional vector into a 200-dimensional speaker speech vector through a fully connected layer. The encoding unit 1212 performs the same processing on the speaker's speech signal for the listener's speech information related to the speech signals of non-speaking participants in the utterance interval, which corresponds to the time interval of the same utterance.
[0027] The encoding unit 1212 uses ResNet-18, a model for object recognition with 1000 classes trained on ImageNet, to convert the speaker's image information relating to the image of the speaker's face during the utterance into a 512-dimensional vector for each frame. The encoding unit 1212 then averages multiple frames within the utterance to create a representative vector. The encoding unit 1212 then converts the image information into a 200-dimensional speaker image vector through a fully connected layer. The encoding unit 1212 performs the same processing as the listener's image signal on the listener's image information relating to the image of the face of a participant who is not the speaker during the utterance, which corresponds to the time period of the utterance.
[0028] The encoding unit 1212 converts variables that are utterance position information related to the position of the utterance in a dialogue, which are assigned to each utterance, into a 200-dimensional utterance position encoding. The method for converting variables that are position information into a 200-dimensional utterance position encoding is the same as that in the following reference, so a detailed description of the process will be omitted. Reference: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS'17). Curran Associates Inc., Red Hook, NY, USA, 6000-6010
[0029] The encoding unit 1212 converts speaker attribute information relating to a binary value indicating whether or not a speaker is assigned to each utterance into 200-dimensional speaker segment embedding using an embedding layer.
[0030] [Synthesis Unit] Returning to the description of FIG. 2, the synthesis unit 1213 synthesizes the vectors encoded by the encoding unit 1212 for each interlocutor. In the example shown in (3)-A of FIG. 1, the synthesis unit 1213 synthesizes vectors for each speaker. The synthesis unit 1213 sums the values of the speaker's text vector, the speaker's voice vector, the speaker's video vector, the positional encoding vector (positional encoding) of the utterance, and the speaker's segment embedding to obtain an input embedding for the speaker. Next, the synthesis unit 1213 simultaneously supplies the sum of the summed values to the Transformer encoder speaker, which is a transformer block that receives the input embedding for the speaker, as an input embedding for multiple consecutive utterances during the dialogue for the speaker. In this way, the synthesis unit 1213 encodes the input token sequence of the input embedding using multi-layer self-attention, allowing the synthesis unit 1213 to capture the interrelationships between the input token sequences.
[0031] Similarly, in the example shown in (3)-B of FIG. 1, the integrator 1213 integrates vectors for the listener. Specifically, the integrator 1213 sums the listener's voice vector value, the listener's video vector value, the utterance position encoding vector value, and the speaker's segment embedding value to obtain an input embedding for the listener. The integrator 1213 then simultaneously supplies the sum of the summed values to the Transformer encoder addressee, a transformer block that receives the input embedding for the listener, as an input embedding for multiple consecutive utterances during a dialogue for the listener. By providing the input embedding for the listener to the Transformer encoder addressee, the integrator 1213 enables the input token sequence of the input embedding to be encoded using multi-layer self-attention, thereby capturing interrelationships between the input token sequences.
[0032] [Fusion Unit] The fusion unit 1214 fuses the vectors integrated for each interlocutor by the integration unit 1213. In the example shown in Fig. 1, the fusion unit 1214 functions as a fusion layer: A that fuses, by a concatenation operation or the like, the vectors integrated for each interlocutor by the integration unit 1213 and output from the two Transformer blocks input as input embeddings. The fusion unit 1214 adjusts the model parameters based on the fused vectors.
[0033] [Individual Characteristic Estimation Unit] Returning to the description of FIG. 2 , the individual characteristic estimation unit 122 estimates the individual characteristics of each of the multiple interlocutors based on the dialogue information acquired by the acquisition unit 11. For example, the individual characteristic estimation unit 122 estimates the degree of the individual characteristics of each of the multiple interlocutors. The individual characteristic estimation unit 122 may estimate the individual characteristics based on a trained model 141 that receives dialogue information as input and outputs individual characteristic information related to the individual characteristics. This allows the individual characteristic estimation unit 122 to estimate the individual characteristics with higher accuracy. The individual characteristic estimation unit 122 may estimate the individual characteristics based on a trained model 141 that receives dialogue information acquired by the acquisition unit 11, converts each piece of information included in the dialogue information into a vector, integrates the vector for each interlocutor, and outputs personal characteristic information. This allows the individual characteristic estimation unit 122 to estimate the individual characteristics with higher accuracy.
[0034] An example of estimation of personal characteristics by the personal characteristic estimation unit 122 will be described below with reference to FIG. 1 and FIGS. 3 to 13. FIGS. 3 to 12 are diagrams for explaining an example of personal characteristics. FIG. 13 is a diagram showing an example of estimation results of personal characteristics. In the example shown in FIG. 1, the personal characteristic estimation unit 122 estimates multiple personal characteristics for each of multiple interlocutors, namely, male and female, based on the trained model 141. The personal characteristic estimation unit 122 estimates the degree of the following 32 types of personal characteristics for males based on the personal characteristic information output from the trained model 141. Similarly, the personal characteristic estimation unit 122 estimates the degree of the following 32 types of personal characteristics for females based on the personal characteristic information output from the trained model 141. The following 32 types of personal characteristics are outlined in FIGS. 3 to 12, and therefore detailed description will be omitted.1. Big Five Scale (Wada, 1996) 2. Immersion Scale (Sakamoto, 1997) 3. LETS 2 (Matsui, Kidzuku, Tateuki, Okubo-Omae, Okamura, & Yoneda, 1990) 4. Leisure (Sense) Scale (Furukawa, Yamashita, & Yagi, 1993) 5. Communication Skills Scale ENDCOREs (Fujimoto & Daibo, 2007) 6. Gender Identity Scale (Dohi, 1996) 7. Self-Monitoring Scale (Iwabuchi, Tanaka-Nakazato, 1982) 8. Privacy Orientation Scale (Yoshida & Mizokami, 1996) 9. Subjective Well-Being Scale (Ito et al., 2003) 10. Value Orientation Scale (Sakai, Yamaguchi, & Kuno, 1998) 11. Goal Orientation in Friendship Situations Scale 12. 1. Friendship Scale (Okada, 1995) 2. Friendship Measurement Scale (Yoshioka, 2001) 3. Multidimensional Empathy Scale (Tobari, 2003) 4. Loneliness Scale (Ochiai, 1983) 5. Family Functioning Scale (Kusada & Okado, 1993) 6. Interpersonal Trust Scale (Horii & Tsuchitani, 1995) 7. Egalitarian Gender Role Attitudes Scale (Short Version) (Suzuki, 1987; 1991; 1994) 8. Attitudes Towards Romantic Relationships Scale (Tetsuki, 1994) 9. Romantic Image Scale (Kanemasa, 2000) 10. Daily Living Skills Scale (University Student Version) (Shimamoto & Ishii, 2CD6) 11. Japanese Version of the Self-Concealment Scale (Kono, 2000) 12. KiSS-18 (Kikuchi, 1988) 13. 25. Trait Shyness Scale (Aikawa, 1991) 26. Situational Interpersonal Anxiety Scale (Mori and Tanno, 2001) 26. Purpose in Life Scale (Kondo and Kamata, 1998) 27. Way of Life Scale (Itatsu, 1992) 28. Self-Esteem Scale (Yamamoto, Matsui, and Yamanari, 1982) 29. Clothing Interest Questionnaire (Kamiyama, 1983) 30. Affiliation Motivation Scale (Sugiura, 2000) 31. Attitudes Towards Divorce Scale (Odagiri, 2003) 32. Self-Consciousness Scale (Sugawara, 1984).
[0035] An example of estimation of personal characteristics by the personal characteristic estimation unit 122 will be described below with reference to FIG. 13 . FIG. 13 is a diagram showing an example of estimation results of personal characteristics. For the "Big Five scale," the personal characteristic estimation unit 122 recursively estimates estimated values for each of the five factors defined in this scale by using a fully connected layer whose activation function is an identity function. For the "Big Five scale," the personal characteristic estimation unit 122 obtains a score between 1 and 7 for the five factors (extraversion, agreeableness, conscientiousness, neuroticism, and openness) for each of multiple male and female interlocutors using a questionnaire such as the TIPI-J questionnaire.
[0036] Next, in the examples shown in the diagram of personal characteristics of men and the diagram of personal characteristics of women in Figure 13, the personal characteristic estimation unit 122 divides the values of each of the five factors by 7 so that they become real numbers between 0 and 1 that can be easily processed by the trained model 141, and then rounds them off to one decimal place to estimate the estimated values of each of the five factors.
[0037] In the above example, the personal characteristic estimation unit 122 estimates the estimated value of each factor as a real number between 0 and 1. However, any value may be used as the estimated value depending on the target of estimation, as long as the degree of the personal characteristic can be estimated. For example, the personal characteristic estimation unit 122 may estimate a real number between −1 and 1, or any real number between negative infinity and positive infinity, etc., as the estimated value. Furthermore, while the personal characteristic estimation unit 122 estimates 32 types of male personal characteristics and female personal characteristics, the number of types may be more or less than 32 and is not limited to the above-mentioned personal characteristics. Furthermore, in the example shown in FIG. 1 , the female personal characteristic estimation unit for estimating female personal characteristics in the personal characteristic estimation unit 122 is depicted as a simplified diagram, but the female personal characteristic estimation unit may be implemented using a structure similar to that of the male personal characteristic estimation unit for estimating male personal characteristics.
[0038] [Relationship Estimation Unit] Returning to the description of FIG. 2 , the relationship estimation unit 123 estimates the relationship between multiple interlocutors based on the personal characteristics estimated by the personal characteristic estimation unit 122. This allows the relationship estimation unit 123 to estimate the relationship between the interlocutors with high accuracy, such as hierarchical estimation of the relationship between the interlocutors, which has not been possible with conventional technology. The relationship estimation unit 123 may estimate the relationship based on the dialogue information acquired by the acquisition unit. This allows the relationship estimation unit 123 to estimate the relationship with higher accuracy. The relationship estimation unit 123 may estimate the degree of the relationship for at least one of a romantic relationship, a friendship relationship, and a family relationship based on the degree of the personal characteristics of each of the multiple interlocutors estimated by the personal characteristic estimation unit 122. By performing estimation based on the degree of the personal characteristics, the relationship estimation unit 123 can estimate the degree of the relationship for at least one of a romantic relationship, a friendship relationship, and a family relationship with high accuracy.
[0039] The relationship estimation unit 123 may further estimate the possibility of the relationship progressing. This allows the relationship estimation unit 123 to recommend smooth human relationships such as romantic relationships and friendships. The relationship estimation unit 123 may estimate at least one of the relationship, the degree of the relationship, and the possibility of the relationship progressing based on the trained model 141 that receives dialogue information as input and outputs personal characteristic information and relationship information. For example, the relationship estimation unit 123 estimates the possibility of the relationship progressing in a romantic relationship based on the trained model 141 that has been trained with learning data that includes output information such as "whether the man will give his contact information to the woman" and "whether the woman will give her contact information to the man," and input information such as dialogue information. This allows the relationship estimation unit 123 to more accurately estimate the relationship, the degree of the relationship, and the possibility of the relationship progressing.
[0040] An example of the relationship estimation by the relationship estimation unit 123 will be described below with reference to FIGS. 1, 13, and 14. FIG. 14 is a diagram showing an example of the relationship estimation result. In the example shown in FIG. 1, the relationship estimation unit 123 estimates the degree of the romantic relationship based on the degree of the man's personal characteristics and the degree of the woman's personal characteristics estimated by the personal characteristic estimation unit 122 and the output information from fusion layer: A. Specifically, the relationship estimation unit 123 fuses the degree of the man's personal characteristics, which is the estimation result of the man's personal characteristics, and the degree of the woman's personal characteristics, which is the estimation result of the woman's personal characteristics, with the output information from fusion layer: A by a linking operation or the like using fusion layer: B or the like, and estimates the degree of liking and interest that each man and woman have in each other as a predicted value of the relationship between the man and woman.
[0041] In the examples shown in the diagram of relationships between men and women and the diagram of relationships between women and men in FIG. 14 , the relationship estimation unit 123, like the personal characteristic estimation unit 122, estimates predicted values of men's liking for women and women's liking for men so that the values are real numbers between 0 and 1 with one decimal point. Here, men tend to have lower liking for women who are highly conscientious, and in the example shown in the diagram of women's personal characteristics in FIG. 13 , the estimated value for conscientiousness is high among the factors of the "Big Five scale" of women's personal characteristics. Therefore, the relationship estimation unit 123 estimates men's liking for women as low predicted values, as shown in the diagram of relationships between men and women in FIG. 14 . Furthermore, women tend to like men who are highly extroverted, and in the example shown in the diagram of men's personal characteristics in Figure 13, the estimated value of extroversion is high among the factors of the "Big Five scale" of men's personal characteristics, so the relationship estimation unit 123 estimates women's liking for men as a high predicted value, as shown in the diagram of women's relationships with men in Figure 14.
[0042] The relationship estimation unit 123 further predicts whether a romantic relationship will be established (estimating the degree of the romantic relationship and the possibility of the relationship progressing) based on the fused fusion information. Specifically, the relationship estimation unit 123 predicts whether the man will provide his contact information to the woman and whether the woman will provide her contact information to the man, based on the fusion information, using two blocks: a block predicting whether the man will provide his contact information to the woman and a block predicting whether the woman will provide her contact information to the man. In this case, the relationship estimation unit 123 may estimate the possibility of the romantic relationship progressing based on the trained model 141 trained with training data including relationship information regarding the possibility of the romantic relationship progressing, such as output information "whether the man will provide his contact information to the woman" and "whether the woman will provide her contact information to the man," as well as personal characteristic information and input information, such as dialogue information.
[0043] In the examples shown in the diagram of relationships between men and women and the diagram of relationships between women and men in Fig. 14 , the relationship estimation unit 123 estimates the predicted value of whether a man will give his contact information to a woman and the predicted value of whether a woman will give her contact information to a man as real numbers between 0 and 1, corresponding to the probability of giving contact information, so that the values are easily processed by the trained model 141. Here, men tend to have low liking for women who are highly conscientious, and in the example shown in the diagram of women's personal characteristics in Fig. 13 , the estimated value of conscientiousness among the factors of the "Big Five scale" of women's personal characteristics is high, so the relationship estimation unit 123 estimates the probability of whether a man will give his contact information to a woman as a low predicted value, as shown in the diagram of relationships between men and women in Fig. 14 . Women tend to like men who are highly extroverted, and in the example shown in the diagram of men's personal characteristics in Figure 13, the estimated value for extroversion among the factors of the "Big Five scale" of men's personal characteristics is high, so the relationship estimation unit 123 estimates the probability of whether a woman will give her contact information to a man as a high predicted value, as shown in the diagram of relationships between women and men in Figure 14.
[0044] In the above example, the relationship between one factor of the "Big Five scale" for men and one factor of the "Big Five scale" for women and the predicted value of the relationship has been mainly described, but the estimation of the predicted value of the relationship by the relationship estimation unit 123 is not limited to the above example. The relationship estimation unit 123 may estimate the predicted value of the relationship based on at least some of the other factors of the "Big Five scale" for men and the other 31 factors of the individual characteristics of men, as well as the other factors of the "Big Five scale" for women and the other 31 factors of the individual characteristics of women, and may estimate the predicted value of the relationship by comprehensively referring to these. This allows the relationship estimation unit 123 to estimate the predicted value of the relationship with higher accuracy based on more factors of the individual characteristics.
[0045] In the above example, the relationship estimation unit 123 estimates the predicted value of each factor of the relationship as a real number between 0 and 1. However, as long as the degree of the relationship can be estimated, any value may be estimated as the predicted value depending on the target of prediction. For example, the relationship estimation unit 123 may estimate a real number between −1 and 1, or any real number between negative infinity and positive infinity, as the predicted value. Regarding whether or not to provide contact information, the relationship estimation unit 123 may estimate a relationship based on the estimated personal characteristics after the personal characteristics estimation unit 122 estimates the personal characteristics. Alternatively, the relationship estimation unit 123 may estimate the relationship in parallel with the estimation of the personal characteristics by the personal characteristic estimation unit 122. That is, the personal characteristic estimation unit 122 and the relationship estimation unit 123 may estimate the personal characteristics and the relationship in two stages, or may estimate them together in one stage. When estimating the personal characteristics and the relationship together in one stage, the relationship estimation unit 123 estimates the relationship based on the trained model 141 that receives dialogue information as input and outputs personal characteristic information and relationship information. In this case as well, the relationship estimation unit 123 can estimate the relationship based on the personal characteristics from the correlation between the personal characteristic information and the relationship information.
[0046] [Presenter] Returning to the description of Fig. 2, the presenter 13 presents the individual characteristics estimated by the individual characteristic estimation unit 122 and the relationship estimated by the relationship estimation unit 123. For example, the presenter 13 presents estimated values of 32 types of individual characteristics of men and estimated values of 32 types of individual characteristics of women estimated by the individual characteristic estimation unit 122. The presenter 13 also presents predicted values of the relationship between men and women in a romantic relationship estimated by the relationship estimation unit 123. Examples of the presenter 13 include a display unit and an audio output unit.
[0047] [Storage Unit] The storage unit 14 stores various types of information such as dialogue information, personal characteristic information, personal characteristics and relationships, the trained model 141, an operating system (OS) and various programs executed by the estimation device 10. Examples of the storage unit 14 include storage devices such as a hard disk drive (HDD), a solid state drive (SSD), and an optical disk, as well as rewritable semiconductor memories such as random access memory (RAM), flash memory, and non-volatile static random access memory (NVSRAM).
[0048] [Example of Processing Procedure] An example of processing procedure executed by the estimation device 10 will be described below with reference to Fig. 15. Fig. 15 is a flowchart showing an example of processing procedure executed by the estimation device. In step S1, the acquisition unit 11 acquires dialogue information related to a dialogue between multiple interlocutors. For example, the acquisition unit 11 acquires the dialogue information during a dialogue between the multiple interlocutors.
[0049] In step S2, the individual characteristic estimation unit 122 estimates the individual characteristics of each of the multiple interlocutors based on the dialogue information acquired by the acquisition unit 11. For example, the individual characteristic estimation unit 122 estimates each of 32 types of male individual characteristics and each of 32 types of female individual characteristics.
[0050] In step S3, the relationship estimation unit 123 estimates relationships between multiple interlocutors based on the personal characteristics estimated by the personal characteristic estimation unit 122. For example, the relationship estimation unit 123 estimates a predicted value of a romantic relationship between a man and a woman.
[0051] [Modification] In the above example, the estimation device 10 acquires dialogue information regarding a dialogue between two interlocutors, estimates personal characteristics of each of the two interlocutors based on the dialogue information, and estimates the relationship between the two interlocutors based on the personal characteristics. However, this embodiment is not limited to this. In this embodiment, the estimation device 10 may acquire dialogue information regarding three or more interlocutors if there are multiple interlocutors, estimate personal characteristics of each of the three or more interlocutors based on the dialogue information, and estimate the relationship between the three or more interlocutors based on the personal characteristics.
[0052] Furthermore, the components of each unit shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of the device is not limited to that shown in the figure, and all or part of it can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by the device can be realized by a CPU and a program executed by the CPU, or can be realized as hardware using wired logic.
[0053] Furthermore, among the above processes, all or part of the processes described as being performed automatically can be performed manually, and all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the processing procedures, specific names, various information, parameters, etc. shown in the above documents and drawings can be changed as desired unless otherwise specified.
[0054] [Program] The estimation device 10 can be implemented by installing an estimation program as package software or online software on a computer. For example, the computer can function as the estimation device 10 by executing the estimation program.
[0055] Fig. 16 is a diagram showing an example of a computer that executes an estimation program. In the example shown in Fig. 16, a computer 1000 has a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.
[0056] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. For example, a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.
[0057] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process executed by the estimation device 10 is implemented as a program module 1093 in which computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing processes similar to those of the functional configuration of the estimation device 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced with an SSD.
[0058] Data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. The CPU 1020 then reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary and executes them.
[0059] The program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.
[0060] REFERENCE SIGNS LIST 10 Estimation device 11 Acquisition unit 12 Control unit 13 Presentation unit 14 Memory unit 111 Language information acquisition unit 112 Speech information acquisition unit 113 Image information acquisition unit 114 Position information acquisition unit 115 Attribute information acquisition unit 121 Model learning unit 122 Personal characteristic estimation unit 123 Relationship estimation unit 141 Trained model 1211 Model input unit 1212 Encoding unit 1213 Integration unit 1214 Fusion unit
Claims
1. An estimation device comprising: an acquisition unit that acquires dialogue information regarding a dialogue between a plurality of interlocutors; a personal characteristic estimation unit that estimates personal characteristics of each of the plurality of interlocutors based on the dialogue information acquired by the acquisition unit; and a relationship estimation unit that estimates a relationship between the plurality of interlocutors based on the personal characteristics estimated by the personal characteristic estimation unit.
2. The estimation device according to claim 1, characterized in that the personal characteristic estimation unit estimates the personal characteristics based on a trained model that receives the dialogue information as input and outputs personal characteristic information related to the personal characteristics.
3. The estimation device described in claim 2, characterized in that the acquisition unit acquires the dialogue information including at least one of linguistic information and voice information regarding the speech of each of the multiple interlocutors, image information regarding an image of each of the multiple interlocutors, position information regarding the position of the utterance, and attribute information regarding an attribute of each of the multiple interlocutors.
4. The estimation device described in claim 3, characterized in that the personal characteristic estimation unit estimates the personal characteristics based on the trained model, in which each piece of information included in the dialogue information acquired by the acquisition unit is converted into a vector, and the vector is integrated for each interlocutor, and the dialogue information is used as input, and the personal characteristic information is output.
5. The estimation device according to any one of claims 1 to 4, characterized in that the personal characteristic estimation unit estimates the degree of the personal characteristic of each of the multiple interlocutors, and the relationship estimation unit estimates the degree of the relationship for at least one of a romantic relationship, a friendship relationship, and a family relationship based on the degree of the personal characteristic of each of the multiple interlocutors estimated by the personal characteristic estimation unit.
6. The estimation device according to claim 1, characterized in that the relationship estimation unit further estimates the possibility of development of the relationship.
7. An estimation method executed by an estimation device, comprising: an acquisition step of acquiring dialogue information regarding a dialogue between a plurality of interlocutors; a personal characteristic estimation step of estimating personal characteristics of each of the plurality of interlocutors based on the dialogue information acquired by the acquisition step; and a relationship estimation step of estimating a relationship between the plurality of interlocutors based on the personal characteristics estimated by the personal characteristic estimation step.
8. An estimation program that causes a computer to execute the following steps: an acquisition step of acquiring dialogue information regarding a dialogue between a plurality of interlocutors; a personal characteristic estimation step of estimating personal characteristics of each of the plurality of interlocutors based on the dialogue information acquired by the acquisition step; and a relationship estimation step of estimating a relationship between the plurality of interlocutors based on the personal characteristics estimated by the personal characteristic estimation step.
Citation Information
Patent Citations
Multi-person dialogue emotion recognition method
CN115329779A
Expert-driven, technology-facilitated intervention system for improving interpersonal relationships
US20220327954A1
