Profile information collection system, profile information collection method, and program

Abstract

To effectively collect profile information of a person while supporting a conversation. [Solution] The profile information collection system comprises a user terminal and a support device that communicates with the user terminal, wherein the support device comprises a data acquisition unit that acquires observation data from the user terminal regarding a subject having a conversation with a user of the user terminal, a data analysis unit that analyzes the observation data, a profile update unit that estimates and updates profile information of the subject based on the analysis results of the observation data, and a support information generation unit that generates support information for the user to support a conversation with the subject based on the profile information of the subject, and the user terminal comprises an output control unit that outputs the support information from the support device to the user.

Description

[Technical field]

[0001] The present invention relates to a profile information collecting system, a profile information collecting method, and a program. [Background technology]

[0002] Many social activities consist of conversations between oneself and another person (the conversation partner). Especially in business, if you can accurately grasp the other person's profile, such as their situation, personality, and feelings, you can have a high-quality conversation that can provide useful information for business purposes.

[0003] However, since communication ability and experience, such as conversation, vary from person to person, people who lack communication ability and experience do not know how to converse in a way that will allow them to accurately grasp the other person's profile, making it difficult for them to accurately gather information about the other person's profile.

[0004] Regarding technology for collecting information from conversations, for example, Patent Document 1 describes an office robot that includes a conversing person communication information collection unit (303, etc.) that approaches a person while recognizing the person who is speaking using a person recognition unit 309, and after personally authenticating the person who is speaking using a personal authentication unit 306, detects and collects communication information of the person who is speaking using a communication information detection unit (310, etc.), and a communication information log recording unit (303, etc.) that records the communication information of the people who are speaking in a communication database by associating it with the person who sent the communication information, the person with whom the person is speaking, and the communication information. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2010-267143 A Summary of the Invention [Problem to be solved by the invention]

[0006] According to the technology described in Patent Document 1, it is possible to understand the relationship between the people in conversation and the content of the conversation. However, it is not possible to suggest what kind of conversation should be held in order to know the other person's profile.

[0007] The present invention has been made in view of the above circumstances, and has an object to make it possible to effectively collect profile information of the other party while supporting the conversation. [Means for solving the problem]

[0008] The present application includes a number of means for solving at least part of the above problems, examples of which are as follows.

[0009] In order to solve the above-mentioned problems, a profile information collection system according to one embodiment of the present invention is a profile information collection system comprising a user terminal and a support device communicating with the user terminal, wherein the support device comprises: a data acquisition unit that acquires observation data relating to a subject having a conversation with a user of the user terminal from the user terminal, a data analysis unit that analyzes the observation data, a profile update unit that estimates and updates profile information of the subject based on a result of the analysis of the observation data, and a support information generation unit that generates support information for the user to support a conversation with the subject based on the profile information of the subject, and the user terminal comprises an output control unit that outputs the support information from the support device to the user.

[0010] The observation data can include audio data of a conversation between the user and the subject, the data analysis unit can analyze the audio data to identify the content of the conversation, and the profile update unit can estimate the profile information of the subject based on the content of the conversation.

[0011] The profile update unit can estimate at least one of the target person's age, gender, place of origin, career history, recent situation, state of mind, personality, financial sense, career aptitude test results, and IQ test results as the profile information based on the conversation content.

[0012] The support information generating unit can generate, as the support information, at least one of a next topic, a question, and a prohibited matter, based on the profile information of the subject.

[0013] The data acquisition unit can acquire environmental data of the subject from the user terminal, and the support information generation unit can generate the support information based on the profile information of the subject and the environmental data.

[0014] The data analysis unit can analyze the voice data to estimate the subject's emotion, and the support information generation unit can generate the support information based on the subject's profile information and the emotion.

[0015] The observation data may include video footage of the user, and the support information generation unit may determine, as the support information, a facial expression that the user should make based on at least one of the content of the conversation, the video footage of the user, and the emotion of the subject.

[0016] The output control unit can display a support screen based on the support information from the support device.

[0017] The support information from the support device can include an emotion of the subject, and the output control unit can display a support screen that visualizes the emotion of the subject.

[0018] The output control unit can deform an image of the subject and display it on the support screen.

[0019] The support information generation unit can generate the support information including advice information from an assistant based on the observation data of the subject.

[0020] The profile update unit can estimate profile information of the user based on the content of the conversation, and the support information generation unit can generate the support information for the user based on the profile information of the target person and the profile information of the user.

[0021] The support information generation unit can generate the user's action as the support information, and the profile update unit can estimate the profile information of the subject based on the subject's reaction to the user's action.

[0022] The assistance device may include an evaluation unit that evaluates the behavior of the user who has had a conversation with the subject, and the output control unit may output to the user a result of the evaluation of the user by the evaluation unit.

[0023] The assistance device may include a simulation unit that executes a simulation of a conversation with the target person for the user before the user has a conversation with the target person.

[0024] The support information generation unit can confirm the estimated profile information of the subject to the user.

[0025] The profile update unit can integrate multiple pieces of profile information about the same subject that are generated in association with multiple different users.

[0026] A profile information collection method according to another aspect of the present invention is a profile information collection method by a profile information collection system including a user terminal and an assistance device communicating with the user terminal, the method including a data acquisition step of the assistance device acquiring from the user terminal observation data relating to a subject having a conversation with a user of the user terminal, a data analysis step of analyzing the observation data, a profile update step of estimating and updating profile information of the subject based on a result of the analysis of the observation data, a support information generation step of generating support information for the user to assist in a conversation with the subject based on the profile information of the subject, and an output control step of the user terminal outputting the support information from the assistance device to the user.

[0027] A program according to yet another aspect of the present invention causes a computer as an assistance device communicating with a user terminal to execute processing including a data acquisition step of acquiring, from the user terminal, observation data relating to a subject having a conversation with a user of the user terminal, a data analysis step of analyzing the observation data, a profile update step of estimating and updating profile information of the subject based on results of the analysis of the observation data, and an assistance information generation step of generating assistance information for the user to assist in a conversation with the subject, based on the profile information of the subject.

[0028] A program according to yet another aspect of the present invention causes a computer to execute processes including a data acquisition step of acquiring observation data regarding a subject having a conversation with a user, a data analysis step of analyzing the observation data, a profile update step of estimating and updating profile information of the subject based on results of the analysis of the observation data, a support information generation step of generating support information for the user based on the profile information of the subject to assist in a conversation with the subject, and an output control step of outputting the support information to the user. Effect of the Invention

[0029] According to the present invention, it is possible to effectively collect profile information of the other party while supporting the conversation.

[0030] Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0031] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a profile information collecting system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of a user terminal and an assistance device. [Diagram 3] FIG. 3 is a diagram illustrating an example of profile information. [Figure 4] FIG. 4 is a sequence diagram illustrating an example of a profile information collection process. [Diagram 5] FIG. 5 is a diagram showing a display example of the support screen. [Figure 6] FIG. 6 is a diagram showing an application example of the profile information collection system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In all the drawings for explaining the embodiment, the same members are generally given the same reference numerals, and the repeated description thereof will be omitted. In the following embodiments, the components (including element steps, etc.) are not necessarily essential, except when specifically stated otherwise or when it is clearly considered essential in principle. In addition, when it is said that "A is made of," "A is made of," "has A," or "includes A," other elements are not excluded, except when it is specifically stated that only that element is included. Similarly, in the following embodiments, when referring to the shape, positional relationship, etc. of components, etc., it is intended to include those that are substantially similar to or similar to the shape, etc., except when it is specifically stated otherwise or when it is clearly considered not to be the case in principle.

[0033] <Profile information collection system 10 according to one embodiment of the present invention> FIG. 1 shows an overview of a profile information collection system 10 according to one embodiment of the present invention.

[0034] The profile information collection system 10 has a profile information collection function, a conversation support function, a user education function, a pre-simulation function, a person reconfirmation function, and a profile information integration function.

[0035] The profile information collection function is a function for estimating, registering, or updating the profile of a subject who has a conversation with a user. According to the profile information collection function, the profile information of the subject can be collected from the contents of the conversation with the subject. The profile information collection function is realized by the data acquisition unit 311, the data analysis unit 312, and the profile update unit 313 of the support device 30.

[0036] The conversation support function is a function that supports a user who is having a conversation with a target by determining support information based on the target's profile information, such as a conversation topic, questions to the target, prohibited matters that should not be said to the target, and facial expressions that the user should make, and outputting the support information to the user. The conversation support function enables a smooth conversation in which the target's profile can be estimated. The conversation support function is realized by the support information generation unit 314 of the support device 30.

[0037] The user education function is a function that analyzes a conversation log that records a conversation between the user and a target person, determines areas for improvement, and outputs the results to the user, thereby educating the user on the areas for improvement, etc. The user education function allows the user to receive instruction that is easier to accept than when the user receives instruction from a person. Furthermore, it saves the effort of a person to instruct and educate the user. The user education function is realized by the evaluation unit 315 of the support device 30.

[0038] The advance simulation function is a function that allows a user to simulate a conversation with a target person with whom a conversation is scheduled to be held in the future, for example, by using an avatar or a digital clone of the target person based on profile information obtained in advance of the target person. The advance simulation function allows a user to practice a conversation before actually having a conversation with the target person. The advance simulation function is realized by the simulation unit 316 of the support device 30.

[0039] The person reconfirmation function is a function that, after a conversation, asks the user what kind of person the target person was, compares the answer with the profile information obtained from the conversation, and adds the user's impression of the target person to the profile information. The person reconfirmation function can improve the accuracy of the target's profile, including the impression. It also makes it possible to measure the user's communication ability, which can be used for future education and evaluation. The person reconfirmation function is realized by the support information generation unit 314 and the profile update unit 313 of the support device 30.

[0040] The profile information integration function is a function that integrates multiple profile information related to the same subject obtained by multiple users having conversations with the subject. The profile information integration function makes it possible to obtain more in-depth and highly accurate profile information such as the personality of the survey subject. The profile information integration function is realized by the profile update unit 313 of the support device 30.

[0041] It is assumed that the conversation between the user and the target person is one-to-one, but it may be one-to-many, many-to-one, or many-to-many. In any case, however, each user uses a user terminal 20. The utterances of each target person must be collected individually, and each target person must be photographed individually.

[0042] The conversation between the user and the target person may be conducted face-to-face, or may be conducted over a network using a web conferencing program such as Zoom (trademark), Microsoft Teams (trademark), etc. The conversation is not limited to one using voice, but may be text-based using SNS (Social Networking Service), email, messages, etc., or may use actions such as sign language, body language, and gestures.

[0043] The profile information collection system 10 includes a user terminal 20, a support device 30, and an assistant terminal 40. However, the assistant terminal 40 is not essential.

[0044] The user terminal 20 used by the user acquires conversation data (audio data, text data, etc.) between the user and a target person, and video data of the target person during the conversation, and transmits the conversation data and video data as observation data to the support device 30. The user terminal 20 also acquires environmental data of the target person during the conversation (location (indoors / outdoors, workplace, tourist spot, dangerous place, etc.), temperature, humidity, vibration, etc.) and transmits it to the support device 30. Furthermore, the user terminal 20 outputs the support information transmitted from the support device 30 to the user (displaying it on a screen, etc.).

[0045] The support device 30 is composed of a general computer such as a personal computer or a server computer. The support device 30 identifies the conversation content from the conversation data included in the observation data transmitted from the user terminal 20, and estimates the target person's profile information (age, sex, place of origin, career (educational history, work history), recent situation, state of mind, personality, etc.) based on the conversation content. The support device 30 also estimates the target person's emotions (joy, anger, sadness, happiness, etc.) based on the conversation content, and the volume, tone, intonation, etc. of the conversation. Furthermore, the support device 30 determines the next topic, questions, prohibited matters, etc. based on the target person's profile information, environmental data, and emotions, and transmits them to the user terminal 20 as support information. Furthermore, the support device 30 transmits the observation data and environmental data transmitted from the user terminal 20 to the assistant terminal 40.

[0046] The assistant terminal 40 is used by an assistant (assistant) who has skills such as counseling and is prepared by the management side of the profile information collection system 10. The assistant may be someone else such as the user's boss or colleague. The assistant terminal 40 is composed of a computer such as a smartphone, a tablet PC (personal computer), a notebook PC, or a desktop PC. The assistant terminal 40 plays conversation data and video data included in the observation data. Then, the assistant terminal 40 transmits advice information for the user, which is input by the assistant who has confirmed the conversation between the user and the target person and the environmental data, to the support device 30. Then, the support device 30 generates support information based on the advice information for the user from the assistant and transmits it to the user terminal 20. By providing the assistant terminal 40, support information with a different perspective and content from that of the support device 30 can be output to the user.

[0047] FIG. 2 shows an example of the configuration of the user terminal 20 and the support device 30. As shown in FIG.

[0048] The user terminal 20 includes a processing unit 21, a storage unit 22, a communication unit 23, a microphone 24, cameras 25A and 25B, a sensor 26, a speaker 27, and a display 28. The user terminal 20 is composed of a computer such as a smartphone, a tablet PC (personal computer), a notebook PC, or a desktop PC, which includes a processor such as a CPU (Central Processing Unit), a memory such as a DRAM (Dynamic Random Access Memory), a storage such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), an input device such as a keyboard, a mouse, or a touch panel, an output device such as a display, and a communication module such as a NIC (Network Interface Card) (all not shown).

[0049] The processing unit 21 is realized by a computer processor. The processing unit 21 has the following functional blocks: a data acquisition unit 211, a data transfer unit 212, a support information acquisition unit 213, and an output control unit 214. These functional blocks are realized by the computer processor executing a predetermined program loaded into memory. However, some or all of these functional blocks may be realized as hardware using an integrated circuit or the like. Furthermore, these functional blocks may be realized by multiple computers.

[0050] The data acquisition unit 211 acquires profile information of the user and the target person based on input from the user. The data acquisition unit 211 also acquires observation data (conversation data between the user and the target person, and video data of the user and the target person during the conversation). Specifically, when the conversation is conducted face-to-face, the data acquisition unit 211 acquires audio data from the microphone 24 and video data from the cameras 25A and 25B. When the conversation is conducted via a network, the data acquisition unit 211 acquires audio data, text data, and video data from a Web conference program or the like used in the conversation.

[0051] Furthermore, the data acquisition unit 211 acquires environmental data of the target person during the conversation. Specifically, when the conversation is conducted face-to-face, the data acquisition unit 211 acquires location information, temperature, humidity, vibration, etc. from the sensor 26. When the conversation is conducted via a network, the data acquisition unit 211 acquires information on items that can be acquired from location information, temperature, humidity, vibration, etc. from a Web conference program or the like used in the conversation.

[0052] The data transfer unit 212 controls the communication unit 23 to transmit the subject's profile information, observation data, and environmental data acquired by the data acquisition unit 211 to the support device 30.

[0053] The support information acquisition unit 213 acquires the support information transmitted from the support device 30 via the communication unit 23 .

[0054] The output control unit 214 generates a support screen 1000 (FIG. 5) for showing the support information to the user, and displays it on the display 28. Note that instead of or in addition to the support screen 1000, the support information may be notified to the user by sound, vibration, or the like.

[0055] The storage unit 22 is realized by a computer memory and storage. A Web conference program used for conversation via a network is stored in the storage unit 22. The storage unit 22 may temporarily store the observation data and environmental data acquired by the data acquisition unit 211, and the support information from the support device 30. Furthermore, the storage unit 22 may temporarily or continuously store data and information other than the above.

[0056] The communication unit 23 is realized by a communication module of a computer. The communication unit 23 connects to the support device 30 via the network N, transmits observation data and environmental data, and receives support information.

[0057] Microphone 24, cameras 25A and 25B, and sensor 26 are realized by computer input devices. Microphone 24 collects the sound of a conversation between the user and the target person to generate audio data. Camera 25A captures an image of the user to generate video data. Camera 25B captures an image of the target person to generate video data. When there is no need to distinguish between cameras 25A and 25B, they are collectively referred to as camera 25. Sensor 26 includes, for example, a positioning sensor such as a Global Positioning System (GPS), a temperature sensor, a humidity sensor, a vibration sensor, etc.

[0058] The speaker 27 and the display 28 are realized by computer output devices. The speaker 27 outputs voice as support information. The display 28 displays a support screen 1000 (FIG. 5).

[0059] The microphone 24, the camera 25, the sensor 26, the speaker 27, and the display 28 may be built in the computer that realizes the user terminal 20, or may be externally connected to the computer (such as a wearable device). For example, earphones, bone conduction earphones, vibrators, etc. may be used instead of or in addition to the speaker 27, and smart glasses, etc. may be used instead of or in addition to the display 28.

[0060] The network N is a two-way communication network such as the Internet or a mobile phone communication network.

[0061] The support device 30 includes a processing unit 31, a storage unit 32, and a communication unit 33. The support device 30 is formed of a computer such as a server computer including a processor, a memory, a storage, an input device, an output device, and a communication module (none of which are shown).

[0062] The processing unit 31 is realized by a computer processor. The processing unit 31 has the following functional blocks: a data acquisition unit 311, a data analysis unit 312, a profile update unit 313, a support information generation unit 314, an evaluation unit 315, and a simulation unit 316. These functional blocks are realized by the computer processor executing a predetermined program loaded into memory. However, some or all of these functional blocks may be realized as hardware using integrated circuits or the like. Furthermore, these functional blocks may be realized by multiple server computers distributed across a network N.

[0063] The data acquisition unit 311 acquires the profile information, observation data, and environmental data transmitted from the user terminal 20 and received by the communication unit 33 via the network N, outputs the profile information to the profile update unit 313, outputs the observation data and environmental data to the data analysis unit 312, and controls the communication unit 33 to transmit to the assistant terminal 40. The data acquisition unit 311 also acquires the advice information transmitted from the assistant terminal 40 and received by the communication unit 33 via the network N, and outputs it to the support information generation unit 314.

[0064] The data analysis unit 312 analyzes the conversation data included in the observation data by voice recognition processing, language processing, etc. to identify the conversation content (for example, data expressing the conversation in text), and analyzes the posture, gestures, appearance, state, and change in facial expression during the conversation, and reactions to the user's remarks and actions (gestures, hand movements, facial expressions, etc.) by image recognition processing, etc. from the video data. Then, the data analysis unit 312 estimates profile information of the user and the subject (age, sex, career (educational history, work history), recent situation, state of mind, personality, sense of money, career aptitude test results, intelligence test results, etc.) based on the analysis results of the conversation content and the video data, and outputs the information to the profile update unit 313. Specifically, the data analysis unit 312 extracts profile information of the subject from the conversation content, or estimates the profile information of the subject by inputting at least a part of the analysis results of the conversation content and the video data to the profile estimation model 321. Note that an estimation model provided by an external service may be used instead of the profile estimation model 321.

[0065] The data analysis unit 312 also inputs the conversation content, and the volume, tone, intonation, etc., of the conversation into an emotion estimation model 322 to estimate the emotion of the subject and output the estimation result to the support information generation unit 314. Furthermore, the data analysis unit 312 generates a conversation log 325 in which the observation data (conversation data, video data), the conversation content, and the emotion of the subject are associated with a time axis, and stores the conversation log in the storage unit 32.

[0066] The profile update unit 313 registers (records) the profile information of the user and the subject transmitted from the user terminal 20 in the storage unit 32. In addition, the profile update unit 313 updates the profile information 324 already registered in the storage unit 32 with the profile information of the user and the subject estimated by the data analysis unit 312.

[0067] The support information generating unit 314 generates support information such as the next topic, questions, and prohibited matters for the target person by inputting the target person's profile information, environmental data, and emotions into the support information generation model 323. The support information generating unit 314 also generates support information based on advice information transmitted from the assistant terminal 40. The support information generating unit 314 then controls the communication unit 33 to transmit the generated support information to the user terminal 20. The support information generating unit 314 may refer to the target person's profile information, extract items for which information is lacking, and generate questions that directly ask about the items or topics for indirectly asking about the items as support information. When generating support information, the support information may be generated by referring to the user's profile information and taking into account the user's personality, qualities, abilities, and the like. In addition,

[0068] Furthermore, the support information generating unit 314 may suggest to the user a facial expression (e.g., a serious face, a smile, etc.) that the user should make when the facial expression of the user is inappropriate for the content of the conversation, for example, when the user has a blank expression when the target is joking, or when the user is grinning when the target is talking seriously, based on the content of the conversation, the target's emotions, and the user's video data. Furthermore, in addition to facial expressions, the support information generating unit 314 may suggest actions (gestures, hand movements, etc.) for the user.

[0069] Furthermore, after the conversation, the support information generating unit 314 asks the user what kind of person the target was, compares the answer with the profile information of the target estimated from the conversation, and has the user answer the user's impression of the target. If the user's answer is to be respected, the profile updating unit 313 registers the impression of the target answered by the user in the profile information 324, and updates other items using the user's answer. If the analysis result by the data analyzing unit 312 is to be respected, differences from the user's answer are detected and presented to the user, or the user's communication ability is evaluated.

[0070] The evaluation unit 315 evaluates the behavior of the user who has finished a conversation with the target person. Specifically, the conversation log 325 is analyzed, and, for example, the user's behavior (tone of voice, tempo, tone of voice, volume of voice, facial expression, etc.) is identified, and the identified user behavior is input to the evaluation model 326, thereby scoring the user's behavior and identifying areas for improvement (behavior to be corrected, etc.). Note that the user's evaluation may reflect, for example, the transition of the target's emotions (for example, the ratio of unpleasant emotions (anger, sadness) / pleasure (joy, pleasure)). The evaluation result by the evaluation unit 315 is transmitted to the user terminal 20 and output to the user.

[0071] The simulation unit 316 generates a digital clone, which is an interactive AI that reflects the personality and thoughts of the target person using the profile information of the target person obtained before the conversation, and uses the digital clone to simulate a conversation with the target person for the user. The digital clone may be built into the support device 30 or an external service may be used. An avatar representing the target person's appearance may be generated based on a video obtained in advance. In the simulation, the user's remarks and video to the avatar displayed on the user terminal 20 are transmitted to the support device 30 and supplied to the simulation unit 316, where the user's remarks and video are analyzed, and the target person's response to the user's remarks and facial expressions are determined based on the analysis results and transmitted to the user terminal 20 and output to the user.

[0072] The memory unit 32 is realized by a computer memory and storage. The memory unit 32 stores a profile estimation model 321, an emotion estimation model 322, a support information generation model 323, profile information 324, a conversation log 325, and an evaluation model 326. The memory unit 32 also temporarily stores profile information of the user and the subject, observation data, environmental data, and the like transmitted from the user terminal 20. The memory unit 32 may store data and information other than those described above.

[0073] The profile estimation model 321 is trained in advance to receive analysis results of the conversation content and video data and output profile information of the speaker (user and target person). The emotion estimation model 322 is trained in advance to receive inputs of the conversation content, and the volume, tone, intonation, etc. of the conversation and output the target person's emotion. The support information generation model 323 is trained in advance to receive inputs of the target person's profile information, environmental data, and emotion and output support information such as the next topic, questions, and prohibited matters for the target person. The evaluation model 326 is trained in advance to receive inputs of the user's behavior (tone of voice, tempo, tone of voice, volume of voice, facial expression, etc.) and output the user's evaluation (scoring result) and improvements (behavior to be corrected, etc.). These learning models can be realized by, for example, a neural network trained by machine learning or deep learning.

[0074] Instead of using a learning model such as the profile estimation model 321, estimation and generation may be performed by rule-based processing using a predetermined table, function, rule, etc.

[0075] The communication unit 33 is realized by a communication module of a computer. The communication unit 33 connects to the user terminal 20 and the assistant terminal 40 via the network N to communicate various data and information.

[0076] <Data structure of profile information 324> FIG. 3 shows an example of the data structure of the profile information 324 stored in the storage unit 32 of the support device 30. As shown in FIG.

[0077] In the profile information 324, the profile information of the target person who had a conversation with each user is recorded in association with the profile information of the user. The profile information of each user and each target person records a user ID (or target person ID) for identifying the user (or target person), name, age, sex, birthplace, career, recent situation, state of mind, personality, sense of money, career aptitude test results, intelligence test results, etc. The profile information of the target person further records the user's impression of the target person.

[0078] When multiple users have a conversation with the same target person, multiple pieces of profile information about the target person are generated in association with each user, but multiple pieces of profile information about the same target person may be integrated into one. Specifically, the profile update unit 313 may prioritize profile information about the target person associated with a specific user (e.g., a user with high communication skills), or may calculate and integrate specific statistics such as median values ​​and weighted averages for each item of the profile information.

[0079] <Operation of profile information collection system 10> Next, FIG. 4 is a sequence diagram illustrating an example of a profile information collection process by profile information collection system 10. In FIG.

[0080] The profile information collection process is started in response to a predetermined operation from the user to the user terminal 20. First, the data acquisition unit 211 of the user terminal 20 collects profile information (currently known information) of the user himself / herself and the target person who will be the conversation partner input by the user, and the data transfer unit 212 transfers the collected profile information to the support device 30 (step S1).

[0081] Next, the data acquisition unit 311 of the support device 30 acquires the profile information of the user and the target person transmitted from the user terminal 20 and received by the communication unit 33, and outputs it to the profile update unit 313. The profile update unit 313 registers the profile information of the user and the target person in the storage unit 32 (step S11).

[0082] If the user's own profile information has already been registered when registering to use the profile information collection system 10, the user's profile information is selected in steps S1 and S11, for example, by the user inputting a user ID or name. If the user knows the person with whom he or she will be having a conversation and the profile information has already been registered, the user's profile information is selected in steps S1 and S11, for example, by the user inputting the person's ID or name. From step S2 onwards, the profile information registered or selected in steps S1 and S11 is subject to update.

[0083] Next, the data transfer unit 212 of the user terminal 20 transmits a support start request to the support device 30 in response to a predetermined operation from the user notifying the start of a conversation (step S2). Next, the data acquisition unit 211 of the user terminal 20 starts acquiring observation data (conversation data between the user and the target person, and video data of the user and the target person during the conversation) and environmental data. Then, the data transfer unit 212 controls the communication unit 23 to transmit the observation data and environmental data to the support device 30 (step S3).

[0084] In the assistance device 30, the observation data and the environmental data transmitted from the user terminal 20 are received by the communication unit 33 and acquired by the data acquisition unit 311. Then, the data acquisition unit 311 outputs the observation data and the environmental data to the data analysis unit 312, and controls the communication unit 33 to transmit them to the assistant terminal 40 (step S12).

[0085] In assistant terminal 40, advice information for the user input by the assistant based on the observation data and the environmental data is transmitted to assistance device 30 (step S21).

[0086] Next, the data analysis unit 312 of the support device 30 analyzes the conversation data included in the observation data to identify the conversation content, and analyzes the video data. Then, the data analysis unit 312 estimates profile information of the user and the target person based on the conversation content and the analysis result of the video data, and outputs the information to the profile update unit 313. The data analysis unit 312 also inputs the conversation content and the volume, tone, intonation, etc. of the conversation to an emotion estimation model 322 to estimate the emotion of the target person, and outputs the information to the support information generation unit 314. Then, the profile update unit 313 updates the profile information 324 registered in the storage unit 32 with the profile information of the user and the target person estimated by the data analysis unit 312 (step S13).

[0087] Next, the support information generation unit 314 generates support information such as the next topic, questions, and prohibited matters for the subject by inputting the target person's profile information, environmental data, and emotions into the support information generation model 323. The support information generation unit 314 also generates support information based on advice information transmitted from the assistant terminal 40. Then, the support information generation unit 314 controls the communication unit 33 to transmit the generated support information (including the target person's emotions) to the user terminal 20 (step S14).

[0088] In the user terminal 20 that has received the support information from the support device 30, the output control unit 214 generates a support screen 1000 (FIG. 5) for showing the support information to the user, and displays it on the display 28 (step S4).

[0089] Thereafter, the process returns to step S3, and the processes of steps S3, S12, S21, S13, S14, and S4 are repeatedly executed until a predetermined operation is performed by the user to notify the end of the conversation.

[0090] <Display example of Support Screen 1000> FIG. 5 shows an example of a support screen 1000 displayed on the display 28 of the support device 30. As shown in FIG.

[0091] The support screen 1000 is provided with an emotion indicator 1001 , an image display field 1002 , a prohibited items display field 1003 , a topic display area 1004 , a question display area 1005 , and an advice display area 1006 .

[0092] The emotion indicator 1001 indicates whether the emotion of the subject tends to be unpleasant (anger, sadness) or pleasant (joy, happiness). In conjunction with the display of the emotion indicator 1001, for example, a vibrator may vibrate quickly in the case of unpleasantness and slowly in the case of pleasantness. The image display field 1002 displays an image of the subject acquired by the data acquisition unit 211. In the image display field 1002, for example, the image of the subject may be displayed after being deformed or illustrated. The emotion of the subject may also be visualized (in the case of the figure, an anger mark 10021 is added) and displayed.

[0093] The prohibited matters display field 1003 displays prohibited matters that should not be discussed with the target person. The topic display area 1004 displays the next topic. The question display area 1005 displays a question for the target person. The advice display area 1006 displays advice for the target person.

[0094] The support screen 1000 is displayed by the output control unit 214 of the user terminal 20 based on the support information, but for example, the support device 30 may prepare the support screen 1000 as, for example, a Web page and display it on the Web browser of the user terminal 20.

[0095] According to the profile information collection process described above, profile information can be collected from conversations with the target person, and based on the collected profile information, support information can be provided to the user that enables the user to promptly and appropriately elicit necessary information from the target person in the ever-changing flow of the conversation, without touching on the target person's subtle emotions and causing the other person discomfort.

[0096] <Application Examples> In the above-described embodiment, it is assumed that the subject is a single person or that statements and images can be collected individually even if there are multiple subjects, but the subject may be a crowd.

[0097] 6 shows an application example in which a user holds a lecture, speech, seminar, etc. for a crowd of people. In this case, the user terminal 20 collects sounds (specifically, applause, cheers, cheers, shouts, etc.) from the entire crowd and can capture an image of the entire crowd.

[0098] In this application example, voices and images of a crowd are collected and analyzed to identify the interests, level of understanding, empathy, and other interests of the crowd, and provide these to the user. This allows the user to hold lectures, speeches, seminars, etc. with content appropriate for the crowd.

[0099] <Other application examples> The present invention is not limited to cases where profile information is collected to assist in conversation, but can also be applied to, for example, diagnosis of a subject's personality, judgment of suitable occupation, and measurement of trustworthiness.

[0100] <Modification> The profile information collection system 10 may be configured by the user terminal 20 alone, without providing the support device 30. In that case, the processing unit 21 of the user terminal 20 may be provided with the data analysis unit 312, the profile update unit 313, and the support information generation unit 314 that the processing unit 31 of the support device 30 has, and further, the storage unit 22 of the user terminal 20 may store the profile estimation model 321, the emotion estimation model 322, and the support information generation model 323.

[0101] The present invention is not limited to the above-described embodiment and modifications, and various modifications are possible. For example, the above-described embodiment and modifications have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the configurations described. In addition, it is possible to replace a part of a modification with another modification, or to combine the modifications.

[0102] In addition, each of the above configurations, functions, processing units, processing means, etc. may be realized in hardware, for example, by designing a part or all of them as an integrated circuit. In addition, each of the above configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as a program, table, file, etc. that realizes each function can be placed in a memory, a recording device such as a hard disk or SSD, or a recording medium such as an IC card, an SD card, or a DVD. In addition, the control lines and information lines are shown as those considered necessary for explanation, and not all control lines and information lines are necessarily shown in the product. In reality, it may be considered that almost all configurations are connected to each other. [Explanation of symbols]

[0103] 10 Profile information collection system, 20 User terminal, 21 Processing unit, 211... data acquisition unit, 212... data transfer unit, 213... support information acquisition unit, 214... output control unit, 22... storage unit, 23... communication unit, 24... microphone, 25A, 25B... camera, 26... sensor, 27... speaker, 28... display, 30... support device, 31... processing unit, 311... data acquisition unit, 312... data analysis unit, 313... Profile update unit, 314, support information generation unit, 315, evaluation unit, 316, simulation unit, 32, memory unit, 321, profile estimation model, 322, emotion estimation model, 323, support information generation model, 324, profile information, 325, conversation log, 326, evaluation model, 33, communication unit, 40, assistant terminal, 1000, support screen

Claims

1. A profile information collection system comprising a user terminal and an assistance device communicating with the user terminal, The support device includes: a data acquisition unit that acquires observation data relating to a target person having a conversation with a user of the user terminal from the user terminal; a data analysis unit that analyzes the observation data and estimates profile information of the subject based on a result of the analysis of the observation data; a profile update unit that updates registered profile information of the subject with the estimated profile information; a support information generating unit that generates support information for the user to support a conversation with the target person based on the profile information of the target person; An evaluation unit that evaluates a behavior of the user, The data acquisition unit acquires the observation data including audio data and video data of a conversation between the user and the target person, The data analysis unit Analyzing the voice data to identify the content of the conversation; extracting the profile information of the subject from the content of the conversation; and inputting at least one of the content of the conversation and an analysis result of the video data into a profile estimation model, thereby estimating the profile information of the subject; Inferring profile information of the user including a personality of the user based on the content of the conversation; generating a conversation log in which the observation data and the conversation content are associated with a time axis; the support information generation unit generates the support information for the user by inputting the profile information of the subject and at least one of the personality, qualities, and abilities of the user into a support information generation model; The evaluation unit analyzes the conversation log, identifies at least one of tone, tempo, tone of voice, volume, and facial expression as the behavior of the user, and inputs the identified behavior into a trained evaluation model, thereby identifying at least one of a scoring result and an area for improvement as an evaluation of the behavior of the user; The user terminal, an output control unit that outputs the assistance information from the assistance device and the behavior evaluation of the user to the user; Profile information collection system.

2. 2. The profile information collection system according to claim 1, The data analysis unit estimates at least one of age, sex, place of origin, career, recent situation, state of mind, and personality as the profile information of the target person based on the content of the conversation. Profile information collection system.

3. 2. The profile information collection system according to claim 1, The support information generating unit generates at least one of the following topics, questions, and prohibited matters as the support information based on the profile information of the subject. Profile information collection system.

4. 2. The profile information collection system according to claim 1, The data acquisition unit acquires environmental data including a location of the subject from the user terminal, The support information generation unit generates the support information based on the profile information of the subject and the environmental data. Profile information collection system.

5. 2. The profile information collection system according to claim 1, The data analysis unit analyzes the voice data to estimate an emotion of the subject, The support information generation unit generates the support information based on the profile information and the emotion of the subject. Profile information collection system.

6. 6. The profile information collection system according to claim 5, The observation data includes an image of the user, The support information generation unit determines, as the support information, a facial expression that the user should make, based on at least one of the content of the conversation, the image of the user, and the emotion of the target person. Profile information collection system.

7. 2. The profile information collection system according to claim 1, The output control unit displays a support screen based on the support information from the support device. Profile information collection system.

8. 6. The profile information collection system according to claim 5, The support information from the support device includes an emotion of the subject, The output control unit displays a support screen in which the emotion of the subject is visualized. Profile information collection system.

9. 9. The profile information collection system according to claim 8, The output control unit deforms an image of the target person and displays the deformed image on the support screen. Profile information collection system.

10. 2. The profile information collection system according to claim 1, The support information generation unit generates the support information including advice information from an assistant based on the observation data of the subject. Profile information collection system.

11. 2. The profile information collection system according to claim 1, the support information generation unit generates the support information based on an action of the user; The data analysis unit estimates the profile information of the subject based on the subject's reaction to the action of the user. Profile information collection system.

12. 2. The profile information collection system according to claim 1, The support device includes: a simulation unit that executes a simulation of a conversation between the user and the target person before the user has a conversation with the target person; Profile information collection system.

13. 2. The profile information collection system according to claim 1, The support information generation unit confirms the estimated profile information of the target person to the user. Profile information collection system.

14. 2. The profile information collection system according to claim 1, The profile update unit integrates a plurality of pieces of profile information related to the same target person that are generated in association with a plurality of different users. Profile information collection system.

15. A profile information collection method by a profile information collection system including a user terminal and an assistance device that communicates with the user terminal, The support device a data acquisition step of acquiring observation data relating to a target person having a conversation with a user of the user terminal from the user terminal; a data analysis step of analyzing the observation data; a profile updating step of estimating and updating profile information of the subject based on an analysis result of the observation data; a support information generating step of generating support information for the user to support a conversation with the target person based on the profile information of the target person; an evaluation step of evaluating a behavior of the user; By the user terminal, an output control step of outputting the generated support information to the user; The data acquisition step acquires the observation data including audio data and video data of a conversation between the user and the target person; The data analysis step includes: Analyzing the voice data to identify the content of the conversation; extracting the profile information of the subject from the content of the conversation; and inputting at least one of the content of the conversation and an analysis result of the video data into a profile estimation model, thereby estimating the profile information of the subject; Inferring profile information of the user including a personality of the user based on the content of the conversation; generating a conversation log in which the observation data and the conversation content are associated with a time axis; The support information generating step generates the support information for the user by inputting the profile information of the subject and at least one of the personality, qualities, and abilities of the user into a support information generation model; The evaluation step includes analyzing the conversation log, identifying at least one of tone, tempo, tone of voice, volume, and facial expression as the behavior of the user, and inputting the identified behavior into a trained evaluation model to identify at least one of a score result and an area for improvement as an evaluation of the behavior of the user; The output control step outputs the behavior evaluation of the identified user to the user. How we collect profile information.

16. A computer as a support device that communicates with a user terminal, a data acquisition step of acquiring observation data relating to a target person having a conversation with a user of the user terminal from the user terminal; a data analysis step of analyzing the observation data; a profile updating step of estimating and updating profile information of the subject based on an analysis result of the observation data; a support information generating step of generating support information for the user to support a conversation with the target person based on the profile information of the target person; evaluating the user's behavior; The data acquisition step acquires the observation data including audio data and video data of a conversation between the user and the target person; The data analysis step includes: Analyzing the voice data to identify the content of the conversation; extracting the profile information of the subject from the content of the conversation; and inputting at least one of the content of the conversation and an analysis result of the video data into a profile estimation model, thereby estimating the profile information of the subject; Inferring profile information of the user including a personality of the user based on the content of the conversation; generating a conversation log in which the observation data and the conversation content are associated with a time axis; The support information generating step generates the support information for the user by inputting the profile information of the subject and at least one of the personality, qualities, and abilities of the user into a support information generation model; The evaluation step includes analyzing the conversation log, identifying at least one of tone, tempo, tone of voice, volume, and facial expression as the behavior of the user, and inputting the identified behavior into a trained evaluation model, thereby identifying at least one of a score result and an area for improvement as an evaluation of the behavior of the user. program.

17. On the computer, A data acquisition step of acquiring observation data on a target person having a conversation with the user; a data analysis step of analyzing the observation data; a profile updating step of estimating and updating profile information of the subject based on an analysis result of the observation data; a support information generating step of generating support information for the user to support a conversation with the target person based on the profile information of the target person; an output control step of outputting the support information to the user; evaluating the user's behavior; The data acquisition step acquires the observation data including audio data and video data of a conversation between the user and the target person; The data analysis step includes: Analyzing the voice data to identify the content of the conversation; extracting the profile information of the subject from the content of the conversation; and inputting at least one of the content of the conversation and an analysis result of the video data into a profile estimation model, thereby estimating the profile information of the subject; Inferring profile information of the user including a personality of the user based on the content of the conversation; generating a conversation log in which the observation data and the conversation content are associated with a time axis; The support information generating step generates the support information for the user by inputting the profile information of the subject and at least one of the personality, qualities, and abilities of the user into a support information generation model; The evaluation step includes analyzing the conversation log, identifying at least one of tone, tempo, tone of voice, volume, and facial expression as the behavior of the user, and inputting the identified behavior into a trained evaluation model, thereby identifying at least one of a score result and an area for improvement as an evaluation of the behavior of the user. program.

Images