COMMUNICATION METHOD, COMMUNICATION TERMINAL, AND PROGRAM

Abstract

In a communication system for performing a teleconference or the like, reaction icons for indicating users' reactions are prepared so that a listener can transmit to a speaker information indicating a reaction icon selected at any timing, regardless of whether video or audio is turned on or off at the listener's site, allowing the speaker to recognize the reaction of a remote listener. In this case, however, it is necessary for the listener to consciously select and operate a reaction icon to indicate the listener's own reaction. Besides, it is difficult for the listener to convey an unconscious feeling of the listener to the speaker because of the unconsciousness.　A communication terminal 3 uses a machine-learned reaction estimation model to estimate the reaction of a user A on the basis of a voice uttered by the user A and a facial expression of the user A, and then transmits information of a reaction estimation result to a communication terminal 7. In this way, a reaction icon 9 indicating the reaction of the user A is displayed on the communication terminal 7 of a user B.

Description

[Technical field]

[0001] The present disclosure relates to a communication method, a communication terminal, and a program. [Background technology]

[0002] In recent years, communication systems for remote conferences, etc. have become widespread. In this communication system, in addition to calls between users, an invention has been proposed that allows users to know the status of the other user by an icon (see Patent Document 1).

[0003] In addition, in remote conferences, listeners often turn off their video and audio to prevent communication from being hindered by line congestion or noise, but this creates a trade-off in that it makes it difficult for the speaker to understand the listener's reactions.

[0004] However, as shown in FIG. 10, reaction icons indicating user reactions are provided, so even if the listener has turned off the video and audio at his / her own location (or has turned them off), the listener can send information indicating the selected reaction icon at any time to the speaker, allowing the speaker to grasp the reaction of the remote listener. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2002-351805 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, in order for the user as a speaker to understand the reaction of the user as a listener, the listener must consciously select the reaction icon that indicates his / her own reaction. In addition, there is also the problem that it is difficult for the listener to convey his / her own unconscious feelings to the speaker because they are unconscious.

[0007] The present invention has been made in consideration of the above circumstances, and has an object to enable a user at one location to communicate his / her own reaction to a user at another location without performing any operation to communicate his / her reaction to a user at the other location. [Means for solving the problem]

[0008] The invention according to claim 1 is a communication method executed by a communication terminal capable of communicating with another communication terminal at another site, the communication terminal executes the following steps: an acquisition step of acquiring data on the behavior of a specific user from the behavior of the specific user; an unconscious emotion part detection step of detecting data on a specific unconscious emotion part in which the specific user unconsciously expresses emotion from the data on the behavior of the specific user; a reaction estimation step of estimating a reaction by the facial expression of the specific user based on the data on the specific unconscious emotion part detected by the unconscious emotion part detection step using a reaction estimation model for estimating a reaction by the facial expression of the specific user to the data on the specific unconscious emotion part in which the specific user unconsciously expresses emotion; and a transmission step of transmitting reaction estimation result information related to the reaction by the facial expression of the specific user estimated by the reaction estimation step to the other communication terminal. the acquiring step includes a process of acquiring data on the predetermined user's actions from the predetermined user's actions, the unconscious emotion portion detecting step includes a process of detecting data on the predetermined unconscious emotion portion in which the predetermined user's unconscious emotion expression occurs from the data on the predetermined user's actions, and the communication terminal includes a recognition step of recognizing at least one of a voice indicating the content uttered by the predetermined user with respect to the data on the predetermined unconscious emotion portion and a facial expression of the predetermined user, a reaction information extraction step of extracting reaction information indicating the reaction of the predetermined user in the predetermined unconscious emotion portion based on the recognition result by the recognition step, and a model generating step of generating the reaction estimation model by performing machine learning so that the reaction of the predetermined user indicated by the reaction information with respect to the data on the predetermined unconscious emotion portion can be estimated, The communication method is characterized by carrying out the following: Effect of the Invention

[0009] As described above, according to the present invention, it is possible to provide an effect that a user at a location can transmit his / her own reaction to another location without any operation by the user at the location. [Brief description of the drawings]

[0010] [Figure 1] 1 is a schematic diagram of a communication system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating an electrical hardware configuration of a communication terminal. [Diagram 3] FIG. 2 is a functional block diagram of a communication terminal. [Figure 4]FIG. 13 is a conceptual diagram showing time information. [Diagram 5] FIG. 1 is a conceptual diagram showing reaction information. [Figure 6] 1 is a flowchart showing a machine learning process. [Figure 7] 13 is a flowchart showing a process for estimating a reaction. [Figure 8] FIG. 13 is a diagram showing a profile image showing a neutral expression. [Figure 9] FIG. 13 is a diagram showing a profile image showing a surprised expression. [Figure 10] FIG. 13 is a diagram showing reaction icons. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] [System Overview] First, an overview of the communication system according to the present embodiment will be described with reference to Fig. 1. Fig. 1 is a schematic diagram of the communication system according to the embodiment of the present invention.

[0013] A communication system 1 shown in Fig. 1 is a system for remote calls such as a video conference. As shown in Fig. 1, the communication system 1 is constructed by a communication terminal 3 at a base of a user A and a communication terminal 7 at a base of a user B. The communication terminals 3 and 7 are general personal computers (PCs) and are communicatively connected to each other via a communication network 100 such as the Internet or a local area network (LAN). The communication network 100 may be connected in either a wireless or wired manner.

[0014] The communication terminal 3 has a function of not transmitting data (audio data, video data) related to the actions of user A at its own base to the communication terminal 7 during a call with the communication terminal 7. Similarly, the communication terminal 7 has a function of not transmitting data (audio data, video data) related to the actions of user B at its own base to the communication terminal 3 during a call with the communication terminal 3.

[0015] FIG. 1 shows a state in which users A and B do not show each other their appearances. However, a profile image 4 of user B is displayed on the communication terminal 3 of user A, and a profile image 8 of user A is displayed on the communication terminal 7 of user B. In addition, users A and B communicate with each other, and FIG. 1 shows a state in which user A causes an Affect Burst of surprise when user B talks to user A. An Affect Burst is an unconscious expression of emotion such as "haha" or "eh" through facial expressions (gestures) or laughter. In addition, a reaction icon 9 showing user A's reaction of surprise is displayed on the communication terminal 7 of user B. User B transmits user B's voice to user A, but user A may or may not transmit user A's voice to user B.

[0016] 1 shows two communication terminals, 3 and 7, but may be three or more. Also, on the communication terminal 3 side of user A, the profile image 4 of user B may not be displayed, and the facial expression of user B may be displayed.

[0017] Furthermore, on the communication terminal 7 of user B, the profile image 8 of user A is not displayed, and the facial expression of user A may or may not be displayed. However, if the facial expression of user A is not displayed, the significance of displaying the reaction icon 9 is enhanced.

[0018] In addition, the display form of the profile images 4, 8 and reaction icons 9 shown in FIG. 1 is merely an example, and they may be displayed in other forms by changing the shape, color, and size, adding text, etc.

[0019] [Hardware configuration] Next, the electrical hardware configuration of the communication terminal 3 will be described with reference to Fig. 2. Fig. 2 is a diagram showing the electrical hardware configuration of the communication terminal.

[0020] As shown in FIG. 2, the communication terminal 3 includes a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, a RAM (Random Access Memory) 303, an SSD (Solid State Drive) 304, a CMOS (Complementary Metal Oxide Semiconductor) sensor 305, and an external device I / F (Interface) 306.

[0021] Of these, the CPU 301 controls the overall operation of the communication terminal 3. The ROM 302 stores the CPU 301 and programs used to drive the CPU 301, such as the IPL. The RAM 303 is used as a work area for the CPU 301.

[0022] The SSD 304 reads and writes various data such as programs for the communication terminal under the control of the CPU 301. Note that instead of an SSD, a storage device such as an HDD (Hard Disk Drive) may also be used.

[0023] The CMOS sensor 305 is a type of built-in imaging means that captures an object or the like to obtain video (image) data under the control of the CPU 301. Note that instead of a CMOS sensor, an imaging element such as a CCD (Charge Coupled Device) sensor may also be used.

[0024] The external device I / F 306 is an interface for connecting various external devices. In this case, the external devices are an external display as an example of a display means, a mouse, a keyboard, or a microphone as an example of an input means, a printer or a speaker as an example of an output means, a USB (Universal Serial Bus) memory as an example of a storage means, etc.

[0025] The communication terminal 3 also includes a microphone 307, a speaker 308, an audio input / output I / F 309, a display 310, a network I / F 311, a communication circuit 312, and an antenna 312a of the communication circuit 312.

[0026] Of these, the microphone 307 is a built-in circuit that converts sound into an electrical signal. The speaker 308 is a built-in circuit that converts the electrical signal into physical vibrations to produce sound such as music or voice. The audio input / output I / F 309 is a circuit that processes input and output of audio signals between the microphone 307 and the speaker 308 under the control of the CPU 301.

[0027] The display 310 is a type of display means such as a liquid crystal display or an organic EL (Electro Luminescence) display that displays an image of a subject, various icons, and the like.

[0028] The network I / F 311 is a circuit for transmitting and receiving data and the like to and from communication terminals other than the communication terminal 3 and servers via the communication network 100.

[0029] The communication circuit 312 is a circuit for communicating data with other devices using an antenna 312a and a short-range wireless communication technology such as NFC (Near field communication) or Bluetooth (registered trademark).

[0030] The communication terminal 3 also includes a bus line 320. The bus line 320 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 301 shown in FIG.

[0031] In addition, since the electrical hardware configuration of the communication terminal 7 is similar to that of the communication terminal 3, in Fig. 2, the reference numbers are simply replaced with numbers in the 700 range instead of the 300 range. Therefore, a description of the electrical hardware configuration of the communication terminal 7 will be omitted.

[0032] [Functional configuration of the embodiment] Next, the functional configuration of this embodiment will be described with reference to Figures 2 to 5. Figure 3 is a functional block diagram of a communication terminal.

[0033] 3, the communication terminal 3 has a learning processing unit 5, an estimation processing unit 6, a transmission / reception unit 30, a sound acquisition unit 31, a video acquisition unit 32, and an unconscious emotion part detection unit 33. Each of these units is a function or means realized by any of the components shown in FIG. 2 operating according to an instruction from the CPU 301 in accordance with a program loaded from the SSD 304 onto the RAM 303. The communication terminal 3 also has a storage unit 40 constructed by the RAM 303 or the SSD 304 shown in FIG. 2.

[0034] Of these, the learning processing unit 5 performs machine learning processing so as to estimate a user's reaction to data on the unconscious emotion parts (parts where an Affect Burst occurs) of each of the sound data acquired by the sound acquisition unit 31 and the video data acquired by the video acquisition unit 32. The estimation processing unit 6 uses a reaction estimation model machine-learned by the learning processing unit 5 to perform processing to infer a specific user's reaction from data on a specific unconscious emotion part detected by the unconscious emotion part unit 33.

[0035] The transmitting / receiving unit 30 of the communication terminal 3 transmits and receives various data to and from the communication terminal 7 via the communication network 100. For example, the transmitting / receiving unit 30 receives, from the communication terminal 7, voice data indicating what the user B said, and video (image) data indicating the facial expression of the user B. The transmitting / receiving unit 30 also receives the voice data and video data of the user B from the communication terminal 7.

[0036] The sound acquisition unit 31 acquires audio data indicating what the user said, collected by the microphone 307. The video acquisition unit 32 acquires video data indicating the user's facial expression, etc., captured by the CMOS sensor 305. Note that the user's actions with the mouth or nose and the user's actions changing the facial expression are examples of user actions. In addition, the audio data and video data in this case are examples of data related to the user's actions. However, the data related to the user's actions may be at least one of audio data and video data.

[0037] The unconscious emotion part detection unit 33 detects data of unconscious emotion parts where the user unconsciously expresses emotion from the audio data acquired by the sound acquisition unit 31 and the video data acquired by the video acquisition unit 32. The extraction method is disclosed in Reference 1.

[0038] Reference 1: BB Turker, S. Marzban, MT Sezgin, Y. Yemez and E. Erzin, "Affect burst detection using multi-modal cues," 2015 23rd Signal Processing and Communications Applications Conference (SIU), Malatya, Turkey, 2015, pp. 1006-1009. In addition, the unconscious emotion part detection unit 33 stores in the storage unit 40 time information including the date and time when the detection of the data of the unconscious emotion part was started among the voice data acquired by the sound acquisition unit 31, and the duration from this date and time to the date and time when the acquisition of the data of the unconscious emotion part was temporarily terminated. FIG. 4 is a conceptual diagram showing the time information. As shown in FIG. 4, the time information is information in which the date and time when the detection of the data of the unconscious emotion part was started (date and time stamp date and time) and the duration (milliseconds) from this date and time to the date and time when the acquisition of the data of the unconscious emotion part was temporarily terminated are associated in the order of the serial number (chronological order). In addition, the unconscious emotion part detection unit 33 processes the video data acquired by the video acquisition unit 32 in the same manner. All the sound data acquired by the sound acquisition unit 31 and all the video data acquired by the video acquisition unit 32 are stored in the storage unit 40.

[0039] Returning to FIG. 3, the learning processing unit 5 has a response extraction unit 50 and a model generation unit 55.

[0040] The reaction extraction unit 50 reads out data of the unconscious emotion part corresponding to the time information stored in the storage unit 40 from the voice data and video data stored in the storage unit 40, and extracts the user's reaction from this unconscious emotion part. To carry out this process, the reaction extraction unit 50 further includes a voice recognition unit 51, a face recognition unit 52, and a reaction information extraction unit 53.

[0041] Among these, the voice recognition unit 51 recognizes the voice indicating the contents uttered by the user in response to the data of the unconscious emotion portion, and the face recognition unit 52 recognizes the facial expression of the user in response to the data of the unconscious emotion portion.

[0042] Then, the reaction information extraction unit 53 extracts reaction information indicating the user's reaction in the unconscious emotion part from the voice recognized by the voice recognition unit 51. Also, the reaction information extraction unit 53 extracts reaction information indicating the user's reaction in the unconscious emotion part from the facial expression recognized by the face recognition unit 52. For example, the extraction method is disclosed in Reference 2.

[0043] Reference 2: BT Nguyen, MH Trinh, TV Phan and HD Nguyen, "An real-time emotion detection using camera and facial landmarks," 2017 Seventh International Conference on Information efficient Science and Technology (ICIST), Da Nang, 2017, pp. 251-255. Reference 2 discloses a method for extracting feature points from a facial image and estimating their patterns using machine learning.

[0044] Fig. 5 is a conceptual diagram showing reaction information. As shown in Fig. 5, the reaction information is information showing percentage scores of happiness, sadness, surprise, neutral, negative, etc. in the order of serial number (chronological order).

[0045] The model generation unit 55 also generates a reaction estimation model (Affect Burst-reaction model) by performing machine learning so that the reaction such as joy of the user can be estimated based on the data of the unconscious emotion part. That is, the model generation unit 55 performs machine learning using the reaction information extracted by the reaction information extraction unit 53 as a correct answer label and the data of the unconscious emotion part as input data. For example, the model generation unit 55 performs learning using a CNN (Convolutional Neural Network) so as to extract respective feature amounts from the voice and video of the unconscious emotion part. The data of the unconscious emotion part used by the model generation unit 55 as input data is obtained from a predetermined user including user A.

[0046] Next, the estimation processing unit 6 has a reaction estimation unit 60. The reaction estimation unit 60 uses a reaction estimation model to estimate a reaction of a specific user, that is, user A, based on data of the unconscious emotion part detected by the unconscious emotion part detection unit 33. In this case, the reaction estimation unit 60 estimates the reaction of user A based on data of a specific unconscious emotion part corresponding to time information stored in the storage unit 40, among data (audio data, video data) related to the action of user A.

[0047] Then, the reaction estimation result information relating to the reaction of user A estimated by the reaction estimation unit 60 is transmitted to the communication terminal 7 by the transmission / reception unit 30 as described above. This reaction estimation result information is information that allows the communication terminal 7 of user B to display a reaction icon 9 indicating the reaction of user A, as shown in Fig. 1. Note that the reaction estimation result information is again used in machine learning as the above-mentioned reaction information (see Fig. 4).

[0048] [Processing or Operation of the Embodiment] Next, the processing or operation of this embodiment will be described with reference to FIGS.

[0049] <Machine learning processing> First, the machine learning process executed by the communication terminal 3 will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the machine learning process.

[0050] As shown in FIG. 6, the sound acquisition unit 31 acquires audio data indicating what is being said by user A through the mouth, etc., and the video acquisition unit 32 acquires video data indicating the facial expression of user A (S11).

[0051] Next, the unconscious emotion part detection unit 33 detects the unconscious emotion part in the audio (video) data (S12). Then, the unconscious emotion part detection unit 33 stores the audio (video) data and time information of the unconscious emotion part in the storage unit 40 (S13).

[0052] Next, the reaction extraction unit 50 reads out data of the unconscious emotion part in the audio (video) data based on the time information stored in the storage unit 40 (S14). Then, the sound recognition unit 51 recognizes the voice of the data of the unconscious emotion part, and the face recognition unit 52 recognizes the facial expression of the data of the unconscious emotion part (S15). Furthermore, the reaction information extraction unit 53 extracts reaction information of the unconscious emotion part based on the recognition results by the sound recognition unit 51 and the recognition results by the face recognition unit 52 (S16).

[0053] Next, the model generation unit 55 performs machine learning using the reaction estimation model so that reaction information can be estimated for the data of the unconscious emotion part (S17). Then, the model generation unit 55 stores the data of the reaction estimation model after the machine learning in the storage unit 40.

[0054] With the above, the machine learning process executed by the communications terminal 3 ends.

[0055] <Process to estimate reaction> Next, the process of estimating a reaction of user A executed by the communication terminal 3 will be described with reference to Fig. 7. Fig. 7 is a flowchart showing the process of estimating a reaction. Note that steps S21 to S24 are the same as steps S11 to S14 described above.

[0056] As shown in FIG. 6, the sound acquisition unit 31 acquires audio data indicating what is being said by the user A through the mouth, etc., and the video acquisition unit 32 acquires video data indicating the facial expression of the user A (S21).

[0057] Next, the unconscious emotion part detection unit 33 detects the unconscious emotion part in the audio (video) data (S22). Then, the unconscious emotion part detection unit 33 stores the audio (video) data and time information of the unconscious emotion part in the storage unit 40 (S23).

[0058] Next, the reaction extraction section 50 reads out data of the unconscious emotion part in the audio (video) data based on the time information stored in the storage section 40 (S24).

[0059] Next, the reaction estimation unit 60 uses the machine-learned reaction estimation model to estimate the reaction of user A to the data of the unconscious emotion part (S25).

[0060] The transmitter / receiver 30 transmits reaction estimation result information relating to the reaction of user A estimated by the reaction estimation unit 60 to the communication terminal 7 (S26).

[0061] As a result, as shown in FIG. 1, a reaction icon 9 indicating user A's reaction is displayed on the display 710 of user B's communication terminal 7.

[0062] This completes the process of estimating a reaction executed by the communication terminal 3. In the above-mentioned <machine learning process>, voice (face) recognition is performed (see S15) and a reaction estimation model is generated (see S17). In contrast, in the <process of estimating a reaction>, voice (face) recognition is not performed, and the reaction of user A is directly estimated from the data of the unconscious emotion portion using the reaction estimation model (S25).

[0063] [Main Effects of the Present Embodiment] As described above, according to this embodiment, the communication terminal 3 can estimate the reaction of the user A based on the voice uttered by the user A and the facial expression of the user A, and transmit the reaction estimation result information to the communication terminal 7. This has the effect of enabling the user A's reaction to be conveyed to a partner (user B) at another base, without the user A having to perform an operation to convey his / her own reaction to user B.

[0064] 〔supplement〕 The present invention is not limited to the above-described embodiment, and may have the following configurations or processes (operations). (1) The communication terminals 3 and 7 of the present invention can be realized by a computer and a program. This program can be recorded on a recording medium or provided through a communication network. (2) In the above embodiment, the reaction icon 9 is displayed as shown in Fig. 1, but the present invention is not limited to this. For example, the communication terminal 7 may initially display a profile image 8a of the user A as shown in Fig. 8, and when reaction estimation result information is sent from the communication terminal 3, the communication terminal 7 may switch to display a profile image 8b in a display mode showing surprise as shown in Fig. 9. (3) The profile image described above is an example of a user identification image. User identification images also include avatars. (4) In the above embodiment, a PC is shown as an example of a communication terminal, but the PC includes a desktop computer and a notebook computer. Other examples of communication terminals include a smart watch and a car navigation device. (5) Each of the CPUs 301 and 701 may be multiple, rather than just single. [Explanation of symbols]

[0065] 1. Communication Systems 3. Communication terminals 4. (User B's) profile picture 5 Learning processing section 6 Estimation processing section 7. Communication terminal (an example of another communication terminal) 8. (User A's) Profile image 9 Reaction Icons 30 Transmitter / receiver unit (an example of a transmitting means) 31 Sound acquisition unit (an example of an acquisition means) 32 Image acquisition unit (an example of an acquisition means) 33 Unconscious emotion part detection unit (an example of unconscious emotion part detection means) 40 Memory unit (an example of a memory means) 50 Reaction extraction unit (an example of a reaction extraction means) 51 Voice recognition unit (an example of a voice recognition means, an example of a recognition means) 52 Face recognition unit (an example of face recognition means, an example of recognition means) 53 Reaction information extraction unit 55 Model generation unit (an example of a model generation means) 60 Reaction estimation unit (an example of a reaction estimation means) 710 Display (an example of a display means)

Claims

1. A communication method executed by a communication terminal capable of communicating with another communication terminal at another location, The communication terminal includes: An acquisition step of acquiring data related to a specific user's behavior from the specific user's behavior; an unconscious emotion part detection step of detecting data of a specific unconscious emotion part where an unconscious emotion expression of the specific user occurs from data related to the action of the specific user; a reaction estimation step of estimating a reaction by a facial expression of the specific user based on the data of the specific unconscious emotion part detected by the unconscious emotion part detection step, using a reaction estimation model for estimating a reaction by the facial expression of the specific user to data of a specific unconscious emotion part in which an unconscious expression of emotion of the specific user occurs; a transmitting step of transmitting reaction estimation result information relating to a reaction by a facial expression of the specific user estimated by the reaction estimating step to the other communication terminal; Run The acquiring step includes a process of acquiring data related to the predetermined user's action from the predetermined user's action, the unconscious emotion portion detection step includes a process of detecting data of the predetermined unconscious emotion portion in which an unconscious expression of emotion of the predetermined user occurs from data related to the motion of the predetermined user, The communication terminal includes: a recognition step of recognizing at least one of a voice indicating the content uttered by the predetermined user with respect to the data of the predetermined unconscious emotion portion and a facial expression of the predetermined user; a reaction information extraction step of extracting reaction information indicating a reaction of the predetermined user in the predetermined unconscious emotion part based on a recognition result by the recognition step; a model generation step of generating the reaction estimation model by performing machine learning so as to estimate the reaction of the predetermined user indicated by the reaction information with respect to the data of the predetermined unconscious emotion part; A communication method comprising:

2. the communication terminal has a storage means for storing time information including a date and time when detection of the specific unconscious emotion part data was started by the unconscious emotion part detection step, and a duration from the date and time to a date and time when the detection was once terminated; The communication method according to claim 1, characterized in that the reaction estimation step includes a process of estimating a reaction of the specific user based on data on the specific unconscious emotion part corresponding to the time information stored in the storage means, among data regarding the actions of the specific user.

3. The communication method according to claim 1, characterized in that, depending on the reaction estimation result indicated in the reaction estimation result information, the display mode of the reaction mark displayed by the other communication terminal changes, or the display mode of the user identification image of the specific user displayed by the other communication terminal changes.

4. In the case where the data on the action of the predetermined user is voice data indicating the contents of the speech of the predetermined user, the recognition step includes a process of recognizing the voice of the predetermined user against the data of the predetermined unconscious emotion part, In the case where the data on the action of the predetermined user is video data showing the facial expression of the predetermined user, the recognition step includes a process of recognizing the facial expression of the predetermined user with respect to the data of the predetermined unconscious emotion portion.

2. The communication method according to claim 1 .

5. 2. The communication method according to claim 1, wherein the communication terminal has a function of not transmitting data relating to the operation of the specific user at the own base to the other communication terminal while in a call with the other communication terminal.

6. A communication terminal capable of communicating with other communication terminals at other locations, An acquisition means for acquiring data related to a specific user's action from the specific user's action; an unconscious emotion part detection means for detecting data on a specific unconscious emotion part where an unconscious emotion expression of the specific user occurs from data on the action of the specific user; a reaction estimation means for estimating a reaction by a facial expression of a specific user from data of a specific unconscious emotion part detected by the unconscious emotion part detection means, using a reaction estimation model that has been machine-learned to estimate a reaction by the facial expression of the specific user to data of a specific unconscious emotion part in which an unconscious expression of emotion of the specific user occurs; a transmitting means for transmitting reaction estimation result information relating to a reaction by a facial expression of the specific user estimated by the reaction estimation means to the other communication terminal; having The acquiring means acquires data related to the predetermined user's action from the predetermined user's action, The unconscious emotion portion detection means detects data on the predetermined unconscious emotion portion in which an unconscious expression of emotion of the predetermined user occurs from data on the motion of the predetermined user, a recognition means for recognizing at least one of a voice indicating the content uttered by the predetermined user in response to the data of the predetermined unconscious emotion portion and a facial expression of the predetermined user; A reaction information extracting means for extracting reaction information indicating a reaction of the predetermined user in the predetermined unconscious emotion part based on a recognition result by the recognizing means; a model generation means for generating the reaction estimation model by performing machine learning so as to be able to estimate the reaction of the predetermined user indicated by the reaction information with respect to the data of the predetermined unconscious emotion part; A communication terminal comprising:

7. A program causing a computer to execute the method according to any one of claims 1 to 5.

Images