Program, information processing method, and information processing device

Abstract

To enable gaze shift processing so as to give a positive emotion to a conversation partner.SOLUTION: A gaze correction unit performs gaze shift processing for shifting a gaze of a user in an image of the user such that the gaze of the user does not match a gaze of a conversation partner of the user on the basis of emotion-related information being information related to an emotion of the conversation partner of the user. The present technique is applicable to, for example, an online conferencing system, which is configured by connecting a plurality of PCs and a cloud computer through a network such as the Internet so as to enable the users to conduct a conference to have conversations, presentations, or the like by transmitting and receiving images and audio of the users between the PCs.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present technology relates to a program, an information processing method, and an information processing device, and in particular to a program, an information processing method, and an information processing device that are capable of performing gaze averting processing so as to give positive feelings to a conversation partner. [Background technology]

[0002] In an online conference system, the user's gaze may not match the gaze of the conversation partner due to a misalignment between the user's gaze position and the camera position, or the user talking while looking at text information displayed away from the display position of the conversation partner's face. In this case, the conversation partner may have a bad impression of the user. Therefore, a gaze correction technology has been developed for the online conference system that corrects the user's gaze so that it matches the gaze of the conversation partner.

[0003] However, with simple gaze correction technology, the gaze of the user and the conversation partner is always forcibly aligned through image processing, which may give the conversation partner an unnatural or intimidating impression.

[0004] Therefore, a communication terminal device has been devised that avoids unnatural eye contact between conference participants by changing the shooting position of the other user if it is determined that the user and the other user's gaze has met for a certain period of time or more during a remote conference (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2010-239583 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, when gaze averting processing is performed to shift the gaze of the user in the image of the user so as not to match the gaze of the conversation partner based only on the time that the gazes of the user and conversation partner are aligned, the gaze averting processing is not necessarily performed at a timing appropriate for the conversation partner. Therefore, the gaze averting processing may cause discomfort to the conversation partner.

[0007] Therefore, there is a demand for a method that can perform gaze averting processing in a way that gives positive feelings to a conversation partner, but such a demand has not yet been fully met.

[0008] The present technology has been made in consideration of such circumstances, and makes it possible to perform gaze averting processing in a way that gives positive feelings to a conversation partner. [Means for solving the problem]

[0009] A program or an information processing device according to one aspect of the present technology is a program for causing a computer to function as a gaze correction unit that performs gaze averting processing to shift the gaze of the user in an image of the user so as not to coincide with the gaze of the conversation partner, based on emotion-related information that is information related to the emotions of the user's conversation partner, or an information processing device that is equipped with the gaze correction unit.

[0010] An information processing method according to one aspect of the present technology is an information processing method that includes an information processing device performing gaze averting processing in which the gaze of the user in an image of the user is shifted so as not to coincide with the gaze of the conversation partner, based on emotion-related information that is information related to the emotions of the conversation partner of the user.

[0011] In one aspect of the present technology, a gaze averting process is performed in which the gaze of the user in the image of the user is shifted so as not to coincide with the gaze of the conversation partner, based on emotion-related information that is information related to the emotions of the conversation partner of the user. [Brief description of the drawings]

[0012] [Figure 1] 1 is a diagram showing an overview of an online conference system to which the present technology is applied. [Diagram 2] FIG. 2 is a block diagram showing an example of a hardware configuration of a cloud computer. [Diagram 3] 3 is a block diagram showing a configuration example of a conversation processing unit in the first embodiment. FIG. [Figure 4] FIG. 2 is a diagram showing a first example of a screen of a display in the first embodiment. [Diagram 5] FIG. 11 is a diagram showing a second example of the screen of the display in the first embodiment. [Figure 6] 4 is a flowchart illustrating a transmission process by the conversation processing unit in FIG. 3. [Figure 7] 7 is a flowchart illustrating the line-of-sight correction process in FIG. 6. [Figure 8] 4 is a flowchart illustrating a receiving process by the conversation processing unit in FIG. 3. [Figure 9] FIG. 13 is a block diagram showing an example of the configuration of a conversation processing unit in the second embodiment. [Figure 10] FIG. 11 is a diagram showing a first example of a screen of a display in the second embodiment. [Figure 11] 10 is a flowchart illustrating a receiving process by the conversation processing unit in FIG. 9. [Figure 12] FIG. 11 is a diagram showing a second example of a screen of the display in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, modes for carrying out the present technology (hereinafter, referred to as embodiments) will be described in the following order. 1. First embodiment (online conference system that performs gaze correction processing) 2. Second embodiment (online conference system that performs processing to encourage users to look away from the line of sight)

[0014] <1. First embodiment> <Overview of the online conference system> FIG. 1 is a diagram showing an overview of an online conference system having a cloud computer that is a first embodiment of an information processing device to which the present technology is applied.

[0015] 1, a PC (personal computer) 11-1 and a PC 11-2 are connected via a cloud computer 12. In the online conference system 10, a user 11-1a of the PC 11-1 and a user 11-2a of the PC 11-2 hold a conference, such as a conversation or a presentation, via a network such as the Internet.

[0016] PC 11-1 includes a camera 21-1, a display 22-1, a microphone 23-1, a speaker 24-1, etc. Camera 21-1 takes an image of user 11-1a and transmits it to cloud computer 12 via the network. Display 22-1 displays an image of user 11-2a transmitted from cloud computer 12. Microphone 23-1 captures the voice of user 11-1a and transmits it to cloud computer 12 via the network. Speaker 24-1 outputs the voice of user 11-2a transmitted from cloud computer 12.

[0017] PC 11-2 is configured similarly to PC 11-1. Specifically, PC 11-2 includes a camera 21-2, a display 22-2, a microphone 23-2, a speaker 24-2, and the like. The camera 21-2 takes an image of the user 11-2a and transmits it to the cloud computer 12 via the network. The display 22-2 displays the image of the user 11-1a transmitted from the cloud computer 12. The microphone 23-2 acquires the voice of the user 11-2a and transmits it to the cloud computer 12 via the network. The speaker 24-2 outputs the voice of the user 11-1a transmitted from the cloud computer 12.

[0018] Cloud computer 12 receives images and audio of user 11-1a transmitted from PC 11-1 and transmits them to PC 11-2. Cloud computer 12 receives images and audio of user 11-2a transmitted from PC 11-2 and transmits them to PC 11-1.

[0019] In the following, only the processing of the cloud computer 12 for PC 11-1 will be described, but the cloud computer 12 can perform the same processing for PC 11-2. The processing of the cloud computer 12 for PC 11-2 may be different from the processing for PC 11-1. Focusing on PC 11-1, since user 11-2a is a conference partner of user 11-1a of PC 11-1, in the following, user 11-2a will also be referred to as conference partner 11-2a (conversation partner) as appropriate.

[0020] <Example of cloud computer hardware configuration> FIG. 2 is a block diagram showing an example of the hardware configuration of the cloud computer 12 in FIG.

[0021] In the cloud computer 12, a central processing unit (CPU) 41, a read only memory (ROM) 42, and a random access memory (RAM) 43 are interconnected by a bus 44.

[0022] An input / output interface 45 is further connected to the bus 44. To the input / output interface 45, an input unit 46, an output unit 47, a storage unit 48, a communication unit 49, and a drive 50 are connected.

[0023] The input unit 46 includes a keyboard, a mouse, a microphone, etc. The output unit 47 includes a display, a speaker, etc. The storage unit 48 includes a hard disk, a non-volatile memory, etc. The communication unit 49 includes a network interface, etc. The drive 50 drives removable media 51 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0024] In cloud computer 12 configured as above, CPU 41 loads and executes a program stored in storage unit 48 into RAM 43 via input / output interface 45 and bus 44. As a result, when the operation mode is, for example, a conversation mode, CPU 41 functions as a conversation processing unit that performs conversation processing for transmitting an image and voice of user 11-1a to PC 11-2 and an image and voice of conference partner 11-2a to PC 11-1.

[0025] The conversation mode is a mode in which users 11-1a and 11-2a have a conversation. In addition to the conversation mode, the operation mode may be a presentation mode in which one of users 11-1a and 11-2b displays materials to the other and gives a presentation. The operation mode is set based on a signal transmitted from PC 11-1 (11-2) in response to an input operation by user 11-1a (11-2a) to PC 11-1 (11-2), for example.

[0026] The program executed by the CPU 41 can be provided by being recorded on a removable medium 51 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0027] In cloud computer 12, the program can be installed in storage unit 48 via input / output interface 45 by mounting removable medium 51 in drive 50. The program can also be received by communication unit 49 via a wired or wireless transmission medium and installed in storage unit 48. Alternatively, the program can be installed in ROM 42 or storage unit 48 in advance.

[0028] <Example of the conversation processing unit configuration> FIG. 3 is a block diagram showing an example of the configuration of the conversation processing unit.

[0029] 3 includes an image acquisition unit 71, a gaze detection unit 72, a direction detection unit 73, a voice acquisition unit 74, a determination unit 75, a gaze correction unit 76, and a transmission control unit 77. The conversation processing unit 70 also includes a partner image acquisition unit 78, a partner voice acquisition unit 79, an eye position detection unit 80, a generation unit 81, a display control unit 82, and an output control unit 83.

[0030] The image acquisition unit 71 acquires an image of the user 11-1a transmitted from the PC 11-1 via the communication unit 49 in FIG. 2, and supplies the image to the line of sight detection unit 72, the direction detection unit 73, and the line of sight correction unit .

[0031] The gaze detection unit 72 performs gaze analysis processing on the image supplied from the image acquisition unit 71, and detects a gaze position coordinate my_gaze_pos that indicates the horizontal and vertical positions of the gaze of the user 11-1a on the screen of the display 22-1. The gaze detection unit 72 supplies the gaze position coordinate my_gaze_pos to the gaze correction unit 76 and the display control unit 82.

[0032] The direction detection unit 73 detects the direction of the face of the user 11-1a with respect to the camera 21-1 based on the image supplied from the image acquisition unit 71, and supplies the result to the line of sight correction unit .

[0033] The voice acquisition unit 74 acquires the voice of the user 11-1 a transmitted from the PC 11-1 via the communication unit 49, and supplies it to the determination unit 75 and the transmission control unit 77.

[0034] The determination unit 75 determines whether the current time is the start or end of a conference based on the voice supplied from the voice acquisition unit 74. For example, the determination unit 75 determines whether the current time is the start or end of a conversation based on a change in the volume of the voice. The determination unit 75 can also perform a voice recognition process on the voice and determine whether the current time is the start or end of a conversation based on the context of the speech content obtained as a result. The determination unit 75 can also determine whether the current time is the start or end of a conversation based on an image of the user 11-1a. The determination unit 75 supplies the determination result to the gaze correction unit 76.

[0035] The gaze correction unit 76 performs gaze averting correction processing on the image of the user 11-1a based on the gaze position coordinate my_gaze_pos, the coordinate range partner_eye_pos, and the emotion flag partner_emotion. The coordinate range partner_eye_pos is a coordinate range indicating the position on the screen of the display 22-1 of the eye area in the image of the conference partner 11-2a displayed on the display 22-1, which is supplied from the eye position detection unit 80. The emotion flag partner_emotion is information supplied from the generation unit 81 indicating whether the emotion of the conference partner 11-2a is positive or negative.

[0036] The gaze averting correction process is an image process that corrects the image of the user 11-1a so that the gaze vector of the user 11-1a in the image of the user 11-1a does not match the gaze vector of the conference partner 11-2a. The gaze vector of the conference partner 11-2a is estimated by machine learning processing from information such as the image of the conference partner 11-2a acquired by the partner image acquisition unit 78 described later and the physical positional relationship between the conference partner 11-2a and the camera 21-2. The gaze averting correction process may be an image process that corrects the gaze vector of the user 11-1a in the image of the user 11-a to a predetermined direction such as a downward direction so that the gaze vector of the user 11-1a is less likely to match the gaze vector of the conference partner 11-2a. In this case, the processing load of the gaze averting correction process can be reduced. The correction of the image of the user 11-1a in the gaze averting correction process is performed by correcting the image near the eyes in the image of the user 11-1a using a rule base or machine learning model after the correction direction of the gaze vector of the user 11-1a is determined.

[0037] The line-of-sight correction unit 76 performs line-of-sight correction processing, thereby realizing line-of-sight averting processing for shifting the line of sight of the user 11-1a in the image of the user 11-1a so as not to coincide with the line of sight of the conference partner 11-2a.

[0038] The line of sight correction unit 76 supplies the image of the user 11-1a for which the line of sight correction process has been performed to the transmission control unit 77. The line of sight correction unit 76 notifies the display control unit 82 of line of sight correction information indicating that the line of sight correction process has been performed.

[0039] The gaze correction unit 76 performs gaze matching processing based on the face direction of the user 11-1a, the gaze position coordinate my_gaze_pos, the determination result by the determination unit 75, the coordinate range partner_eye_pos, and the emotion flag partner_emotion. The gaze matching processing is image processing that corrects the image of the user 11-1a so that the gaze vector of the user 11-1a in the image of the user 11-1a matches the gaze vector of the conference partner 11-2a. The gaze vector of the conference partner 11-2a is estimated, for example, in the same manner as in the gaze deviation correction processing. The gaze matching processing may be processing that corrects the gaze vector of the user 11-1a in the image of the user 11-a to a predetermined direction such as the front direction so that the gaze vector of the user 11-1a roughly matches the gaze vector of the conference partner 11-2a. In this case, the processing load of the gaze matching processing can be reduced. The gaze correction unit 76 supplies the image of the user 11-1a on which the gaze matching processing has been performed to the transmission control unit 77.

[0040] The transmission control unit 77 controls the communication unit 49 to transmit the image supplied from the line of sight correction unit 76 and the sound supplied from the sound acquisition unit 74 to the PC 11-2.

[0041] The other party image acquisition unit 78 acquires an image of the conference partner 11-2a transmitted from the PC 11-2 via the communication unit 49, and supplies it to the eye position detection unit 80, the generation unit 81, and the display control unit 82. The other party voice acquisition unit 79 acquires the voice of the conference partner 11-2a transmitted from the PC 11-2 via the communication unit 49, and supplies it to the generation unit 81 and the output control unit 83.

[0042] The eye position detection unit 80 detects the coordinate range partner_eye_pos by performing image recognition processing or the like on the image supplied from the partner image acquisition unit 78, and supplies it to the line of sight correction unit .

[0043] The generation unit 81 performs an analysisEmotion process that estimates the emotion (partner_emotion) of the conference partner 11-2a based on at least one of an image and a voice of the conference partner 11-2a using AI (Artificial Intelligence) technology. In this way, the generation unit 81 generates emotion-related information that is information related to the emotion of the conference partner 11-2a. Note that the emotion-related information is, for example, information related to at least one of discomfort, tension, excitement, intimacy, and relief of the conference partner 11-2a.

[0044] As the input for the analysisEmotion process, for example, the analysis result of the facial expression in the image of the conference partner 11-2a, the analysis result of the speech content of the conference partner 11-2a, the detection result of the silence period, etc. can be used. In general, the estimation accuracy can be improved by increasing the number of inputs for the analysisEmotion process. Part of the input for the analysisEmotion process can be a default value instead of information obtained in real time from the image and voice of the conference partner 11-2a. In this case, the amount of calculation in the analysisEmotion process can be reduced.

[0045] The generation unit 81 generates the emotion flag partner_emotion based on the emotion-related information. For example, when the generation unit 81 determines based on the emotion-related information that the conference partner 11-2a feels a high level of discomfort or tension, the generation unit 81 sets the emotion flag partner_emotion to negative, which indicates a negative emotion. On the other hand, when the generation unit 81 determines based on the emotion-related information that the conference partner 11-2a feels a high level of excitement, intimacy, or security, the generation unit 81 sets the emotion flag partner_emotion to positive, which indicates a positive emotion. The generation unit 81 supplies the emotion flag partner_emotion to the gaze correction unit 76.

[0046] The display control unit 82 reads out manuscript data of the speech of the user 11-1a, such as a manual, stored in the storage unit 48 in Fig. 2, and converts it into character information. The display control unit 82 superimposes the manuscript information, which is the character information of the manuscript data, on the image of the conference partner 11-2a, based on the gaze position coordinate my_gaze_pos. Note that the manuscript data can be transmitted from the PC 11-1 to the cloud computer 12 and stored in the storage unit 48, for example.

[0047] The display control unit 82 (notification unit) generates gaze averting notification information, which is text information for notifying the user that gaze averting correction processing is being performed, based on the gaze averting correction information supplied from the gaze correction unit 76. The display control unit 82 further superimposes the gaze averting notification information on the image of the conference partner 11-2a on which the manuscript information is superimposed, based on the gaze position coordinate my_gaze_pos.

[0048] The display control unit 82 controls the communication unit 49 to transmit an image of the conference partner 11-2a, on which manuscript information and, if necessary, gaze averting notification information are superimposed, to the PC 11-1, thereby causing the image to be displayed on the display 22-1 (display unit).

[0049] The color and transparency of text information such as manuscript information and gaze averting notification information are adjusted in real time according to the image of the conference partner 11-2a on which the text information is superimposed. This allows the user 11-1a to talk while comfortably viewing both the image and text information of the conference partner 11-2a. As a result, the user 11-1a can concentrate on the conversation.

[0050] The output control unit 83 controls the communication unit 49 to transmit the voice supplied from the other party image acquisition unit 78 to the PC 11-1, thereby outputting the voice from the speaker 24-1.

[0051] <First example of the screen> FIG. 4 is a diagram showing an example of a screen of display 22-1 when the operation mode is the conversation mode and the gaze averting correction process is not being performed.

[0052] In this case, on the screen 100 of the display 22-1, manuscript information 101 is displayed superimposed near position 100a based on the gaze position coordinate my_gaze_pos of position 100a on the screen 100 of the gaze of the user 11-1a.

[0053] 4, the position 100a is located near the face of the conference partner 11-2a in the image. Therefore, the manuscript information 101 is displayed superimposed on the face of the conference partner 11-2a in the image. This allows the user 11-1a to talk to the conference partner 11-2a while looking at the face of the conference partner 11-2a in the image and the manuscript information 101.

[0054] As described above, the manuscript information 101 is superimposed near the position 100a based on the gaze position coordinate my_gaze_pos, so the user 11-1a does not need to move his / her gaze to see the manuscript information 101. Therefore, the user 11-1a can place his / her gaze at a desired position. As a result, for example, the user 11-1a can reliably match his / her gaze with the conference partner 11-2a.

[0055] <Second example of the screen> FIG. 5 is a diagram showing an example of a screen of the display 22-1 when the gaze deviation correction process is being performed.

[0056] In screen 120 in Fig. 5, the same reference numerals are used for the parts corresponding to screen 100 in Fig. 4. Therefore, the description of those parts will be omitted as appropriate, and the description will focus on the parts different from screen 100. Screen 120 differs from screen 100 in that line-of-sight notification information 121 is further superimposed on conference partner 11-2a, and is otherwise configured in the same way as screen 100.

[0057] Specifically, on the screen 120, the gaze averting notification information 121 is displayed superimposed near the position 100a of the image of the conference partner 11-2a based on the gaze position coordinate my_gaze_pos of the position 100a. In the example of FIG. 5, the gaze averting notification information 121 is displayed superimposed on the upper part of the manuscript information 101 near the position 100a. The gaze averting notification information 121 is composed of, for example, text information "Negative!!" notifying the emotion flag partner_emotion as the reason why the gaze averting correction process has been performed, and text information "Averting your gaze" notifying that the gaze averting correction process is being performed. The emotion flag partner_emotion is supplied from, for example, the generation unit 81 to the display control unit 82. The gaze averting notification information 121 may be displayed in a highlighted manner.

[0058] <Description of sending process> Fig. 6 is a flowchart for explaining a transmission process for transmitting an image and voice of the user 11-1a to the PC 11-2 by the conversation processing unit 70 in Fig. 3. This transmission process is started, for example, when one frame of the image and voice of the user 11-1a is transmitted from the PC 11-1.

[0059] In step S11 of FIG. 6, image acquisition unit 71 acquires one frame of an image (my_image) of user 11-1a transmitted from PC 11-1, and supplies it to gaze detection unit 72, direction detection unit 73, and gaze correction unit .

[0060] In step S12, the audio acquisition unit 74 acquires audio corresponding to one frame of image transmitted from the PC 11-1, and supplies the audio to the determination unit 75 and the transmission control unit 77.

[0061] In step S13, the gaze detection unit 72 detects the gaze position coordinate my_gaze_pos based on the image acquired in the process of step S11, and supplies it to the gaze correction unit 76 and the display control unit .

[0062] In step S14, the direction detection unit 73 detects the direction of the face of the user 11-1a based on the image acquired by the processing in step S11, and supplies the result to the line of sight correction unit .

[0063] In step S15, the line-of-sight correction unit 76 performs line-of-sight correction processing to correct the line of sight of the user 11-1a in the image as necessary (correctEyeContact). Details of this line-of-sight correction processing will be described later with reference to FIG. 7. After the processing of step S15, the processing proceeds to step S16.

[0064] In step S16, the transmission control unit 77 transmits the image supplied from the line-of-sight correction unit 76 and the sound acquired in the process of step S12 to the PC 11-2, and then ends the process.

[0065] The transmission process in FIG. 6 may be performed in units of multiple frames.

[0066] <Explanation of gaze correction process> FIG. 7 is a flowchart illustrating the line-of-sight correction process in step S15 of FIG.

[0067] In step S31 of Fig. 7, the determination unit 75 determines whether or not the current time is the start or end of a conversation based on the voice acquired by the process of step S12 of Fig. 6. If it is determined in step S31 that the current time is not the start or end of the conversation, the determination unit 75 supplies the determination result that the current time is not the start or end of the conversation to the gaze correction unit 76, and the process proceeds to step S32.

[0068] In step S32, the gaze correction unit 76 initializes needGazeShift, which indicates whether or not gaze averting correction processing is to be performed, by setting it to False, which indicates that gaze averting correction processing is not to be performed.

[0069] In step S33, the line-of-sight correction unit 76 performs checkMyGazePos processing based on the line-of-sight position coordinate my_gaze_pos and the coordinate range partner_eye_pos detected in step S13. The checkMyGazePos processing is processing for determining whether the user 11-1a is attempting to make eye contact with the conference partner 11-2a and generating needUpdateFrame representing the determination result.

[0070] For example, when the gaze position coordinate my_gaze_pos is within the coordinate range partner_eye_pos, i.e., when the user 11-1a is looking at the vicinity of the eyes of the conference partner 11-2a, the gaze correction unit 76 sets needUpdateFrame to True, which indicates that gaze alignment is being attempted. On the other hand, when the gaze position coordinate my_gaze_pos is outside the coordinate range partner_eye_pos, i.e., when the user 11-1a is not looking at the vicinity of the eyes of the conference partner 11-2a, the gaze correction unit 76 sets needUpdateFrame to False, which indicates that gaze alignment is not being attempted.

[0071] In step S34, the line-of-sight correction unit 76 determines whether or not needUpdateFrame is True. If it is determined in step S34 that needUpdateFrame is True, the process proceeds to step S35.

[0072] In step S35, the line-of-sight correction unit 76 increments by one the t_contact, which indicates the time during which the lines of sight of the user 11-1a and the conference partner 11-2a continuously match.

[0073] In step S36, the gaze correction unit 76 determines whether or not t_contact is greater than a threshold T_TH_CONTACT and the emotion flag partner_emotion indicates a negative emotion. The threshold T_TH_CONTACT is a value corresponding to a time range from 1 second to 3 seconds, for example.

[0074] If it is determined in step S36 that t_contact is greater than the threshold T_TH_CONTACT and the emotion flag partner_emotion indicates a negative emotion, the process proceeds to step S37. That is, if the time during which the gaze of the user 11-1a and the gaze of the conference partner 11-2a coincide exceeds a predetermined time and it is estimated that the conference partner 11-2a is feeling a negative emotion, the process proceeds to step S37.

[0075] In step S37, the gaze correction unit 76 judges whether or not the rand_value exceeds the threshold th_gaze_shift. The rand_value is a random number in the range of 0.0 to 1.0 that is generated to determine whether or not gaze averting correction processing is performed. Thus, the rand_value exceeds the threshold th_gaze_shift randomly, but the lower the threshold th_gaze_shift is, the higher the rate at which the rand_value exceeds the threshold th_gaze_shift, i.e., the rate at which gaze averting correction processing is performed in step S39 described later. Thus, the threshold th_gaze_shift is a value in the range of 0.0 to 1.0 that determines the rate at which gaze averting correction processing is performed randomly. For example, when the threshold th_gaze_shift is 0.7, the rate at which gaze averting correction processing is performed is 30%.

[0076] This threshold th_gaze_shift is calculated (calcGazeShiftThreshold) based on, for example, at least one of the characteristics of the user 11-1a, the characteristics of the conference partner 11-2a, and the state of the conversation between the user 11-1a and the conference partner 11-2a.

[0077] For example, the threshold value th_gaze_shift is set based on the eye power strength eye_power as a characteristic of the user 11-1a so that the stronger the eye power strength eye_power, the smaller the threshold value th_gaze_shift. This allows the rate at which the gaze deviation correction process is performed to be increased as the eye power strength eye_power of the user 11-1a increases. A user 11-1a with strong eye power strength eye_power is more likely to give the conference partner 11-2a a sense of intimidation by matching their gaze with the conference partner 11-2a than a user 11-1a with weak eye power strength eye_power. Therefore, by setting the threshold value th_gaze_shift based on the eye power strength eye_power as described above, it is possible to prevent the conference partner 11-2a from feeling negative emotions.

[0078] The eye power strength "eye_power" is calculated, for example, from the aspect ratio of the eyes in the image of the user 11-1a.

[0079] The threshold th_gaze_shift can also be set based on isListner, which indicates whether the situation of the conversation between the user 11-1a and the conference partner 11-2a is a situation in which the user 11-1a is the listener. In this case, for example, when isListner indicates that the user 11-1a is the listener, the threshold th_gaze_shift is set to be larger than when isListner indicates that the user 11-1a is not the listener. This can reduce the rate at which the gaze deviation correction process is performed when the user 11-1a is the listener. Psychologically, it is said that a listener can have a better conversation by focusing his or her gaze on the conversation partner than the speaker. Therefore, by setting the threshold th_gaze_shift based on isListner as described above, it is possible to give a positive feeling to the conference partner 11-2a.

[0080] The threshold th_gaze_shift can also be set based on the nationality as a characteristic of the conference partner 11-2a. In this case, for example, when the conference partner 11-2a is of a country where eye contact is not welcomed, the threshold th_gaze_shift is set to be smaller than when the conference partner 11-2a is of a country where eye contact is welcomed. This makes it possible to increase the rate at which gaze averting correction processing is performed when the conference partner 11-2a is of a country where eye contact is not welcomed. As a result, it is possible to prevent the conference partner 11-2a from feeling negative emotions.

[0081] The threshold th_gaze_shift can also be set based on a combination of genders as characteristics of the user 11-1a and the conference partner 11-2a. In this case, for example, when the genders of the user 11-1a and the conference partner 11-2a are different, the threshold th_gaze_shift is set to be smaller than when they are the same. This can increase the rate at which the gaze deviation correction process is performed when the genders of the user 11-1a and the conference partner 11-2a are different. As a result, it is possible to prevent the conference partner 11-2a from misunderstanding that the user 11-1a has romantic feelings due to eye contact in which the gaze of the user 11-1a matches the gaze of the conference partner 11-2a.

[0082] The parameters used in the calculation of the threshold th_gaze_shift are not limited to the characteristics of the user 11-1a and the conference partner 11-2a, the state of the conversation between the user 11-1a and the conference partner 11-2a, and the like.

[0083] If it is determined in step S37 that rand_value exceeds the threshold th_gaze_shift, the process proceeds to step S38. In step S38, the gaze correction unit 76 sets needGazeShift to True, which indicates that gaze averting correction processing is to be performed.

[0084] In step S39, the line-of-sight correction unit 76 performs line-of-sight correction processing on the image acquired in the processing of step S11 in FIG. 6, based on the needGazeShift set in the processing of step S38.

[0085] As described above, the gaze correction unit 76 determines whether or not to perform gaze averting correction processing depending on whether or not the rand_value exceeds the threshold th_gaze_shift. Therefore, when the rand_value is updated for each frame, the execution or non-execution of the gaze averting correction processing, i.e., the presence or absence of a mismatch in the gaze between the user 11-1a and the conference partner 11-2a, switches for each frame, and the gaze of the user 11-1a in the image may become unnatural. Therefore, it is desirable to update the rand_value every several frames.

[0086] The line-of-sight correction unit 76 supplies the image that has been subjected to the line-of-sight correction process to the transmission control unit 77 as an image (new_my_image) of the user 11-1a to be transmitted, and notifies the display control unit 82 of the line-of-sight correction information. Then, the process returns to step S15 in FIG. 6, and proceeds to step S16.

[0087] On the other hand, if it is determined in step S34 that needUpdateFrame is not True, that is, if needUpdateFrame is False, the process proceeds to step S40. In step S40, the line of sight correction unit 76 updates t_contact to the larger of 0 and a value obtained by subtracting 1 from the current t_contact. Then, the line of sight correction unit 76 supplies the image acquired by the process of step S11 in Fig. 6 to the transmission control unit 77 as is, and the process returns to step S15 and proceeds to step S16.

[0088] As described above, when needUpdateFrame is False, that is, when the user 11-1a is not attempting to make eye contact with the conference partner 11-2a, the eye-avoidance correction process and the eye-contact process are not performed. Therefore, the user 11-1a can make natural eye contact with the conference partner 11-2a, just like during an offline conversation.

[0089] On the other hand, if it is determined in step S31 that it is the start or end of the conversation, the determination unit 75 supplies the determination result that it is the start or end of the conversation to the gaze correction unit 76, and the process proceeds to step S41. That is, if the situation of the conversation between the user 11-1a and the conference partner 11-2a is such that the user 11-1a is likely to give the conference partner 11-2a a positive feeling by matching gaze with the conference partner 11-2a, the process proceeds to step S41.

[0090] If it is determined in step S36 that t_contact is not greater than the threshold T_TH_CONTACT or that the emotion flag partner_emotion does not represent a negative emotion, the gaze correction unit 76 determines that it is necessary to perform gaze matching processing. That is, if the time during which the gaze of the user 11-1a and the gaze of the conference partner 11-2a match does not exceed a predetermined time, or if it is estimated that the conference partner 11-2a does not have a negative emotion, the gaze correction unit 76 determines that it is necessary to perform gaze matching processing. Then, the process proceeds to step S41.

[0091] If it is determined in step S37 that rand_value does not exceed the threshold th_gaze_shift, that is, if rand_value is equal to or smaller than the threshold th_gaze_shift, the gaze correction unit 76 determines that gaze matching processing needs to be performed, and the process proceeds to step S41.

[0092] In step S41, the line of sight correction unit 76 determines whether or not the face direction detected in the process of step S14 in Fig. 6 is facing forward. If it is determined in step S41 that the face direction is facing forward, the process proceeds to step S42.

[0093] In step S42, the gaze correction unit 76 performs gaze matching processing on the image acquired in the processing of step S11 in Fig. 6 based on the needGazeShift set in the processing of step S32. The gaze correction unit 76 supplies the image of the user 11-1a for which the gaze matching processing has been performed to the transmission control unit 77 as an image of the user 11-1a to be transmitted. Then, the processing returns to step S15 in Fig. 6 and proceeds to step S16.

[0094] If it is determined in step S41 that the face direction is not facing forward, the process of step S42 is skipped. Then, the line of sight correction unit 76 supplies the image acquired in the process of step S11 in Fig. 6 to the transmission control unit 77 as it is, and the process returns to step S15, and proceeds to step S16.

[0095] As described above, when the face of the user 11-1a is not facing forward, the eye contact process is not performed. Therefore, for example, when the direction of the camera 21-1 and the direction of the display 22-1 with respect to the user 11-1a are different and the eye contact between the user 11-1a and the conference partner 11-2a does not need to match, the eye contact process can be prevented from being performed.

[0096] <Description of reception process> Fig. 8 is a flowchart for explaining a receiving process for receiving an image and a voice of the conference party 11-2a by the conversation processing unit 70 in Fig. 3. This receiving process is started, for example, when one frame of an image and a voice of the conference party 11-2a are transmitted from the PC 11-2.

[0097] In step S50 in FIG. 8, the partner image acquisition unit 78 acquires one frame of an image (partner_img) of the conference partner 11-2a transmitted from the PC 11-2, and supplies it to the eye position detection unit 80, the generation unit 81, and the display control unit .

[0098] In step S51, the partner audio acquisition unit 79 acquires audio (partner_audio) corresponding to one frame of an image of the conference partner 11-2a transmitted from the PC 11-2, and supplies it to the generation unit 81 and the output control unit 83.

[0099] In step S52, the eye position detection unit 80 detects the coordinate range partner_eye_pos based on the image acquired by the processing in step S50, and supplies it to the line of sight correction unit .

[0100] In step S53, the generation unit 81 generates an emotion flag partner_emotion based on at least one of the image acquired in the process of step S50 and the sound acquired in the process of step S51, and supplies the emotion flag partner_emotion to the gaze correction unit .

[0101] In step S54, the display control unit 82 reads out manuscript data of the speech of the user 11-1a stored in the storage unit 48 or the like, and generates manuscript information.

[0102] In step S55, the display control unit 82 determines whether or not gaze averting correction information has been supplied from the gaze correction unit 76. If it is determined in step S55 that gaze averting correction information has been supplied, the process proceeds to step S56.

[0103] In step S56, the display control unit 82 generates gaze averting notification information. In step S57, the display control unit 82 superimposes the manuscript information and gaze averting notification information on the image acquired by the process of step S50 based on the gaze position coordinate my_gaze_pos, and transmits the superimposed information to the PC 11-1 to display the information on the display 22-1. Then, the process proceeds to step S59.

[0104] On the other hand, if it is determined in step S55 that the gaze deviation correction information has not been provided, the process proceeds to step S58. In step S58, the display control unit 82 superimposes the document information on the image acquired in the process of step S50 based on the gaze position coordinate my_gaze_pos, and transmits the superimposed document information to the PC 11-1 to display the superimposed document information on the display 22-1. Then, the process proceeds to step S59.

[0105] In step S59, the output control unit 83 transmits the sound acquired in the process of step S51 to the PC 11-1, thereby outputting the sound from the speaker 24-1, and the reception process then ends.

[0106] It should be noted that the display control unit 82 does not have to superimpose the document information on the image of the conference partner 11-2a.

[0107] As described above, the line of sight correction unit 76 performs the line of sight correction process based on the emotion-related information to perform the line of sight averting process. Therefore, for example, the line of sight correction unit 76 can perform the line of sight averting process when the emotion flag partner_emotion generated based on the emotion-related information indicates a negative emotion. This allows the line of sight averting process to give the conference partner 11-2a a positive emotion.

[0108] As a result, it is possible to alleviate the discomfort such as the unnatural impression and intimidation felt by the conference partner 11-2a due to excessive eye contact with the user 11-1a. Also, for example, when the conference partner 11-2a is feeling nervous, it is possible to prevent the conference partner 11-2a's feeling of tension from increasing further by the conference partner 11-2a's eye contact with the user 11-1a's. Furthermore, the user 11-1a no longer needs to worry about the emotions that his or her eye contact may cause the conference partner 11-2a, and can concentrate on the conversation.

[0109] The gaze correction unit 76 also performs gaze matching processing based on emotion-related information. Therefore, for example, the gaze correction unit 76 can perform gaze matching processing when the emotion flag partner_emotion generated based on emotion-related information indicates a positive emotion. This allows the conference partner 11-2a to have a conversation with the user 11-1a while their gazes are matched, when the conference partner 11-2a feels a positive emotion as a result of their gazes matching. As a result, the user 11-1a can give the conference partner 11-2a a sense of excitement, security, intimacy, satisfaction, and the like.

[0110] Cloud computer 12 functions as conversation processor 70 and performs gaze correction processing when the operation mode is conversation mode, i.e., when user 11-1a needs to make eye contact with conference partner 11-2a. Therefore, when the operation mode is presentation mode, i.e., when user 11-1a is displaying materials and does not need to make eye contact with conference partner 11-2a, gaze correction processing is not performed. As a result, the load of gaze correction processing on cloud computer 12 can be reduced.

[0111] <2. Second embodiment> <Example of the conversation processing unit configuration> An online conference system having a cloud computer, which is a second embodiment of an information processing device to which the present technology is applied, differs from the online conference system 10 in Fig. 1 in that a gaze-avoidance promotion process is performed in place of a gaze-avoidance correction process in a conversation processing unit. The rest of the configuration is the same as that of the online conference system 10. Therefore, the following description will focus on the conversation processing unit.

[0112] FIG. 9 is a block diagram showing an example of the configuration of the conversation processing unit.

[0113] In the conversation processing unit 270 in Fig. 9, parts corresponding to the conversation processing unit 70 in Fig. 3 are given the same reference numerals. Therefore, explanations of those parts will be omitted as appropriate, and the following explanation will focus on the parts that differ from the conversation processing unit 70. The conversation processing unit 270 differs from the conversation processing unit 70 in that it includes a gaze correction unit 276 and a display control unit 282 instead of the gaze correction unit 76 and the display control unit 82, but is otherwise configured in the same way as the conversation processing unit 70.

[0114] The gaze correction unit 276 performs gaze averting promotion processing based on the gaze position coordinate my_gaze_pos, the coordinate range partner_eye_pos, and the emotion flag partner_emotion. The gaze averting promotion processing is processing for encouraging the user 11-1a to change his / her gaze so that the gaze vector of the user 11-1a in the image of the user 11-1a does not match the gaze vector of the conference partner 11-2a. The gaze vector of the conference partner 11-2a is estimated, for example, in the same manner as in the gaze averting correction processing. The gaze averting promotion processing may be processing for encouraging the user 11-1a to move his / her gaze in a predetermined direction, such as downward, so that the gaze vector of the user 11-1a is less likely to match the gaze vector of the conference partner 11-2a. In this case, the processing load of the gaze averting promotion processing can be reduced.

[0115] The process of prompting the user to change his / her line of sight is performed by determining the direction of change of the line of sight, generating position coordinates on the screen corresponding to the changed line of sight as display position coordinates, and supplying them to the display control unit 282. The display position coordinates may be set to be position coordinates outside the coordinate range partner_eye_pos. In this case, the user can be prompted to set the line of sight position coordinate my_gaze_pos outside the coordinate range partner_eye_pos. The display control unit 282 changes the display position of the manuscript information based on the display position coordinates. As a result, the user 11-1a changes his / her line of sight. Therefore, the line of sight correction unit 276 can realize the line of sight averting process by performing the line of sight averting promotion process.

[0116] The gaze correction unit 276 performs gaze matching processing on the image of the user 11-1a in the same manner as the gaze correction unit 76 in Fig. 3. The gaze correction unit 276 supplies the image of the user 11-1a for which the gaze matching processing has been performed or the image of the user 11-1a supplied from the image acquisition unit 71 to the transmission control unit 77.

[0117] The display control unit 282 generates manuscript information in the same manner as the display control unit 82 in Fig. 3. The display control unit 282 superimposes the manuscript information on the image of the conference party 11-2a based on the gaze position coordinate my_gaze_pos or the display position coordinate. The display control unit 282 transmits the image of the conference party 11-2a on which the manuscript information is superimposed to the PC 11-1, thereby causing it to be displayed on the display 22-1. The color and transmittance of the manuscript information are adjusted in real time according to the image of the conference party 11-2a on which the manuscript information is superimposed.

[0118] <First example of the screen> FIG. 10 is a diagram showing an example of a screen of the display 22-1 when the gaze averting prompting process is being performed.

[0119] In screen 300 in Fig. 10, parts corresponding to screen 100 in Fig. 4 are given the same reference numerals. Therefore, the description of those parts will be omitted as appropriate, and the description will focus on the parts that differ from screen 100. Screen 300 differs from screen 100 in that manuscript information 301 is superimposed instead of manuscript information 101, and is otherwise configured in the same way as screen 100.

[0120] On screen 300, manuscript information 301 is displayed superimposed below the vicinity of position 100a based on the display position coordinates. As a result, user 11-1a moves his / her line of sight from position 100a to the display position of manuscript information 301. As a result, the line of sight of user 11-1a does not match the line of sight of conference partner 11-2a.

[0121] The transmission process of conversation processing unit 270 in Fig. 9 differs from the transmission process of Fig. 6 in the content of the gaze correction process in step S15 in Fig. 6, but is otherwise similar to the transmission process of Fig. 6. Specifically, the gaze correction process of conversation processing unit 270 differs from the gaze correction process of Fig. 7 in that gaze averting promotion processing is performed instead of gaze averting correction processing in step S39 in Fig. 7, and the image supplied from image acquisition unit 71 is supplied directly to transmission control unit 77. The rest is similar to the gaze correction process of Fig. 7. Therefore, a flowchart explaining the transmission process and gaze correction processing of conversation processing unit 270 will be omitted.

[0122] <Description of reception process> Fig. 11 is a flowchart for explaining the receiving process by the conversation processing unit 270 of Fig. 9. This receiving process is started, for example, when one frame of image and audio of the conference partner 11-2a is transmitted from the PC 11-2.

[0123] The processing in steps S150 to S154 in FIG. 11 is similar to the processing in steps S50 to S54 in FIG. 8, and therefore a description thereof will be omitted.

[0124] In step S155, the display control unit 282 determines whether or not display position coordinates have been supplied from the line of sight correction unit 276. If it is determined in step S155 that display position coordinates have been supplied, the process proceeds to step S156.

[0125] In step S156, the display control unit 282 superimposes the document information on the image acquired in the process of step S150 based on the display position coordinates, and transmits the superimposed document information to the PC 11-1 to display the superimposed document information on the display 22-1. Then, the process proceeds to step S158.

[0126] On the other hand, if it is determined in step S155 that the display position coordinates have not been provided, the process proceeds to step S157. In step S157, the display control unit 282 superimposes the manuscript information on the image acquired by the process of step S150 based on the gaze position coordinate my_gaze_pos, and transmits the superimposed image to the PC 11-1 to display the superimposed image on the display 22-1. Then, the process proceeds to step S158.

[0127] In step S158, the output control unit 83 transmits the sound acquired in the process of step S51 to the PC 11-1, thereby outputting the sound from the speaker 24-1, and the reception process then ends.

[0128] The display control unit 282 does not have to superimpose the manuscript information on the image of the conference partner 11-2a. In this case, the display control unit 282 superimposes the gaze averting promotion related information, which is text information related to the gaze averting promotion process, on the image of the conference partner 11-2a based on the display position coordinates.

[0129] <Second example of the screen> FIG. 12 is a diagram showing an example of the screen of the display 22-1 on which an image of the conference partner 11-2a with gaze averting promotion related information superimposed thereon is displayed.

[0130] In screen 320 in Fig. 12, the parts corresponding to screen 300 in Fig. 10 are given the same reference numerals. Therefore, the description of those parts will be omitted as appropriate, and the description will focus on the parts that are different from screen 300. Screen 320 differs from screen 300 in that gaze averting promotion related information 321 is superimposed instead of manuscript information 301, and is otherwise configured in the same way as screen 300.

[0131] On the screen 320, the gaze averting prompt related information 321 is displayed superimposed below the vicinity of the position 100a based on the display position coordinates. As a result, the user 11-1a moves his / her gaze from the position 100a to the display position of the gaze averting prompt related information 321. As a result, the gaze of the user 11-1a does not match the gaze of the conference partner 11-2a.

[0132] 12, the gaze averting promotion related information 321 is text information "Negative!!" that indicates the emotion flag partner_emotion as the reason for performing the gaze averting promotion process. The gaze averting promotion related information 321 may be displayed in a highlighted manner.

[0133] The display control unit 282 may also superimpose the gaze averting promotion related information when superimposing the manuscript information on the image of the conference partner 11-2a.

[0134] For example, when the gaze position coordinate my_gaze_pos does not move to the vicinity of the display position coordinate even after a predetermined time has elapsed, the gaze correction unit 276 may instruct the display control unit 282 to change the display color of the manuscript information and the gaze averting promotion related information, etc. This changes the display color of the manuscript information and the gaze averting promotion related information, thereby further encouraging the user 11-1a to move his / her gaze.

[0135] As described above, the line-of-sight correction unit 276 performs the line-of-sight correction process by performing the line-of-sight correction process based on the emotion-related information. Therefore, as in the first embodiment, the line-of-sight correction process can be performed so as to give the conference partner 11-2a a positive emotion.

[0136] The PC 11-1 may selectively perform a line-of-sight correction process and a line-of-sight promotion process as the line-of-sight averting process.

[0137] In this case, the user 11-1a may select either the gaze averting correction process or the gaze averting promotion process as the gaze averting process. As a result, for example, the user 11-1a who becomes nervous or impatient when the gaze averting promotion process is executed can select the gaze averting correction process as the gaze averting process. On the other hand, for example, the user 11-1a who wants to shift the gaze at his / her own timing can select the gaze averting promotion process as the gaze averting process.

[0138] The selection of the gaze averting correction process and the gaze averting promotion process may be performed based on the emotion flag partner_emotion. In this case, for example, if the emotion flag partner_emotion indicates a positive emotion for a predetermined period of time or more during the selection of the gaze averting promotion process, it is determined that the gaze of the user 11-1a is appropriately guided. Then, the selection of the gaze averting promotion process is continued.

[0139] The program executed by PC11-1 may be a program that is processed chronologically in the order described in this specification, or it may be a program that is processed in parallel or at the required timing, such as when called.

[0140] The present technology can be applied to an online conference system in which two or more users hold a conference.

[0141] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0142] For example, it is possible to adopt a configuration in which all or part of the above-described multiple embodiments are combined.

[0143] For example, some functions of the conversation processing unit 70 (270) may be provided in the PCs 11-1 and 11-2. For example, the display control unit 82 (282) of the conversation processing unit 70 (270) may transmit document information, gaze averting notification information, gaze averting promotion related information, etc. to the PC 11-1 separately from the image of the conference partner 11-2a, and the PC 11-1 may superimpose them on the image of the conference partner 11-2a.

[0144] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0145] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.

[0146] It should be noted that the effects described in this specification are merely examples and are not limiting, and there may be effects other than those described in this specification.

[0147] The present technology can take the following configurations. (1) Computer, A gaze correction unit that performs gaze averting processing to shift the gaze of the user in the image of the user so as not to coincide with the gaze of the conversation partner based on emotion-related information that is information related to the emotion of the conversation partner of the user. A program to function as a (2) The emotion-related information is information related to at least one of the following: discomfort, tension, excitement, intimacy, and comfort of the conversation partner. The program according to (1) above. (3) The computer, a notification unit that notifies the user that the line-of-sight correction unit is performing the line-of-sight correction process The program according to (1) or (2) above for further functioning as a (4) The notification unit also notifies the user of information representing the emotion of the conversation partner based on the emotion-related information. The program according to (3) above. (5) The line of sight correction unit performs the line of sight averting process based on the emotion-related information when the operation mode is a conversation mode in which conversation is performed. The program according to any one of (1) to (4). (6) The line of sight correction unit performs the line of sight averting process based also on a time during which the line of sight of the user and the line of sight of the conversation partner are aligned. The program according to any one of (1) to (5). (7) The line of sight correction unit performs the line of sight averting process at a ratio based on at least one of a characteristic of the user, a characteristic of the conversation partner, and a situation of the conversation between the user and the conversation partner. The program according to any one of (1) to (6). (8) The gaze averting process is a process of changing the vertical position of the user's gaze within the image of the user. The program according to any one of (1) to (7). (9) The gaze averting process is a process for prompting the user to change the vertical position of the user's gaze. The program according to any one of (1) to (7). (10) The computer, A display control unit that displays text information on a display unit by superimposing the text information on the image of the conversation partner. It also functions as The gaze averting process is a process of prompting the user to change the vertical position of the user's gaze by changing the vertical display position of the character information. The program according to (9) above. (11) The line of sight correction unit selectively performs, as the line of sight averting process, a process of changing a vertical position of the line of sight of the user in an image of the user and a process of prompting the user to change the vertical position of the line of sight of the user. The program according to any one of (1) to (7). (12) The gaze correction unit also performs a gaze matching process to correct the gaze of the user so as to match the gaze of the conversation partner based on the emotion-related information. The program according to any one of (1) to (11). (13) The line of sight correction unit performs the line of sight matching process based on the direction of the user's face. The program according to (12) above. (14) The gaze correction unit also performs a gaze matching process for correcting the gaze of the user to match the gaze of the conversation partner based on a conversation situation between the user and the conversation partner. The program according to any one of (1) to (11). (15) The computer, A generating unit that generates the emotion-related information based on at least one of an image and a voice of the conversation partner. The program according to any one of (1) to (14) above, for further functioning as a (16) An information processing device, performing gaze averting processing for shifting the gaze of the user in the image of the user so as not to coincide with the gaze of the conversation partner based on emotion-related information that is information related to the emotion of the conversation partner of the user; An information processing method comprising: (17) A gaze correction unit that performs gaze averting processing to shift the gaze of the user in the image of the user so as not to coincide with the gaze of the conversation partner based on emotion-related information that is information related to the emotion of the conversation partner of the user. An information processing device comprising: [Explanation of symbols]

[0148] 12 cloud computer, 41 CPU, 76 gaze correction unit, 81 generation unit, 82 display control unit, 121 gaze averting notification information, 276 gaze correction unit, 282 display control unit, 301 manuscript information, 321 gaze averting promotion related information

Claims

1. Computer, A gaze correction unit that performs gaze averting processing to shift the gaze of the user in the image of the user so as not to coincide with the gaze of the conversation partner based on emotion-related information that is information related to the emotion of the conversation partner of the user. A program to function as a

2. The emotion-related information is information related to at least one of the following: discomfort, tension, excitement, intimacy, and comfort of the conversation partner. The program according to claim 1.

3. The computer, a notification unit that notifies the user that the line-of-sight correction unit is performing the line-of-sight correction process The program according to claim 1, further functioning as a

4. The notification unit also notifies the user of information representing the emotion of the conversation partner based on the emotion-related information. The program according to claim 3.

5. The line of sight correction unit performs the line of sight averting process based on the emotion-related information when the operation mode is a conversation mode in which conversation is performed. The program according to claim 1.

6. The line of sight correction unit performs the line of sight averting process based also on a time during which the line of sight of the user and the line of sight of the conversation partner are aligned. The program according to claim 1.

7. The line of sight correction unit performs the line of sight averting process at a ratio based on at least one of a characteristic of the user, a characteristic of the conversation partner, and a situation of the conversation between the user and the conversation partner. The program according to claim 1.

8. The gaze averting process is a process of changing the vertical position of the user's gaze within the image of the user. The program according to claim 1.

9. The gaze averting process is a process for prompting the user to change the vertical position of the user's gaze. The program according to claim 1.

10. The computer, A display control unit that displays text information on a display unit by superimposing the text information on the image of the conversation partner. It also functions as The gaze averting process is a process of prompting the user to change the vertical position of the user's gaze by changing the vertical display position of the character information. The program according to claim 9.

11. The line of sight correction unit selectively performs, as the line of sight averting process, a process of changing a vertical position of the line of sight of the user in an image of the user and a process of prompting the user to change the vertical position of the line of sight of the user. The program according to claim 1.

12. The gaze correction unit also performs a gaze matching process to correct the gaze of the user so as to match the gaze of the conversation partner based on the emotion-related information. The program according to claim 1.

13. The line of sight correction unit performs the line of sight matching process based on the direction of the user's face. The program according to claim 12.

14. The gaze correction unit also performs a gaze matching process for correcting the gaze of the user to match the gaze of the conversation partner based on a conversation situation between the user and the conversation partner. The program according to claim 1.

15. The computer, A generating unit that generates the emotion-related information based on at least one of an image and a voice of the conversation partner. The program according to claim 1, further functioning as a

16. An information processing device, performing gaze averting processing for shifting the gaze of the user in the image of the user so as not to coincide with the gaze of the conversation partner based on emotion-related information that is information related to the emotion of the conversation partner of the user; An information processing method comprising:

17. A gaze correction unit that performs gaze averting processing to shift the gaze of the user in the image of the user so as not to coincide with the gaze of the conversation partner based on emotion-related information that is information related to the emotion of the conversation partner of the user. An information processing device comprising:

Images