Electronic apparatus

Abstract

To provide an electronic apparatus in which it can be easily noticed that a hearer is being seen by a speaker, even if the hearer of conversation does not see the speaker of the conversation.SOLUTION: An electronic apparatus comprises: display control means which controls to display an object corresponding to a hearer of conversation for a speaker of the conversation; acquisition means which acquires information on line-of-sight of the speaker; and control means which controls to notify the hearer, when the speaker is seeing the object corresponding to the hearer, on the basis of the information on the line-of-sight of the speaker.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to electronic devices, and more particularly to online conversations and conferences. [Background technology]

[0002] In recent years, there has been an increasing demand for conversations and meetings between people in remote locations, and systems (online conference systems) that allow conversations and meetings while displaying materials and participating users on computers or smartphones via a network are becoming widespread. There are also systems (VR conference systems) that display avatars that represent users in a virtual reality (VR) space that is generated by a computer and can be experienced as if it were real space. In a VR conference system, each user in a remote location can experience a sense of realism and immersion as if they were actually holding a conference on the spot.

[0003] Technologies that use the user's line of sight have also been proposed. Patent Document 1 discloses a technology that notifies another user of the position where a user is looking. Patent Document 2 discloses a technology that executes a dialogue with an object (non-user) in a VR space in response to a user uttering a voice while looking at the object.

[0004] When the speaker (utterer) of a conversation changes, the current speaker often looks at the next speaker (listener of the conversation), and in a conference held in the real world, the speaker to switch to is sometimes determined based on such behavior (gaze behavior). If the user's gaze is reflected in the gaze of the avatar, it is possible to determine who the speaker to switch to in a VR conference system based on the gaze behavior of the speaker (speaker's avatar). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2018 / 186031 [Patent Document 2] JP 2017-78893 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, in a real-world conference, if the listener is looking at the conference materials instead of the speaker, he / she may not notice that the speaker is watching him / her. Even in a conventional VR conference system (a VR conference system in which the user's gaze is reflected in the gaze of the avatar), the listener may not notice that the speaker is watching him / her unless he / she is looking at the speaker's avatar. In such a situation, if the listener cannot notice that he / she is being spoken to from the content of the speaker's speech, an additional action such as naming the listener is required. Also, in a conventional VR conference system, if an avatar whose gaze is difficult to grasp is used as the speaker's avatar, the listener cannot easily notice that the speaker is watching him / her. In a conventional online conference system, since an avatar is not used, the listener cannot easily notice that the speaker is watching him / her.

[0007] An object of the present invention is to enable a listener to easily become aware that he or she is being watched by a speaker of a conversation, even if the listener is not watching the speaker of the conversation. [Means for solving the problem]

[0008] The electronic device of the present invention transmits an object corresponding to a listener of a conversation to a speaker of the conversation. The present invention is characterized in that it has a display control means for controlling the display of the object corresponding to the listener based on the information on the speaker's line of sight, an acquisition means for acquiring information regarding the speaker's line of sight, and a control means for controlling the display of the object corresponding to the listener based on the information on the speaker's line of sight. Effect of the Invention

[0009] According to the present invention, even if a listener of a conversation does not look at the speaker of the conversation, the listener can easily become aware that the speaker is looking at him / her. [Brief description of the drawings]

[0010] [Figure 1] 1 is an external view and a block diagram of a display control device. [Diagram 2] This is a schematic diagram of VR space. [Diagram 3] FIG. 1 is a schematic diagram of a screen of a VR conference system application. [Figure 4] 4 is a flowchart of the operation of the display control device. [Diagram 5] 13 is a flowchart of a notification determination process. [Figure 6] 13 is a flowchart of a notification stop process. [Figure 7] FIG. 13 is a schematic diagram of a screen on which a notification is being made. [Figure 8] 11A to 11C are schematic diagrams illustrating screen transitions due to notifications. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] FIG. 1A is an external view of a display control device 100, which is a type of electronic device to which the present invention is applied. The display control device 100 is, for example, a display device such as a smartphone. The display 105 is a display unit that displays images and various information. The display 105 is integrally configured with a touch panel 106a, and is configured to detect a touch operation on the display surface of the display 105. The display control device 100 is capable of VR display of a VR image (VR content) on the display 105. The operation unit 106b is a power button that accepts an operation to switch the power of the display control device 100 on and off. The operation unit 106c and the operation unit 106d are volume buttons that increase and decrease the volume of the sound output from the speaker 112b, an earphone connected to the audio output terminal 112a, an external speaker, or the like. The operation unit 106e is a home button for displaying a home screen on the display 105. The audio output terminal 112a is an earphone jack, and is a terminal that outputs an audio signal to an earphone, an external speaker, or the like. The speaker 112b is a built-in speaker that outputs audio. The audio input terminal 114a is an earphone jack, and is a terminal for inputting audio (audio signal) from a microphone (microphone), an earphone with a microphone, etc. The audio output terminal 112a and the audio output terminal 112b may be a common (same) earphone jack. The microphone 114b is a built-in microphone for inputting audio.

[0013] 1(B) is a block diagram showing an example of the configuration of the display control device 100. A CPU 101, a memory 102, a non-volatile memory 103, an image processing unit 104, a display 105, an operation unit 106, a recording medium I / F 107, an external I / F 109, and a communication I / F 110 are connected to an internal bus 150. In addition, an audio output unit 112, a posture detection unit 113, an audio input unit 114, and a gaze detection unit 115 are also connected to the internal bus 150. Each unit connected to the internal bus 150 is configured to be able to exchange data with each other via the internal bus 150.

[0014] The CPU 101 is a control unit that controls the entire display control device 100, and is composed of at least one processor or circuit. The memory 102 is composed of, for example, a RAM (a volatile memory using a semiconductor element, etc.). The CPU 101 stores, for example, data stored in a non-volatile memory 103. According to the stored programs, the memory 102 is used as a work memory to control each unit of the display control device 100. The non-volatile memory 103 stores image data, audio data, other data, and various programs for the operation of the CPU 101. The non-volatile memory 103 is composed of, for example, a flash memory or a ROM.

[0015] The image processing unit 104 performs various image processing on images stored in the non-volatile memory 103 or the recording medium 108, video signals acquired via the external I / F 109, images acquired via the communication I / F 110, etc., based on the control of the CPU 101. The image processing performed by the image processing unit 104 includes A / D conversion processing, D / A conversion processing, image data encoding processing, compression processing, decoding processing, enlargement / reduction processing (resizing), noise reduction processing, color conversion processing, etc. In addition, various image processing such as panoramic development, mapping processing, conversion of VR images, which are omnidirectional images or wide-range images having wide-range images even if not omnidirectional, is also performed. The image processing unit 104 may be configured with a dedicated circuit block for performing specific image processing. Depending on the type of image processing, the CPU 101 can perform image processing according to a program without using the image processing unit 104.

[0016] The display 105 displays images, GUI screens constituting a GUI (Graphical User Interface), and the like under the control of the CPU 101. The CPU 101 generates a display control signal according to a program, and controls each unit of the display control device 100 to generate a video signal for display on the display 105 and output it to the display 105. The display 105 displays an image based on the generated and output video signal. Note that the configuration of the display control device 100 itself is limited to an interface for outputting a video signal for display on the display 105, and the display 105 may be configured as an external monitor (such as a television or a head mounted display (HMD)).

[0017] The operation unit 106 is an input device for receiving user operations, including a character information input device such as a keyboard, a pointing device such as a mouse or a touch panel, a button, a dial, a joystick, a touch sensor, a touch pad, etc. In this embodiment, the operation unit 106 includes a touch panel 106a and operation units 106b, 106c, 106d, and 106e.

[0018] A recording medium 108 such as a memory card, CD, or DVD can be detachably attached to the recording medium I / F 107. The recording medium I / F 107 reads data from the attached recording medium 108 and writes data to the recording medium 108 under the control of the CPU 101. The recording medium 108 is a storage unit that stores data such as images to be displayed on the display 105. The external I / F 109 is an interface that connects to an external device via a wired cable (such as a USB cable) or wirelessly, and inputs and outputs (data communication) video signals and audio signals. The communication I / F 110 is an interface that communicates (wireless communication) with an external device, the Internet 111, etc., and transmits and receives (data communication) various data such as files and commands.

[0019] The audio output unit 112 outputs the sound of video and music data played back by the display control device 100, operation sounds, ringtones, various notification sounds, etc. The audio output unit 112 includes an audio output terminal 112a for connecting earphones or the like, and a speaker 112b that is a built-in speaker of the main body, but the audio output unit 112 may output audio data to an external speaker via wireless communication or the like.

[0020] The attitude detection unit 113 detects the attitude (tilt) of the display control device 100 with respect to the direction of gravity and the attitude of the display control device 100 with respect to each axis of the yaw direction, pitch direction, and roll direction, and notifies the CPU 101 of the attitude information. It is possible to determine whether the device 100 is held horizontally, vertically, facing up, facing down, or at an angle, etc. It is also possible to determine the presence or absence and magnitude of inclination of the display control device 100 in a rotation direction such as a yaw direction, pitch direction, or roll direction, and whether the display control device 100 has rotated in the rotation direction, etc. One or a combination of multiple sensors from among an acceleration sensor, a gyro sensor, a geomagnetic sensor, a direction sensor, an altitude sensor, etc. can be used as the attitude detection unit 113.

[0021] The audio input unit 114 inputs audio (audio data) from a microphone or the like. The audio input unit 114 includes an audio input terminal 114a for connecting a microphone or the like, and a microphone 114b that is a built-in microphone in the main body, but the audio input unit 114 may input audio via wireless communication.

[0022] Gaze detection unit 115 detects the user's gaze. For example, gaze detection unit 115 captures an image of the user's face and eyes, and detects the gaze (the position and direction the user is looking at) based on the image of the user's face and eyes. CPU 101 can determine where the user is looking at the image displayed on display 105, based on gaze information obtained by gaze detection unit 115 (detection result of the user's gaze).

[0023] As described above, the operation unit 106 includes the touch panel 106a. The touch panel 106a is configured to be planar and overlaid on the display 105, and is an input device that outputs coordinate information according to the touched position. The CPU 101 can detect the following operations or states on the touch panel 106a. A finger or pen that has not been touching the touch panel 106a touches the touch panel 106a again, that is, the start of touching (hereinafter referred to as Touch-Down). A state in which a finger or a pen is touching the touch panel 106a (hereinafter referred to as Touch-On) A finger or a pen is moved while touching the touch panel 106a (hereinafter referred to as Touch-Move). The finger or pen that has been touching the touch panel 106a is removed from the touch panel 106a, that is, the touch ends (hereinafter referred to as "touch-up"). A state in which nothing is touching the touch panel 106a (hereinafter referred to as Touch-Off)

[0024] When touch-down is detected, touch-on is also detected at the same time. After touch-down, touch-on will usually continue to be detected unless touch-up is detected. Touch-on will also continue to be detected if touch-move is detected. Even if touch-on is detected, touch-move will not be detected if the touch position does not move. Touch-off is detected when it is detected that all fingers or pens that were touching have touched up.

[0025] These operation states and the position coordinates of the touch panel 106a touched by a finger or pen are notified to the CPU 101 via an internal bus, and the CPU 101 determines what kind of operation (touch operation) has been performed on the touch panel 106a based on the notified information. Regarding touch-move, the moving direction of the finger or pen moving on the touch panel 106a can also be determined for each vertical and horizontal component on the touch panel 106a based on the change in the position coordinates. If a touch-move of a predetermined distance or more is detected, it is determined that a slide operation has been performed.

[0026] The operation of touching the touch panel 106a with a finger, moving the finger quickly for a certain distance, and then releasing the finger is called a flick. This is an operation to quickly trace the top of 6a with a finger as if flicking it. If a touch-move is detected over a certain distance or at a certain speed or faster, and then a touch-up is detected, it can be determined that a flick has been performed (it can be determined that a flick has occurred following a slide operation).

[0027] Furthermore, a touch operation in which multiple points (for example, two points) are touched simultaneously and the touch positions are brought closer to each other is called pinch in, and a touch operation in which the touch positions are moved away from each other is called pinch out. Pinch out and pinch in are collectively called pinch operation (or simply pinch). The touch panel 106a may be of any of various touch panel types, such as a resistive film type, a capacitive type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, and an optical sensor type. There are types that detect a touch by contact with the touch panel, and types that detect a touch by the approach of a finger or a pen to the touch panel, and either type may be used.

[0028] FIG. 1C is an external view of VR goggles (head mount adapter) 130 to which the display control device 100 can be attached. The display control device 100 can also be used as a head mount display by attaching it to the VR goggles 130. The insertion port 131 is an insertion port for inserting the display control device 100. The entire display control device 100 can be inserted into the VR goggles 130 with the display surface of the display 105 facing the headband 132 side (i.e., the user side) for fixing the VR goggles 130 to the user's head. The user can view the display 105 without holding the display control device 100 with his / her hands while wearing the VR goggles 130 to which the display control device 100 is attached on his / her head. In this case, when the user moves his / her head or entire body, the posture of the display control device 100 also changes. The posture detection unit 113 detects the posture change of the display control device 100 at this time, and the CPU 101 performs processing for VR display based on this posture change. In this case, the detection of the orientation of the display control device 100 by the orientation detection unit 113 is equivalent to detecting the orientation of the user's head. The head orientation may be regarded as the direction in which the user's gaze is directed, and the gaze detection unit 115 may acquire the detection result of the head orientation as gaze information. Note that the display control device 100 itself may be an HMD that can reach the head without VR goggles.

[0029] A VR image is an image that can be displayed in VR (displayed in the "VR View" display mode). VR images include omnidirectional images (spherical images) captured by an omnidirectional camera (spherical camera) and panoramic images with a wider image range (effective image range) than the display range that can be displayed at one time on the display unit. VR images include not only still images, but also videos and live view images (images acquired from the camera almost in real time). A VR image has an image range (effective image range) of up to 360 degrees in the vertical direction (vertical angle, angle from the zenith, elevation angle, depression angle, altitude angle, pitch angle) and 360 degrees in the left and right direction (horizontal angle, azimuth angle, yaw angle).

[0030] In addition, VR images also include images that have a wider angle of view (field of view) than the angle of view that can be captured by a normal camera, or a wider image range (effective image range) than the display range that can be displayed at one time on a display unit, even if the angle of view is less than 360 degrees up and down and less than 360 degrees left and right. For example, an image captured by a spherical camera capable of capturing an object with a field of view (field of view) of 360 degrees in the left and right directions (horizontal angle, azimuth angle) and a vertical angle of 210 degrees centered on the zenith is a type of VR image. In addition, an image captured by a camera capable of capturing an object with a field of view (field of view) of 180 degrees in the left and right directions (horizontal angle, azimuth angle) and a vertical angle of 180 degrees centered on the horizontal direction is a type of VR image. In other words, an image that has a field of view of 160 degrees (±80 degrees) or more in the up and down and left and right directions and has a wider image range than the range that a human can see at one time is a type of VR image.

[0031] When this VR image is displayed in VR (displayed in "VR view" display mode), seamless omnidirectional images can be viewed in the left-right direction (horizontal rotation direction) by changing the position of the display device (the display device displaying the VR image) in the left-right rotation direction. In the up-down direction (vertical rotation direction), seamless omnidirectional images can be viewed within a range of ±105 degrees from directly above (the zenith), but the range beyond 105 degrees from directly above becomes a blank area with no images. A VR image can also be described as "an image whose image range is at least a part of a virtual space (VR space)."

[0032] VR display (VR view) is a display method (display mode) that can change the display range, displaying an image of a VR image with a viewing range according to the orientation of the display device. When viewing by wearing a head-mounted display (HMD), which is a display device, an image with a viewing range according to the orientation of the user's face is displayed. For example, assume that a VR image is displayed with a viewing angle (field of view) centered on 0 degrees left and right (a specific direction, for example, north) and 90 degrees up and down (90 degrees from the zenith, i.e., horizontal) at a certain point in time. From this state, if the orientation of the display device is flipped over (for example, if the display surface is changed from facing south to facing north), the display range is changed to an image with a viewing angle centered on 180 degrees left and right (the opposite direction, for example, south) and 90 degrees up and down (horizontal) in the same VR image. In the case where a user is viewing an HMD, if the user turns his or her face from north to south (i.e., turns around), the image displayed on the HMD will also change from a north image to a south image. This type of VR display can provide users with a sense of immersion, as if they were visually inside the VR image (VR space). A smartphone attached to a VR goggle (head-mounted adapter) can be considered a type of HMD.

[0033] The display method of the VR image is not limited to the above. The display range may be moved (scrolled) in response to a user operation on a touch panel or directional buttons, instead of a change in posture. During VR display (when in the "VR view" display mode), in addition to changing the display range due to a change in posture, the display range may be changed in response to a touch move on the touch panel, a drag operation on the mouse, pressing of a directional button, etc.

[0034] An example will be described in which a user wears VR goggles 130 equipped with the display control device 100 on his / her head and uses a VR conference system application.

[0035] 2(A) and 2(B) are schematic diagrams showing a VR space constructed by a VR conference system application. FIG. 2(A) shows a conference (VR conference) viewed from the side in a bird's-eye view, and FIG. 2(B) shows a VR conference viewed from the top. In the VR conference system application, avatars (virtual objects that are the avatars of the participants) corresponding to the participants of the VR conference are placed in the VR space. In FIG. 2(A) and 2(B), four avatars 251 to 254 corresponding to the four participants respectively are placed around a table that is a virtual object, and each of the avatars 251 to 254 is sitting on a chair that is a virtual object. In FIG. 2(A) and 2(B), a screen 250 that is a virtual object is also placed in the VR space. Materials for the conference and the like are displayed on the screen 250. Each participant can view the screen 250 from the viewpoint of his or her own avatar. A screen may be prepared for each participant (avatar), for example, four screens may be prepared corresponding to four participants (four avatars 251 to 254). If there is another object between the avatar and the screen, the participant corresponding to the avatar may be able to see the screen through the other object.

[0036] Assume that avatar 251 is the avatar of a participant who is the speaker of the conversation, and avatars 252 to 254 are the avatars of participants who are listeners to the conversation. Arrow 261 in FIG. 2B indicates the line of sight of avatar 251, and arrow 262 indicates the line of sight of avatar 252. Avatar 251 (speaker) looks at avatar 252 (listener) hoping for a response from avatar 252, but avatar 252 looks at screen 250 and not at avatar 251.

[0037] FIG. 3(A) is a schematic diagram showing a screen 300 of a VR conference system application displayed on the display 105. The screen 300 is a VR view screen that displays a view from the position of an avatar. The screen 300 is rendered based on the arrangement of each avatar and the posture of the display control device 100 detected by the posture detection unit 213 (the posture of the user's head, the direction of the user's face (head)). The avatar corresponding to the user of the display control device 100 is assumed to be the avatar 252. Therefore, the view from the position of the avatar 252 is displayed on the screen 300. As described above, the avatar 252 (listener) is looking at the screen 250. Therefore, the screen 250 occupies the majority of the screen 300 (the field of view of the avatar 252), and the avatar 251 (speaker) is cut off and difficult to see.

[0038] The screen of the VR conference system application is not limited to the VR view screen. For example, the screen of the VR conference system application may be a screen view screen that displays the same image as the screen 250 on the entire surface or most of the display area of ​​the display 105.

[0039] FIG. 3B is a schematic diagram showing a screen 310 of a VR conference system application. The screen 310 is a screen view screen. In an area 311 of the screen 310, the same display as the screen 250 is performed. The screen 310 also displays a participant list 312 and an operation panel 313. The participant list 312 is a list showing each participant by icon or name (ID). The operation panel 313 includes various operation objects for user operation. For example, the operation panel 313 includes a button for entering and leaving the VR conference, a button for switching a microphone on and off, a button for switching a camera on and off, and the like.

[0040] The participant list 312 and the operation panel 313 may be displayed on the VR view screen. The operation panel 313 may also include a button for switching the displayed screen between a VR view screen and a plurality of screens including the screen view screen. The screen view screen may be a screen in which an area 311 (a screen that displays the same as the screen 250) is superimposed on the VR view screen. In this case, the VR view screen may be viewable through the area 311. A reduced VR view screen may be displayed on the screen view screen.

[0041] Fig. 4 is a flowchart showing the operation of the display control device 100. This operation is realized by the CPU 101 expanding a program stored in the non-volatile memory 103 into the memory 102 and executing it. For example, when the display control device 100 is started and an instruction is given to start a VR conference using a VR conference system application, the operation of Fig. 4 starts.

[0042] In S401, the CPU 101 displays a screen of a VR conference system application on the display 105. For example, the CPU 101 displays a VR view screen or a screen view screen on the display 105. As a result, objects corresponding to participants of the VR conference are displayed on the display 105. For example, the objects corresponding to participants are avatars in the case of a VR view screen, and icons or names included in a participant list in the case of a screen view screen. Note that an object corresponding to a user of the display control device 100 may or may not be displayed.

[0043] In S402, the CPU 101 detects the user's voice using the voice input unit 114, and detects the user's line of sight using the line of sight detection unit 115.

[0044] In S403, the CPU 101 detects the user's voice based on the result of the voice detection in S402. If voice is detected (if the user is the speaker), proceed to S404, otherwise proceed to S402.

[0045] In S404, the CPU 101 acquires voice information related to the voice uttered by the user (speaker) based on the result of the voice detection in S402. The voice information includes, for example, information (time code) indicating the time period during which the user (speaker) is uttering the voice. The voice information may include voice data, or may include text data generated by transcribing the voice data.

[0046] In S405, CPU 101 acquires information about the line of sight of the user (speaker). In this embodiment, CPU 101 acquires object information about an object that the user is looking at. The object information includes, for example, information indicating the object that the user is looking at (participant name, ID) and information indicating the time period during which the user is looking at the object (time code).

[0047] For example, the CPU 101 acquires gaze position information on the gaze position of the user (speaker) based on the result of the gaze detection in S402, and acquires gaze object information on an object (object at which the user's gaze is directed) displayed at the gaze position. The CPU 101 may acquire face direction object information on an object at which the user's face is directed, based on the posture of the display control device 100 detected by the posture detection unit 113. The CPU 101 may acquire one of the gaze object information and the face direction object information, or may acquire both. The object information may include information indicating the type of the object information (information indicating whether it is gaze object information or face direction object information). The CPU 101 may acquire object information for all displayed objects, or may acquire object information for only avatars of participants other than the user of the display control device 100.

[0048] In S406, CPU 101 determines whether or not the user (speaker) is looking at and speaking to an object corresponding to a listener (a participant other than the user) based on the voice information acquired in S404 and the object information acquired in S405. For example, CPU 101 determines whether or not object information indicating a time period corresponding to the time period indicated by the voice information indicates an object of the listener. If the user is looking at and speaking to an object corresponding to the listener (if object information indicating a time period corresponding to the time period indicated by the voice information indicates an object of the listener), proceed to S407; otherwise, proceed to S402. If the user is looking at but not speaking to an object corresponding to the listener, proceed to S402.

[0049] Alternatively, it may be determined whether the user (speaker) is looking at the object corresponding to the listener based on information about the user's (speaker's) line of sight without using voice information. If the user is looking at the object corresponding to the listener, the process may proceed to S407, and if not, the process may proceed to S402. In this case, S404 may be omitted.

[0050] In S407, the CPU 101 performs a notification determination process to determine whether or not to notify the listener detected in S406 (the listener corresponding to the object that the user (speaker) is looking at and talking about). Details of the notification determination process will be described later with reference to Figs. 5(A) to 5(D).

[0051] In S408, CPU 101 switches the process depending on the result of the notification determination process in S407. If notification is to be performed for the listener detected in S406, the process proceeds to S409, otherwise the process proceeds to S402. Note that S407 and S408 may be omitted, and the process may proceed from S406 to S409 if the user is looking at (looking at and talking to) an object corresponding to the listener.

[0052] In S409, CPU 101 notifies the listener detected in S406. For example, CPU 101 causes the display control device of the listener detected in S406 to notify the listener by transmitting a control signal to the display control device of the listener detected in S406 via communication I / F 110. The notification method is not particularly limited, and a visual notification by display, an auditory notification by audio output, or a tactile notification by vibration may be performed. Any one of these notifications or a combination of two or more notifications may be performed. The notification information may be specified by the control signal.

[0053] In S410, the CPU 101 determines whether or not the user has instructed to end the VR conference using the operation unit 106. If an instruction to end the VR conference has been given, the operation in Fig. 4 ends, and if not, the process proceeds to S402.

[0054] Figures 5(A) to 5(D) are flowcharts showing the notification determination process of S407 in Figure 4. CPU 101 may be able to execute only one of the four notification determination processes shown in Figures 5(A) to 5(D), or may be able to execute multiple notification determination processes. CPU 101 may switch between multiple notification determination processes as appropriate, or may execute a combination of multiple notification determination processes.

[0055] In the notification determination process of Figure 5 (A), if the user's level of attention (priority) to the object of the listener detected in S406 of Figure 4 (the listener corresponding to the object that the user (speaker) is looking at and talking about) is higher than a threshold value, it is determined that a notification should be sent to that listener.

[0056] In S501a, CPU 101 calculates the degree of gaze of the user to the object of the listener detected in S406. For example, CPU 101 calculates the degree of gaze from the time during which the gaze is directed to the object (gaze time), the weight of the gaze time (gaze weight value), the time during which the face is directed to the object (face direction time), and the weight of the face direction time (face direction weight value) using the following formula 1. The gaze weight value is greater than the face direction weight value. If multiple listeners are detected in S406, the degree of gaze is calculated for each of the multiple listeners. Gaze intensity = gaze weight value x gaze duration + face direction weight value x face direction duration

[0057] In S502a, CPU 101 determines whether or not the gaze degree calculated in S501a is greater than a predetermined threshold value. If the gaze degree is greater than the predetermined threshold value, the process proceeds to S503a, and if not, the process proceeds to S504a.

[0058] In S503a, the CPU 101 determines (decides) to notify the listener detected in S406.

[0059] In S504a, the CPU 101 determines not to issue a notification.

[0060] If multiple listeners are detected in S406, the processes of S502a to S504a are performed for each of the multiple listeners.

[0061] In the notification determination process in Fig. 5(B), it is determined that a notification should be made after the user (speaker) stops speaking and a predetermined time has passed with no participants speaking. If no participants speak, it is considered that each participant does not know who is the next participant to speak and is unable to determine whether or not it is OK for each participant to speak. By making a notification in such a case, it is possible to encourage participants to speak.

[0062] In S501b, the CPU 101 performs the process of It is determined whether a predetermined time has elapsed. If the predetermined time has elapsed, the process proceeds to S502b, and if not, the process proceeds to S504b. Although it has been stated that if the user's voice is not detected in S402, the process proceeds from S403 to S402, there is an exception. If the user's voice is detected in S402, and none of the participants speaks, and the user's voice is not detected in the next S402, there is no change of speaker, so the process proceeds from S403 to S404. If the user's voice is detected in S402, and a participant other than the user speaks, and the user's voice is not detected in the next S402, the participant becomes the speaker (there has been a change of speaker), so the process proceeds from S403 to S402.

[0063] In S502b, the CPU 101 judges whether or not the participant has made a sound. If the participant has not made a sound, the process proceeds to S503b, otherwise the process proceeds to S504b. Voice information (voice data) of the user of the display control device 100 is acquired using the voice input unit 114, and voice information (voice data) of the participant other than the user is acquired from the display control device of the participant via the communication I / F 110.

[0064] In S503b, the CPU 101 determines (decides) to notify the listener detected in S406.

[0065] In step S504b, the CPU 101 determines not to issue a notification.

[0066] In the notification determination process of FIG. 5C, when multiple participants are uttering sounds, it is determined that no notification should be made since a notification prompting them to speak is unnecessary or disturbing.

[0067] In S501c, the CPU 101 acquires voice information of participants other than the user from the display control devices of the participants via the communication I / F 210.

[0068] In S502c, the CPU 101 determines whether or not voice information was acquired in S501c. If voice information was acquired, that is, if the user and the other participants are emitting voices at the same timing, the process proceeds to S504c, and if not, the process proceeds to S503c.

[0069] In S503c, the CPU 101 determines (decides) to notify the listener detected in S406.

[0070] In S504c, the CPU 101 determines not to issue a notification.

[0071] In the notification determination process of FIG. 5(D), if the number of participants in the VR conference is not within a predetermined range, it is determined that no notification will be issued. For example, if the number of participants in the VR conference is two, the participant being spoken to is clear. Also, depending on the VR conference, one participant (host) may speak to many participants (guests) and may not want to prompt the guests to speak. In the notification determination process of FIG. 5(D), unnecessary (obtrusive) notifications in such cases can be suppressed.

[0072] In S501d, the CPU 101 acquires user information of participants other than the user from the display control devices of the participants via the communication I / F 210, and counts the number of participants in the VR conference (the number of participants in the VR conference including the user). Instead of the user information of participants other than the user, other information such as voice information of the participants may be used.

[0073] In S502d, the CPU 101 judges whether the number of participants counted in S501d is within a predetermined range. If the number of participants is within the predetermined range, the process proceeds to S503d, and if not, the process proceeds to S504d. The predetermined range is a fixed range common to a plurality of participants. Alternatively, the range may be a range specified by the user.

[0074] In S503d, the CPU 101 determines (decides) to notify the listener detected in S406.

[0075] In S504d, the CPU 101 determines not to issue a notification.

[0076] The notification determination process is not limited to the above process. For example, the user may specify the participants to be notified in advance, and the CPU 101 may determine to perform notification if the object the user is looking at (looking at and talking about) is an object of the participant to be notified, and may determine not to perform notification if it is not. The CPU 101 may determine to perform notification if the number of listeners detected in S406 is within a predetermined range, and may determine not to perform notification if it is not. The CPU 101 may set whether each participant is a notification target or not, or set the above-mentioned predetermined range for the number of listeners detected in S406, in response to an instruction from the user. Various settings including these may be made during the VR conference.

[0077] Fig. 6 is a flowchart showing the notification stop processing. This processing is realized by the CPU 101 expanding a program stored in the non-volatile memory 103 into the memory 102 and executing it. When the notification is performed (started) in S409 in Fig. 4, the processing in Fig. 6 starts.

[0078] The notification in S409 is performed to make the listener detected in S406 the next speaker. If the listener detected in S406 emits a voice (if the listener detected in S406 becomes the speaker), the notification may of course be stopped. Even if the listener detected in S406 does not recognize the notification, if another participant (another listener) emits a voice, the notification role is no longer played, so the notification may be stopped. If the listener detected in S406 looks at the object of the user (speaker), the notification role is also no longer played, so the notification may be stopped. In the notification stop processing in FIG. 6, the notification is stopped in these cases.

[0079] In this embodiment, it is assumed that the process of Fig. 6 is performed when the user of the display control device 100 is a listener and the display control device 100 is notifying the user. When the user of the display control device 100 is a speaker and is notifying a listener (a listener detected in S406) who is a participant other than the user, the display control device 100 may perform the process of Fig. 6. In this case, the display control device 100 stops the notification to the listener by transmitting a control signal to the display control device of the listener to which the notification is being performed.

[0080] In S601, the CPU 101 determines whether or not there is voice information of the listener. That is, the CPU 101 determines whether or not the listener has made a sound. If there is voice information of the listener (if the listener has made a sound), the process proceeds to S604, and if not, the process proceeds to S602. Voice information (voice data) of the user of the display control device 100 is acquired using the voice input unit 114, and voice information (voice data) of participants other than the user is acquired from the display control device of the participant via the communication I / F 110. It is assumed that who is the speaker and who is the listener is managed by each display control device based on the voice information of each participant.

[0081] In S602, CPU 101 uses gaze detection unit 115 and posture detection unit 113 to obtain object information (gaze object information and face direction object information) indicating an object that a user (listener to whom a notification is being given) is looking at.

[0082] In S603, the CPU 101 determines whether or not the user (the listener to whom the notification is being sent) is looking at the speaker based on the object information acquired in S602. If the hand is being viewed, the process proceeds to S604, otherwise the process proceeds to S601.

[0083] In S604, the CPU 101 stops the notification to the user.

[0084] The notification stop process is not limited to the above process. For example, one of the judgments of S601 and S603 may be omitted. The notification may be stopped when proceeding from S403 or S406 in FIG. 4 to S402. That is, the notification may be stopped in response to the speaker stopping uttering a voice, or in response to the speaker stopping looking at the listener (the listener to whom the notification is being made). When the speaker stops looking at the listener (the listener to whom the notification is being made), this includes the case where the speaker stops looking at the listener and looks at another listener.

[0085] 7(A) and 7(B) are schematic diagrams showing the screen on which the notification in S409 of FIG. 4 is being performed.

[0086] The screen 700 in FIG. 7(A) is a VR view screen. In FIG. 7(A), a notification frame 705 is displayed along the outline (edge) of the screen 700 as the notification in S409. The VR space may be visible through the notification frame 705. The listener can easily notice the notification frame 705, and therefore can easily notice that the speaker is watching them. When the listener notices that the speaker is watching them, they may try to look at the speaker's avatar. For this reason, it is preferable that the notification in S409 includes a notification of the position of the speaker's avatar (an object corresponding to the speaker). The notification frame 705 is displayed with a gradation in which the color becomes darker toward the position of the speaker's avatar. This allows the listener to easily grasp the position of the speaker's avatar. The notification frame 705 may be made more noticeable with animation to encourage the listener to look at the speaker's avatar.

[0087] The screen 710 in Fig. 7(B) is a screen view screen. In Fig. 7(B), a notification frame 715 is displayed along the outline (edge) of the screen 710 as a notification in S409. Since the listener can easily notice the notification frame 705, the listener can easily notice that the speaker is watching them.

[0088] The display of the notification frames 705 and 715 is effective as a display to prevent the notification from interfering with the VR conference, but the notification in S409 is not limited to this. For example, a pop-up display may be performed as the notification in S409. In the pop-up display, direct content such as being seen by the speaker or being spoken to by the speaker may be notified. A character string (text data) representing the words uttered by the speaker may be generated by transcribing the speaker's voice information (voice data). The generated character string may then be displayed. A summary of the character string (the words uttered by the speaker) may be generated and displayed. The intention of the words uttered by the speaker (for example, a question or a warning) may be determined from the generated character string and the speaker's voice information, and the intention may be notified in a recognizable manner.

[0089] The notification in S409 may be the notification shown in Fig. 8. Participant list 800 in Fig. 8 is a participant list in a state where there is no speaker and no notification has been made, and participant list 810 is a participant list in a state where there is a speaker and notification has been made. In participant lists 800 and 810, the names (IDs) and icons of four participants are listed. The names of the four participants are User01 to User04, and icons 850 to 853 correspond to User01 to User04, respectively.

[0090] Here, assume that User02 becomes a speaker and speaks while looking at User03, who is a listener. In this case, the participant list is transitioned from participant list 800 to participant list 810. In participant list 810, icon 851 of User02 (speaker) is highlighted with a frame 812. The icon 852 of User03 (listener) seen by User02 (speaker) is highlighted by a frame 813. The frames 812 and 813 are displayed in a distinguishable manner. For example, the frames 812 and 813 are displayed in different colors or patterns. This makes it possible to specify that User02 is the speaker and is speaking while looking at User03, the listener. Then, a participant looking at the participant list 810 can easily notice that User02 is the speaker and is speaking while looking at User03, the listener. Furthermore, User02 (speaker) is displayed at the top, and User03 (listener) seen by User02 (speaker) is displayed second. This makes it possible to specify that User03 (listener) is a candidate for the next speaker. Then, a participant looking at the participant list 810 can easily notice that User03 (listener) is a candidate for the next speaker.

[0091] The above transition of the participant list may be performed only by the display control device of the notified User03 (listener), or may be performed by the display control devices of all participants.

[0092] The above-described embodiment (including the modified examples) is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the configurations of the above-described embodiment within the scope of the gist of the present invention. The present invention also includes configurations obtained by appropriately combining the configurations of the above-described embodiment.

[0093] For example, the present invention is not limited to HMDs (including smartphones attached to VR goggles) but can also be applied to information processing devices separate from HMDs (for example, personal computers connected to HMDs). The present invention is not limited to client devices on the user side but can also be applied to host server devices (server devices) connected to client devices via a network.

[0094] The present invention can also be applied to online conference systems other than VR conference systems. For example, images of multiple participants are displayed in multiple areas of a computer screen, and when a user is talking while looking at an area of ​​a participant other than the user, a notification may be sent to the participant. The present invention can also be applied to cases where MR technology or AR technology is used instead of VR technology. In that case, an object of each participant will be displayed in real space.

[0095] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0096] The disclosure of the present embodiment includes the following configuration, method, program, and medium. (Configuration 1) a display control means for controlling display of an object corresponding to a listener of a conversation to a speaker of the conversation; An acquisition means for acquiring information regarding the speaker's gaze; a control means for controlling to notify the listener when the speaker is looking at the object corresponding to the listener based on the information regarding the speaker's line of sight; 1. An electronic device comprising: (Configuration 2) a determining means for determining whether the speaker is looking at the object corresponding to the listener based on the information regarding the speaker's line of sight; and The control means performs control so as to notify the listener when it is determined that the speaker is looking at the object corresponding to the listener. 2. The electronic device according to configuration 1. (Configuration 3) A second acquisition means for acquiring voice information relating to a voice uttered by the speaker. and the acquiring means acquires, as the information on the speaker's line of sight, object information on an object that the speaker is looking at; the determining means determines, based on the voice information and the object information, whether the speaker is speaking while looking at the object corresponding to the listener; The control means if the speaker is looking at the object corresponding to the listener but is not speaking, then not notifying the listener; When it is determined that the speaker is speaking while looking at the object corresponding to the listener, control is performed so as to notify the listener. 3. The electronic device according to configuration 2. (Configuration 4) The voice information indicates a time period during which the speaker is uttering the voice. 4. The electronic device according to configuration 3. (Configuration 5) The object information indicates a time period during which the speaker is viewing the object corresponding to the object information. 5. The electronic device according to configuration 4. (Configuration 6) The determining means determines whether the speaker is speaking while looking at the object corresponding to the listener, based on the object information indicating a time period corresponding to a time period indicated by the voice information. 6. The electronic device according to configuration 5. (Configuration 7) the display control means controls to display a plurality of objects corresponding to a plurality of listeners of the conversation to the speaker; The control means performs control to notify, among the plurality of listeners, a listener corresponding to an object that the speaker is looking at with a gaze intensity higher than a threshold, based on the information regarding the speaker's gaze. 7. The electronic device according to any one of configurations 1 to 6. (Configuration 8) The control means does not issue the notification when the number of participants in the conversation is not within a predetermined range. 8. The electronic device according to any one of configurations 1 to 7. (Configuration 9) The control means does not issue the notification when multiple participants of the conversation are emitting voices. 9. The electronic device according to any one of configurations 1 to 8. (Configuration 10) The control means performs control so as to stop the notification in response to the listener uttering a voice after starting the notification. 10. The electronic device according to any one of configurations 1 to 9. (Configuration 11) The control means controls to stop the notification in response to the listener seeing the speaker after starting the notification. 11. The electronic device according to any one of configurations 1 to 10. (Configuration 12) The control means controls the notification to be made after the speaker stops uttering a sound and a predetermined time has elapsed with no sound being uttered by any participant in the conversation. 12. The electronic device according to any one of configurations 1 to 11. (Configuration 13) The control means controls to stop the notification in response to the speaker stopping uttering a voice after starting the notification. 12. The electronic device according to any one of configurations 1 to 11. (Configuration 14) The control means controls to stop the notification in response to the speaker stopping looking at the listener after starting the notification. 14. The electronic device according to any one of configurations 1 to 13. (Configuration 15) The notification includes a notification of a position of an object corresponding to the speaker. 15. The electronic device according to any one of configurations 1 to 14. (Configuration 16) The notification includes notification of information that enables the speaker to be identified. 16. The electronic device according to any one of configurations 1 to 15. (Configuration 17) The control means controls to perform the notification by display. 17. The electronic device according to any one of configurations 1 to 16. (Configuration 18) The control means controls so as to give the notification by audio output. 17. The electronic device according to any one of configurations 1 to 16. (Configuration 19) A detection means for detecting the speaker's line of sight Further having 19. The electronic device according to any one of configurations 1 to 18. (Configuration 20) The electronic device is worn by the speaker. 20. The electronic device according to any one of configurations 1 to 19. (method) controlling a display of objects corresponding to listeners of a conversation to a speaker of said conversation; acquiring information regarding the speaker's gaze; a control step of controlling to notify the listener when the speaker is looking at the object corresponding to the listener based on the information regarding the speaker's gaze; 13. A method for controlling an electronic device comprising: (program) A program for causing a computer to function as each of the means of the electronic device according to any one of configurations 1 to 20. (medium) A computer-readable storage medium storing a program for causing a computer to function as each of the means of the electronic device according to any one of configurations 1 to 20. [Explanation of symbols]

[0097] 100: Display control device 101: CPU

Claims

1. a display control means for controlling display of an object corresponding to a listener of a conversation to a speaker of the conversation; An acquisition means for acquiring information regarding the speaker's gaze; a control means for controlling to notify the listener when the speaker is looking at the object corresponding to the listener based on the information regarding the speaker's line of sight; 1. An electronic device comprising:

2. a determining means for determining whether the speaker is looking at the object corresponding to the listener based on the information regarding the speaker's line of sight; and The control means performs control so as to notify the listener when it is determined that the speaker is looking at the object corresponding to the listener.

2. The electronic device according to claim 1 .

3. A second acquisition means for acquiring voice information relating to a voice uttered by the speaker. and the acquiring means acquires, as the information on the speaker's line of sight, object information on an object that the speaker is looking at; the determining means determines, based on the voice information and the object information, whether the speaker is speaking while looking at the object corresponding to the listener; The control means if the speaker is looking at the object corresponding to the listener but is not speaking, then not notifying the listener; When it is determined that the speaker is speaking while looking at the object corresponding to the listener, control is performed so as to notify the listener.

3. The electronic device according to claim 2.

4. The voice information indicates a time period during which the speaker is uttering the voice.

4. The electronic device according to claim 3.

5. The object information indicates a time period during which the speaker is viewing the object corresponding to the object information.

5. The electronic device according to claim 4.

6. The determining means determines whether the speaker is speaking while looking at the object corresponding to the listener, based on the object information indicating a time period corresponding to a time period indicated by the voice information.

6. The electronic device according to claim 5,

7. the display control means controls to display a plurality of objects corresponding to a plurality of listeners of the conversation to the speaker; The control means controls to issue the notification to a listener corresponding to an object that the speaker is looking at with a gaze intensity higher than a threshold, among the plurality of listeners, based on the information regarding the speaker's gaze.

2. The electronic device according to claim 1 .

8. The control means performs the notification when the number of participants in the conversation is not within a predetermined range. Not 2. The electronic device according to claim 1 .

9. The control means does not issue the notification when multiple participants of the conversation are emitting voices.

2. The electronic device according to claim 1 .

10. The control means performs control so as to stop the notification in response to the listener uttering a voice after starting the notification.

2. The electronic device according to claim 1 .

11. The control means controls to stop the notification in response to the listener seeing the speaker after starting the notification.

2. The electronic device according to claim 1 .

12. The control means controls the notification to be made after the speaker stops uttering a sound and a predetermined time has elapsed with no sound being uttered by any participant in the conversation.

2. The electronic device according to claim 1 .

13. The control means controls to stop the notification in response to the speaker stopping uttering a voice after starting the notification.

2. The electronic device according to claim 1 .

14. The control means controls to stop the notification in response to the speaker stopping looking at the listener after starting the notification.

2. The electronic device according to claim 1 .

15. The notification includes a notification of a position of an object corresponding to the speaker.

2. The electronic device according to claim 1 .

16. The notification includes notification of information that enables the speaker to be identified.

2. The electronic device according to claim 1 .

17. The control means controls to perform the notification by display.

2. The electronic device according to claim 1 .

18. The control means controls so as to give the notification by audio output.

2. The electronic device according to claim 1 .

19. A detection means for detecting the speaker's line of sight Further having 2. The electronic device according to claim 1 .

20. The electronic device is worn by the speaker.

2. The electronic device according to claim 1 .

21. controlling a display of objects corresponding to listeners of a conversation to a speaker of said conversation; acquiring information regarding the speaker's gaze; Before the speaker responds to the listener based on the information regarding the speaker's gaze a control step of controlling to notify the listener when the listener is viewing the object; 13. A method for controlling an electronic device comprising:

22. A program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 20.

23. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 20.

Images