Information display system

Abstract

To properly express a motion of the head of a first user while clearly expressing where the first user is looking when displaying a first user image on an electronic device of a second user.SOLUTION: An information display system comprises a plurality of electronic devices connected to each other via a communication network. A first electronic device used by a first user acquires line-of-sight information, including a line-of-sight direction of the first user, and information in a head direction on the basis of a camera image, and transmits the line-of-sight direction and the head direction to a second electronic device. The second electronic device used by a second user displays a first user image, which is a three-dimensional image representing the first user, on a second display (user display processing). The user display processing includes: determining arrangement of the first user image on the second display on the basis of the line-of-sight information; and rotating the head of the first user image in accordance with rotation of the head of the first user with reference to a relationship between the line-of-sight direction and the head direction of the first user when the arrangement of the first user image is determined.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to information display technology using electronic devices. [Background technology]

[0002] Patent Document 1 discloses a video conference system provided between a local participant located at a local location and a remote participant located at a remote location. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4257298 Summary of the Invention [Problem to be solved by the invention]

[0004] Considering information display technology using multiple electronic devices that can communicate with each other, when a three-dimensional image (first user image) representing a first user is displayed on an electronic device of a second user, a technology is desired that can display the first user image in a way that clearly shows where the first user is looking and appropriately represents the first user's head movements. [Means for solving the problem]

[0005] The information display system according to the present disclosure includes a plurality of electronic devices connected to each other via a communication network. The plurality of electronic devices include a first electronic device including a first camera and a first display and used by a first user, and a second electronic device including a second camera and a second display and used by a second user. The first electronic device acquires gaze information including a gaze direction of the first user and head direction information based on an image of the first user captured by the first camera, and transmits the gaze information and head direction information to the second electronic device. The second electronic device executes a user display process to display a first user image, which is a three-dimensional image representing the first user, on the second display. The user display process includes determining an arrangement of the first user image on the second display based on the gaze information, and rotating the head of the first user image in accordance with a rotation of the first user's head, based on the relationship between the gaze direction and head direction of the first user when the arrangement of the first user image was determined. [Effects of the Invention]

[0006] According to the present disclosure, the arrangement of the first user image on the second display is determined based on the gaze information of the first user. Then, the head of the first user image is rotated in accordance with the rotation of the first user's head, based on the relationship between the gaze direction and head direction of the first user when the arrangement of the first user image is determined in this manner. This makes it possible to display the first user image in a way that clearly shows where the first user is looking and appropriately shows the movement of the first user's head. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a conceptual diagram for explaining an overview of an information display system according to an embodiment. [Figure 2] FIG. 1 is a block diagram illustrating an example of the configuration of an electronic device. [Figure 3] 10A and 10B are diagrams for explaining the expression of the movements of a user U using a user image UIMG in the fixed display mode. [Figure 4]FIG. 2 is a diagram showing a processing flow of the information display system according to the embodiment. [Figure 5] 10 is a diagram for explaining a method of expressing the non-verbal actions of a first user U-1 using a first user image UIMG-1. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0008] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0009] 1. Overview of the information display system FIG. 1 is a conceptual diagram for explaining an overview of an information display system 1 according to this embodiment. The information display system 1 includes a plurality of electronic devices 100. Examples of the electronic devices 100 include a personal computer, a tablet, and a dedicated terminal. The plurality of electronic devices 100 are used by a plurality of users U, respectively. In the example shown in FIG. 1, electronic devices 100-A, 100-B, 100-C, 100-D, and 100-E and their respective users UA, UB, UC, UD, and UE are shown. However, the number of electronic devices 100 and the number of users U are arbitrary.

[0010] The electronic devices 100 are connected to each other via a communication network and can communicate with each other. The communication network may be a wireless communication network or a wired communication network. The electronic devices 100 may be connected to a server 10 and communicate with each other via the server 10.

[0011] Each of the multiple electronic devices 100 includes a camera (image capture device) and a display (display device). The camera is installed so as to be able to capture an image of the user U. Typically, the camera is installed so as to be able to capture an image of the user U's face. The electronic device 100-i (i=A, B, C, D, E, etc.) acquires an image IMG-i of the user Ui captured by the camera. The image IMG-i includes at least the face of the user Ui.

[0012] The information display system 1 has, for example, an all-users display mode and a fixed display mode as display modes for supporting smooth communication between multiple users U. In the all-users display mode, multiple electronic devices 100 transmit and receive images IMG of users U acquired by the individual electronic devices 100 to acquire images (camera images) IMG of other users U and display them on the display. This allows the multiple users U to communicate while checking each other's facial expressions.

[0013] The fixed display mode is a mode in which one of multiple users U becomes a fixed user and is fixed like a background on the displays of the other users. For example, a user who wants to become a fixed user operates the electronic device 100 to select the fixed display mode. As a result, the display mode switches to the fixed display mode. Details of the fixed display mode will be described later with reference to FIG. 3.

[0014] The information display system 1 according to the present embodiment is applicable to, for example, online learning, online classes, online meetings, etc. In these cases, the information display system 1 can also be called an online learning system, an online class system, an online meeting system, etc.

[0015] 2. Example of electronic device configuration 2 is a block diagram showing a configuration example of the electronic device 100. In the example shown in FIG. 2, the electronic device 100 includes a display 110, one or more cameras 120 (hereinafter simply referred to as cameras 120), a user interface 130, a communication interface 140, and a controller 150.

[0016] The display 110 is configured to display various types of information. Examples of the display 110 include a liquid crystal display, an organic electroluminescence (EL) display, and a head-up display (HUD). The display 110 has a display surface (display unit) 111 (see FIG. 3). Typically, a user U of the electronic device 100 faces the display surface 111 of the display 110 when using the electronic device 100.

[0017] The camera 120 is installed so as to be able to capture an image of the user U of the electronic device 100. Typically, the camera 120 is installed so as to be able to capture an image of the face of the user U of the electronic device 100. For example, the camera 120 is provided around the display surface 111 of the display 110.

[0018] The user interface 130 includes an input device that accepts input of various information from the user U. Examples of the input device include a touch panel, a keyboard, a mouse, a microphone, etc. The user interface 130 may also include an output device such as a speaker.

[0019] The communication interface 140 is connected to a communication network and communicates with the outside of the electronic device 100. The communication network may be a wireless communication network or a wired communication network. For example, the communication interface 140 communicates with the server 10 or another electronic device 100.

[0020] The controller 150 performs various information processing operations. The controller 150 may also be called a processing circuitry. A processing circuit is hardware that is programmed to realize a function or hardware that executes a function.

[0021] For example, the controller 150 includes one or more processors 151 (hereinafter simply referred to as processors 151) and one or more memories 152 (hereinafter simply referred to as memories 152). The processor 151 performs various types of information processing. Examples of the processor 151 include a general-purpose processor, a special-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), an integrated circuit, a conventional circuit, and / or a combination thereof. The memory 152 stores various types of information. Examples of the memory 152 include a volatile memory, a non-volatile memory, a hard disk drive (HDD), a solid-state drive (SSD), etc. A control program is a computer program executed by the processor 151. The processor 151 executing the control program and the memory 152 may cooperate to realize the functions of the controller 150. The control program is stored in the memory 152. Alternatively, the control program may be recorded on a computer-readable recording medium.

[0022] 3. Representing user actions using user images The "user image UIMG" here is a three-dimensional image representing the user U, and may be, for example, an avatar image. The representation of the movements of the user U using the user image UIMG according to this embodiment is applied to, for example, a fixed display mode.

[0023] FIG. 3 is a diagram for explaining the expression of the user U's behavior using the user image UIMG in the fixed display mode. FIG. 3 shows a state in which the user UA is using the electronic device 100-A. On the display surface 111 of the display 110 of the electronic device 100-A that the user UA is looking at, the user UE of the five users shown in FIG. 1 is fixed as a fixed user like the background. When the user UE is a fixed user in this way, the remaining four users including the user UA are looking at the user UE. That is, like the user UA shown in FIG. 3, the users UB, UC, and UD are looking at the fixed user UE displayed on their respective displays 110.

[0024] While the fixed display mode is selected, as shown in Fig. 3, the camera image IMG-E of user UE, who is the fixed user, is displayed on the display 110 of user UA, but the camera images IMG-B, IMG-C, and IMG-D of the remaining users UB, UC, and UD are not displayed. In order to promote smooth communication between multiple users U while the fixed display mode is selected, it is possible to represent the movements of other users U whose camera images IMG are not displayed using user images UIMG. The movements of other users U here refer to head movements (non-verbal movements) such as "nodding," "tilting the head," and "shaking the head from side to side."

[0025] For ease of explanation, the following discussion will consider a first electronic device 100-1 used by a first user U-1 and a second electronic device 100-2 used by a second user U-2. The first user U-1 is a user among multiple users excluding fixed users who is displayed as a user image UIMG together with a camera image IMG of the fixed user. The second user U-2 is a user who views the user image (first user image) UIMG-1 of the first user U-1 together with the camera image IMG of the fixed user on the display 110. In the example shown in FIG. 3, each of users UB, UC, and UD corresponds to the first user U-1, and user UA corresponds to the second user U-2.

[0026] When attempting to represent the actions of another user U using the user image UIMG as described above, the following problem arises. Specifically, the second electronic device 100-2 may determine the head rotation direction of the first user image UIMG-1 based on the eye gaze direction of the first user U-1 acquired from the first electronic device 100-1. However, this method may not be able to represent non-verbal actions of the first user U-1, such as nodding or tilting their head. Alternatively, the second electronic device 100-2 may simply determine the head rotation direction of the first user image UIMG-1 based on the head direction of the first user U-1 acquired from the first electronic device 100-1. However, this method may accurately represent non-verbal actions, but may make it difficult for the second user U-2, who is looking at the first user image UIMG-1, to understand where the first user U-1 is looking. Therefore, when displaying the first user image UIMG-1 on the second display 110-2 of the second electronic device 100-2, a technology is desired that can display the first user image UIMG-1 in a way that clearly shows where the first user U-1 is looking while also appropriately showing the head movement of the first user U-1.

[0027] In view of the above-described problems, in this embodiment, a "user display process" for displaying the first user image UIMG-1 on the second display 110-2 is executed by the second electronic device 100-2 as will be described with reference to Fig. 4. Fig. 4 is a diagram showing the flow of processing in the information display system 1 according to this embodiment.

[0028] When any user (a third user other than the first and second users U-1 and U-2) using the electronic device 100 selects the fixed display mode, the information display system 1 executes the fixed display mode (step S10). Specifically, the electronic devices 100 of users other than the user (fixed user) who selected the fixed display mode acquire camera images IMG of the fixed users from the electronic devices 100 of the fixed users. Then, the controllers 150 of each electronic device 100, including the electronic device 100 of the fixed users, display the camera images IMG of the fixed users on the displays 110. At this time, the fixed users are displayed with their faces facing the display 110 (for example, user UE in FIG. 3), and the other users are looking at the fixed users.

[0029] During execution of the fixed display mode, the second electronic device 100-2 executes a user display process for displaying the first user image UIMG-1 on the second display 110-2 as follows: The second electronic device 100-2 determines the arrangement of the first user image UIMG-1 on the second display 110-2 based on gaze information of the first user U-1 (step S20). Then, the second electronic device 100-2 rotates the head of the first user image UIMG-1 in accordance with the rotation of the head of the first user U-1, based on the relationship between the gaze direction and head direction of the first user U-1 when the arrangement of the first user image UIMG-1 was determined (S30).

[0030] The processes of steps S20 and S30 will be specifically described below. Note that the processes of steps S20 and S30 are repeatedly executed while the fixed display mode is being executed.

[0031] The user display process uses gaze information and head direction information of the first user U-1. The gaze information and head direction information are acquired by the first electronic device 100-1 based on a camera image IMG-1 of the first user U-1 captured by the camera (first camera) 120 of the first electronic device 100-1. The gaze information includes information on the gaze direction and point of gaze of the first user U-1. For example, the first electronic device 100-1 (controller 150) detects the gaze direction based on the position and rotation angle of the pupils of each eye of the first user U-1 in the camera image IMG-1. Furthermore, for example, the first electronic device 100-1 detects the point (focus point) where the gazes of the right eye and left eye of the first user U-1 intersect in the camera image IMG-1 as the point of gaze (the position where the first user U-1 is looking). Alternatively, the first electronic device 100-1 may calculate coordinate information of the point where the line of sight intersects with the display surface 111, taking into account a coordinate system on the display surface 111. The coordinate information of the point where the line of sight intersects with the display surface 111 may be used as information on the gaze point of the first user U-1. Furthermore, the first electronic device 100-1 detects the head direction (head orientation) of the first user U-1 based on the camera image IMG-1. The first electronic device 100-1 transmits the gaze information (gaze direction and gaze point) and head direction information thus acquired to the second electronic device 100-2.

[0032] In step S20, the second electronic device 100-2 generates a first user image UIMG-1 and then determines the placement of the first user image UIMG-1 on the second display 110-2 based on the gaze information of the first user U-1. The second electronic device 100-2 generates the first user image UIMG-1, for example, as follows: Image information other than the face of each user U-1 is stored in at least one of the memory 152 of each electronic device 100 and the memory of the server 10. The second electronic device 100-2 generates the first user image UIMG-1 by performing a rendering process based on the image information other than the face of the first user U-1. For example, like user images UIMG-B, UIMG-C, and UIMG-D of users UB, UC, and UD shown in FIG. 3 , the first user image UIMG-1 is generated as a three-dimensional image of the back of the head, neck, and back of the first user U-1 and displayed on the second display 110-2.

[0033] In addition, because each person's face is different, including facial image information as image information for generating a three-dimensional user image UIMG increases the image processing load. In contrast, by using image information other than facial information as described above, the user image UIMG can be generated while reducing the image processing load. Furthermore, according to the above-described technique, non-facial image information is not directly acquired from each user's electronic device 100 during execution of the fixed display mode, thereby reducing the communication load between the electronic devices 100. In other words, a decrease in communication speed can be suppressed. Therefore, each user can communicate with other users without stress during execution of the fixed display mode.

[0034] In step S20, the arrangement of the first user image UIMG-1 based on the gaze information of the first user U-1 is determined, for example, as follows: That is, the gaze information transmitted from the first electronic device 100-1 to the second electronic device 100-2 includes information about the first target as information about the first user U-1's point of gaze. Based on the gaze direction and gaze point of the first user U-1, the first electronic device 100-1 (controller 150) recognizes that the first user U-1 is looking at the "first target" displayed on the display 110 of the first electronic device 100-1. Because the fixed display mode is being executed, the first target is basically the fixed user displayed on the display 110 of the first electronic device 100-1. However, the first target may also be a predetermined object on the display 110 on which the fixed user is displayed. In an example in which the fixed user is the first target, the fixed user is identified as the first target for the first user U-1. The line-of-sight information transmitted from the first electronic device 100-1 to the second electronic device 100-2 also includes the image viewed by the first user U-1, that is, the first target image.

[0035] In the user display process, the second electronic device 100-2 (controller 150) acquires a first target image from the first electronic device 100-1. Then, based on line-of-sight information including information about the first target, the second electronic device 100-2 determines the placement of the first user image UIMG-1 relative to the first target image so that the first user U-1 appears to be gazing at the first target. In other words, the second electronic device 100-2 displays the first user image UIMG-1 and the image of the first target on the second display 110-2 so that the first user U-1 appears to be gazing at the first target. In an example (see FIG. 3) where the first user U-1 is each of users UB, UC, and UD, the first target is user UE, and the second user U-2 is user UA, the electronic device 100-A (second electronic device 100-2) displays user images UIMG-B, UIMG-C, and UIMG-D and the camera image IMG-E of user UE on the display 110-A (second display 110-2) of the electronic device 100-A so that each of users UB, UC, and UD appear to be gazing at the user UE.

[0036] The basic posture of the first user U-1 to be displayed on the second display 110-2 is determined by the processing of step S20 described above. Then, while the first user U-1 is gazing at the first target, the first user U-1 may perform non-verbal actions such as a "nodding action," a "head tilting action," or a "head shaking action." To represent such non-verbal actions of the first user U-1 using the first user image UIMG-1, in step S30 the second electronic device 100-2 rotates the first user image UIMG-1 in accordance with the rotation of the head of the first user U-1.

[0037] In step S30, as described above, the relationship between the gaze direction and head direction of the first user U-1 when the placement of the first user image UIMG-1 was determined in step S20 is used as the basis for rotating the first user image UIMG-1.

[0038] Specifically, FIG. 5 is a diagram illustrating a method for expressing the non-verbal behavior of the first user U-1 using the first user image UIMG-1. Here, the description will be given using user UB (see FIG. 3) as an example of the first user U-1. FIG. 5 illustrates the gaze direction DR1 and head direction DR2 of the first user U-1 when the placement of the first user image UIMG-1 is determined in step S20, as well as the roll axis, pitch axis, and yaw axis as rotation axes representing the rotation of the head of the first user image UIMG-1 in three dimensions. As shown in FIG. 5, the roll axis is parallel to the head direction DR2. The pitch axis and yaw axis are parallel to the left-right and up-down directions of the head of the first user image UIMG-1, respectively, when the placement of the first user image UIMG-1 is determined in step S20. In this way, these three axes are set based on the head posture of the first user image UIMG-1 when the placement is determined. That is, the coordinate system having these three axes is used as a reference coordinate system for the rotation of the head of the first user image UIMG-1 in step S30.

[0039] Then, in step S30, the second electronic device 100-2 (controller 150) calculates a rotation angle difference of the head of the first user U-1 relative to the head direction DR2 for each of the above three axes (roll axis, pitch axis, and yaw axis) based on information about the head direction of the first user U-1 acquired from the first electronic device 100-1. Then, the second electronic device 100-2 rotates the first user image UIMG-1 displayed on the second display 110-2 based on the rotation angle difference for each of the above three axes. As a result, non-verbal actions of the first user U-1, such as a "nodding action," "tilting action," and "shaking action," are expressed using the first user image UIMG-1. Such rotational movements of the first user image UIMG-1 can be realized using, for example, inverse kinematics.

[0040] Additionally, the second electronic device 100-2 may display the first user image UIMG-1 on the second display 110-2 (display surface 111) so that the first user U-1 appears to float above the display surface 111.

[0041] As described above, according to the user display process of the present embodiment, the arrangement of first user image UIMG-1 on second display 110-2 is determined based on line-of-sight information of first user U-1. That is, the basic line of sight of first user image UIMG-1 is identified. Then, based on the relationship between line-of-sight direction DR1 and head direction DR2 of first user U-1 when the arrangement of first user image UIMG-1 is determined in this manner, the head of first user image UIMG-1 rotates in response to the rotation of the head of first user U-1. As a result, when first user image UIMG-1 is displayed on second display 110-2, it is possible to display first user image UIMG-1 in a manner that clearly shows where first user U-1 is looking and appropriately depicts the head movement of first user U-1.

[0042] More specifically, in this embodiment, the determination of the arrangement of the first user image UIMG-1 based on the gaze information is performed as follows: That is, based on the gaze information including the information about the above-described first target, the arrangement of the first user image UIMG-1 relative to the first target image is determined so that the first user U-1 appears to be gazing at the first target. This makes it possible to accurately express non-verbal actions while reliably preventing the second user U-2 from becoming confused about where the first user U-1 is looking.

[0043] 4. Further examples of user display processing Furthermore, if the size of the first user image UIMG-1 displayed on the second display 110-2 is small, non-verbal actions such as nodding by the first user U-1 may appear small. Therefore, if the size of the first user image UIMG-1 displayed on the second display 110-2 is less than a threshold, the second electronic device 100-2 may increase the head rotation angle of the first user image UIMG-1 when rotating it in response to the head rotation of the first user U-1 compared to when the size is equal to or greater than the threshold. In other words, non-verbal actions may be exaggerated. This allows non-verbal actions to be appropriately expressed taking into account the size of the first user image UIMG-1. Additionally, the size of the first user image UIMG-1 includes, for example, the lengths of the first user image UIMG-1 in both the vertical and horizontal directions of the second display 110-2.

[0044] Furthermore, the first electronic device 100-1 may obtain the distance between the first display 110-1 and the first user U-1 and transmit the distance to the second electronic device 100-2. The distance may be obtained, for example, based on a camera image of the first user U-1. The second electronic device 100-2 may then display the first user image UIMG-1 on the second display 110-2 so that the shorter the first distance, the larger the image. This allows the perspective of the first user image UIMG-1 displayed on the second display 110-2 to be appropriately expressed.

[0045] Furthermore, the second electronic device 100-2 may display an arrow image indicating the direction from the first user image UIMG-1 to the first target on the second display 110-2. In the example shown in Fig. 3, arrow images (arrow objects) AR indicating the direction from each of the user images UIMG-B, UIMG-C, and UIMG-D of users UB, UC, and UD to the first target (the camera image IMG of the fixed user UE) are displayed on the display 110-A of the electronic device 100-A of user UA (second user U-2). This makes it easier for the second user U-2 (user UA in Fig. 3) to understand the situation of the first user U-1 (each of users UB, UC, and UD in Fig. 3). [Explanation of symbols]

[0046] 1 Information display system, 10 Server, 100 Electronic device, 110 Display, 120 Camera, 150 Controller

Claims

1. a plurality of electronic devices connected to each other via a communications network; the plurality of electronic devices a first electronic device including a first camera and a first display, the first electronic device being used by a first user; a second electronic device including a second camera and a second display, the second electronic device being used by a second user; Including, The first electronic device is acquiring line-of-sight information including a line-of-sight direction of the first user and information about a head direction of the first user based on an image of the first user captured by the first camera; transmitting the line of sight information and the head direction information to the second electronic device; the second electronic device executes a user display process to display a first user image, which is a three-dimensional image representing the first user, on the second display; The user display process includes: determining an arrangement of the first user image on the second display based on the line-of-sight information; Rotating the head of the first user image in accordance with a rotation of the head of the first user, based on a relationship between the line of sight direction and the head direction of the first user when the arrangement of the first user image is determined; Contains Information display system.

2. 2. The information display system according to claim 1, the line-of-sight information includes information on a line-of-sight direction and a gaze point of the first user, The first electronic device is Recognizing that the first user is looking at a first target displayed on the first display based on the line of sight direction and the gaze point; transmitting the line of sight information including information about the first target as information about the gaze point to the second electronic device; In the user display process, the second electronic device: acquiring a first target image representing the first target; Based on the line-of-sight information, a position of the first user image relative to the first target image is determined so that the first user appears to be gazing at the first target. Information display system.

3. 3. The information display system according to claim 1 or 2, When the size of the first user image displayed on the second display is less than a threshold, the second electronic device increases the rotation angle of the head of the first user image when rotating in accordance with the rotation of the head of the first user compared to when the size is equal to or greater than the threshold. Information display system.

4. 3. The information display system according to claim 1 or 2, the first electronic device acquires a distance between the first display and the first user and transmits the acquired distance to the second electronic device; The second electronic device displays the first user image on the second display so that the first user image becomes larger as the distance becomes shorter. Information display system.

5. 3. The information display system according to claim 2, The second electronic device displays an arrow image on the second display, which indicates a direction from the first user image to the first target. Information display system.

Images