Electronic device and information display system

Abstract

The present invention relates to an electronic device and an information display system that mitigate discordance in appearance when a plurality of users are stereoscopically displayed. An electronic device that displays information includes: a display having a display surface; and a controller. The first user image is an image representing a first user that is a user of a first electronic device communicating with the electronic device. The second user image is an image representing a second user that is a user of a second electronic device communicating with the electronic device. The controller executes stereoscopic display processing for displaying the first user image and the second user image on the display such that the first user and the second user appear on the display surface. In a fixed mode in which the first user is set as a fixed user, the controller executes stereoscopic display processing such that the first user appears closest to the display surface and the second user appears in front of the first user.

Description

Technical Field

[0001] The present disclosure relates to an information display technology using an electronic device. Background Art

[0002] Patent Document 1 discloses a remote conferencing system. The remote conferencing system displays images of a plurality of users on a display screen. In addition, the remote conferencing system sets seat arrangements of the plurality of users in a virtual space based on attribute information such as the positions of the respective users.

[0003] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2022-054192 Summary of the Invention

[0004] An object of the present disclosure is to provide a technique capable of reducing the apparent incongruity when a plurality of users are three-dimensionally displayed.

[0005] A first aspect relates to an electronic device that performs information display.

[0006] The electronic device includes: a display having a display surface; and a controller.

[0007] The first user image is an image representing a user of a first electronic device that communicates with the electronic device, that is, a first user.

[0008] The second user image is an image representing a user of a second electronic device that communicates with the electronic device, that is, a second user.

[0009] The controller acquires the first user image and the second user image.

[0010] The controller performs a three-dimensional display process of displaying the first user image and the second user image on the display so that the first user and the second user appear to emerge from the display surface.

[0011] In a fixed mode in which the first user is set as a fixed user, the controller performs a three-dimensional display process so that the first user appears closest to the display surface and the second user is in front of the first user.

[0012] A second aspect relates to an information display system.

[0013] The information display system includes a plurality of electronic devices connected to each other via a communication network.

[0014] Each of the plurality of electronic devices includes: a display having a display surface; and a controller.

[0015] The first user image is an image representing a user of a first electronic device among the plurality of electronic devices, that is, a first user.

[0016] The second user image is an image representing the user of the second electronic device among a plurality of electronic devices, that is, the second user.

[0017] The controller of the third electronic device among the plurality of electronic devices acquires the first user image and the second user image.

[0018] The controller of the third electronic device performs a stereoscopic display process of displaying the first user image and the second user image on the display in such a way that the first user and the second user appear to emerge from the display surface.

[0019] In the fixed mode in which the first user is set as the fixed user, the controller of the third electronic device performs the stereoscopic display process so that the first user appears to be closest to the display surface and the second user is in front of the first user.

[0020] The third viewpoint relates to an information display method in an electronic device having a display with a display surface.

[0021] The first user image is an image representing the user of the first electronic device communicating with the electronic device, that is, the first user.

[0022] The second user image is an image representing the user of the second electronic device communicating with the electronic device, that is, the second user.

[0023] The information display method includes:

[0024] acquiring the first user image and the second user image;

[0025] performing a stereoscopic display process of displaying the first user image and the second user image on the display in such a way that the first user and the second user appear to emerge from the display surface; and

[0026] in the fixed mode in which the first user is set as the fixed user, performing the stereoscopic display process so that the first user appears to be closest to the display surface and the second user is in front of the first user.

[0027] The fourth viewpoint relates to an information display program. The information display program causes the electronic device to execute the information display method related to the third viewpoint. The information display program can also be recorded on a computer-readable recording medium.

[0028] According to the present disclosure, in the fixed mode in which the first user is set as the fixed user, a stereoscopic display process is performed so that the first user (fixed user) appears to be closest to the display surface and the other second user is in front of the first user. Since the front-back relationship between the first user (fixed user) and the other second user is shown, the apparent incongruity when stereoscopically displaying a plurality of users is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1It is a schematic diagram for explaining the outline of an information display system.

[0030] Figure 2 It is a block diagram showing a structural example of an electronic device.

[0031] Figure 3 It is a block diagram showing a functional structural example of a controller of an electronic device.

[0032] Figure 4 It is a schematic diagram showing the relative positional relationship between the left and right eyes of a user and a display surface.

[0033] Figure 5 It is a schematic diagram for explaining an example of stereoscopic display processing in a fixed mode.

[0034] Figure 6 It is a schematic diagram for explaining another example of stereoscopic display processing in a fixed mode.

[0035] Explanation of reference numerals

[0036] 1... Information display system; 10... Server; 100... Electronic device; 110... Display; 111... Display surface; 120... Camera; 150... Controller; 151... Processor; 152... Memory. Detailed implementation manners

[0037] With reference to the accompanying drawings, the implementation manners of the present disclosure will be described.

[0038] 1. Outline of the information display system

[0039] Figure 1 It is a schematic diagram for explaining the outline of the information display system 1 according to the present embodiment. The information display system 1 includes a plurality of electronic devices 100. As the electronic device 100, a personal computer, a tablet, a dedicated terminal, etc. are exemplified. The plurality of electronic devices 100 are respectively used by a plurality of users U. In Figure 1 the example shown, the electronic devices 100 - A, 100 - B, 100 - C, and 100 - D and the respective users U - A, U - B, U - C, and U - D are shown. Among them, the number of the electronic devices 100 and the number of the users U are arbitrary.

[0040] The plurality of electronic devices 100 are connected to each other via a communication network and can communicate with each other. The communication network can be a wireless communication network or a wired communication network. The plurality of electronic devices 100 can also be connected to the server 10 and communicate with each other via the server 10.

[0041] In addition, multiple electronic devices 100 each include a camera (imaging device) and a display (display device). The camera is set to be able to image the user U. Typically, the camera is set to be able to image the face of the user U.

[0042] The electronic device 100-i (i = A, B, C, D...) acquires an image IMG-i of the user U-i captured by the camera. The image IMG-i at least includes the face of the user U-i. The electronic device 100-i transmits the image IMG-i to other electronic devices 100-j (j ≠ i). The electronic device 100-i may also extract feature parameters of the user U-i based on the image IMG-i and transmit information on the feature parameters of the user U-i to other electronic devices 100-j. Examples of the feature parameters include the line-of-sight information, fixation point, expression parameters, and posture parameters of the user U-i. Typically, the electronic device 100-i transmits at least one of the image IMG-i and the feature parameters to other electronic devices 100-j in real time.

[0043] The electronic device 100-j receives the information transmitted from the electronic device 100-i. The electronic device 100-j acquires a user image UIMG-i representing the user U-i based on at least one of the image IMG-i and the feature parameters. The user image UIMG-i may be the original image IMG-i or an avatar image. The avatar image is generated, for example, by applying at least one of the original image IMG-i and the feature parameters to a specified character model. The electronic device 100-j displays the user image UIMG-i representing the user U-i on the display. The user U-j of the electronic device 100-j can observe the user image UIMG-i displayed on the display. That is, the electronic device 100-j displays the user image UIMG-i of the user U-i of the other electronic device 100-i that is the communication partner, and the user U-j can see the user image UIMG-i of the other user U-i that is the communication partner.

[0044] The information display system 1 according to the present embodiment can be applied, for example, to online learning, online teaching, online meetings, etc. In these cases, the information display system 1 can also be referred to as an online learning system, an online teaching system, an online meeting system, etc.

[0045] 2. Structural example of the electronic device

[0046] Figure 2 is a block diagram showing a structural example of the electronic device 100. In Figure 2 the example shown, the electronic device 100 includes a display 110, one or more cameras 120 (hereinafter simply referred to as the camera 120), a user interface 130, a communication interface 140, and a controller 150.

[0047] The display 110 is configured to display various information. Examples of the display 110 include a liquid crystal display, an organic EL display, a head-up display (HUD), etc. The display 110 has a display surface (display unit) 111. Typically, the user U of the electronic device 100 uses the electronic device 100 with the display surface 111 of the display 110 facing each other.

[0048] The camera 120 is set to be able to photograph the user U of the electronic device 100. Typically, the camera 120 is set to be able to photograph the face of the user U of the electronic device 100. For example, the camera 120 is provided around the display surface 111 of the display 110.

[0049] The user interface 130 includes an input device that receives input of various information from the user U. Examples of the input device include a touch panel, a keyboard, a mouse, a microphone, etc. The user interface 130 may also include an output device such as a speaker.

[0050] The communication interface 140 is connected to a communication network and communicates with the outside of the electronic device 100. The communication network may be a wireless communication network or a wired communication network. For example, the communication interface 140 communicates with the server 10 and other electronic devices 100.

[0051] The controller 150 performs various information processing. The controller 150 can also be referred to as a processing circuit. The processing circuit is hardware programmed to implement a function or hardware that executes a function.

[0052] For example, the controller 150 includes one or more processors 151 (hereinafter simply referred to as the processor 151) and one or more memories 152 (hereinafter simply referred to as the memory 152). The processor 151 performs various information processes. Examples of the processor 151 include a general-purpose processor, a special-purpose processor, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), an integrated circuit, a conventional circuit, and / or a combination thereof. The memory 152 stores various information. Examples of the memory 152 include a volatile memory, a non-volatile memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc. The control program is a computer program executed by the processor 151. The functions of the controller 150 can also be implemented by the cooperation of the processor 151 that executes the control program and the memory 152. The control program is stored in the memory 152. Alternatively, the control program can also be recorded on a computer-readable recording medium.

[0053] Figure 3 is a block diagram showing an example of the functional structure of the controller 150. In Figure 3 the example shown, the controller 150 includes a face / eye extraction unit 161, a positional relationship calculation unit 162, a feature extraction unit 163, a transmission unit 164, a reception unit 171, a rendering unit 172, and a display processing unit 173 as functional modules.

[0054] The face / eye extraction unit 161 acquires an image IMG of the user U captured by the camera 120. The image IMG includes at least the face of the user U. Typically, the face / eye extraction unit 161 acquires the image IMG in real time. Moreover, the face / eye extraction unit 161 extracts a face image of the face portion of the user U from the image IMG. And the face / eye extraction unit 161 can also extract images of the left and right eyes of the user U from the image IMG and the face image. The extraction of the face and eye images is performed, for example, by using a machine learning model pre-generated through machine learning.

[0055] The position relationship calculation unit 162 calculates the relative position relationship between the left and right eyes of the user U and the display surface 111 (display unit) of the display 110. More specifically, the position relationship calculation unit 162 acquires the image IMG of the user U captured by the camera 120. In addition, the position relationship calculation unit 162 grasps the position and size of the face of the user U in the image IMG and the positions of the left and right eyes based on the extraction results of the face / eye extraction unit 161. The installation position, installation orientation, and viewing angle of the camera 120 are known information. The size of the face of the general user U is also known information. The position relationship calculation unit 162 calculates (infers) the relative position relationship between the camera 120 and the left and right eyes of the user U based on these known information, the position and size of the face of the user U in the image IMG, and the positions of the left and right eyes. As another example, the distance between the camera 120 and the user U can also be calculated based on the blurring state of the image IMG. In addition, the relative position relationship between the camera 120 and the display surface 111 of the display 110 is also known information. The position relationship calculation unit 162 calculates the relative position relationship between the left and right eyes of the user U and the display surface 111 by combining the relative position relationship between the camera 120 and the left and right eyes of the user U and the relative position relationship between the camera 120 and the display surface 111 of the display 110.

[0056] Figure 4 It is a schematic diagram showing the relative position relationship between the left and right eyes of the user U and the display surface 111 of the display 110. The relative position relationship at least includes the distance Dl between the left eye and the display surface 111, and the distance Dr between the right eye and the display surface 111. The relative position relationship may also include the distance Dw between the left eye and the right eye.

[0057] The feature extraction unit 163 extracts the feature parameters of the user U. As the feature parameters, examples include the gaze information, fixation point, expression parameters, posture parameters, etc. of the user U. The feature extraction unit 163 extracts the feature parameters of the user U based on the images of the left and right eyes and the face image of the user U.

[0058] For example, the feature extraction unit 163 identifies the positions and rotation angles of the pupils (black eyeballs) of each eye based on the images of the left and right eyes of the user U. Moreover, the feature extraction unit 163 identifies the gaze directions of each eye based on the positions and rotation angles of the pupils of each eye. The gaze information includes the gaze directions of each eye. The feature extraction unit 163 may also calculate the coordinate information of the point where the gaze intersects the display surface 111 considering the coordinate system on the display surface 111. As the gaze information of the user U, the coordinate information of the point where the gaze intersects the display surface 111 may also be given.

[0059] The feature extraction unit 163 may also calculate the point where the line of sight of the left eye of the user U intersects with the line of sight of the right eye. The point where the line of sight of the left eye intersects with the line of sight of the right eye is also called the fixation point. If the pupil approaches the inside, the fixation point gets closer, and if the pupil approaches the outside, the fixation point gets farther. This fixation point corresponds to the "gaze point" that the user U is looking at. In addition, not only the real-time information of the gaze point is used, but also the statistical information of the gaze point within a certain period can be used.

[0060] The feature extraction unit 163 may also extract an expression parameter representing the expression of the user U based on the face image of the user U. The extraction of the expression parameter is performed, for example, by using a machine learning model pre-generated through machine learning.

[0061] The feature extraction unit 163 may also extract a pose parameter (such as the angle of the face, etc.) representing the pose of the user U based on the face image of the user U or the image IMG. The extraction of the pose parameter is performed, for example, by using a machine learning model pre-generated through machine learning.

[0062] The sending unit 164 sends information related to the user U to other electronic devices 100 via the communication interface 140. Typically, the sending unit 164 sends real-time information related to the user U to other electronic devices 100. The sent information includes at least one of at least a part of the image IMG, the face image, and the feature parameters of the user U.

[0063] The receiving unit 171 receives information related to other users U (for convenience, called user U-i) from other electronic devices 100 via the communication interface 140. The received information includes at least one of at least a part of the image IMG-i, the face image, and the feature parameters of the user U-i.

[0064] The rendering unit 172 obtains the received information from the receiving unit 171. The rendering unit 172 obtains a user image UIMG-i representing the user U-i by performing a rendering process based on the received information. For example, the user image UIMG-i is an avatar image. A prescribed three-dimensional character model is prepared in advance. The rendering unit 172 applies the face image or feature parameters (expression, pose) of the user U-i to the prescribed three-dimensional character model and performs a rendering process, thereby obtaining an avatar image representing the user U-i.

[0065] The display processing unit 173 displays the user image UIMG-i representing the user U-i on the display 110 (display surface 111). The user U of the electronic device 100 can see the displayed user image UIMG-i.

[0066] The display processing unit 173 can also display the user image UIMG-i on the display 110 (display surface 111) in such a way that it appears as if the user U-i emerges from the display surface 111. Hereinafter, this process will be referred to as "stereoscopic display processing". The display processing unit 173 performs stereoscopic display processing as needed. More specifically, the display processing unit 173 obtains information on the relative positional relationship between the left and right eyes of the user U and the display surface 111 from the above-mentioned positional relationship calculation unit 162. Moreover, the display processing unit 173 can perform stereoscopic display processing based on the relative positional relationship between the left and right eyes of the user U and the display surface 111.

[0067] For example, in Figure 4 , the position X on the display surface 111 is the position corresponding to the intermediate position between the left and right eyes of the user U. The range RNG on the display surface 111 is the range corresponding to the range between the left and right eyes of the user U and includes the position X. Based on the relative positional relationship between the left and right eyes of the user U and the display surface 111, the display processing unit 173 can identify the position X and the range RNG. Moreover, the display processing unit 173 displays the user image UIMG-i on the display surface 111 in such a way that it is accommodated within the range RNG. That is, the display processing unit 173 displays the user image UIMG-i between the left and right eyes of the user U. Thus, when observed from the user U, the user U-i appears to emerge from the display surface 111.

[0068] In addition, the stereoscopic display processing can also be implemented by other methods. For example, the stereoscopic display processing can be implemented by the lenticular lens method using lenticular lenses. In the case of the lenticular lens method, a target image for the left eye and a target image for the right eye are prepared in such a way that the target appears to emerge from the display surface 111 and are respectively displayed on the display surface 111.

[0069] 3. Stereoscopic display processing in the fixed mode

[0070] Figure 5 An example of the stereoscopic display processing in the electronic device 100-B used by the user U-B is schematically shown. The stereoscopic display processing is performed in such a way that multiple other users U-A, U-C, U-D other than the user U-B appear to emerge from the display surface 111. Typically, the stereoscopic display processing is performed in such a way that the apparent positions of the users U-A, U-C, U-D are on the line connecting the left and right eyes of the user U-B and the display surface 111.

[0071] Here, as a mode of the stereoscopic display processing, the "fixed mode" is considered. In the fixed mode, a specific one of the multiple users U is set as the "fixed user". The fixed user can be automatically set by the information display system 1 or specified by any user U. In the following description, the case where the user U-A is set as the fixed user is considered.

[0072] Figure 5 also schematically shows the "apparent positions" of users U-A, U-C, and U-D in the fixed mode. As Figure 5 shown, when observed from user U-B, user U-A, who appears to be a fixed user, is closest to the display surface 111, and the other users U-C and U-D are in front of user U-A. In other words, when observed from user U-B, the apparent position of user U-A, who is a fixed user, is closest to the display surface 111, and compared with user U-A, the apparent positions of the other users U-C and U-D are farther from the display surface 111. Since the front-back relationship between user U-A, who appears to be a fixed user, and the other users U-C and U-D is shown, the apparent incongruity when stereoscopically displaying multiple users U is reduced.

[0073] For example, information display system 1 is an online teaching system, user U-A is a teacher, and the other users U-B, U-C, and U-D are students. In this case, user U-A, who is a teacher, is set as the fixed user. When observed from user U-B, who is a student, it appears that the teacher is closest to the display surface 111, and the other students are in front of the teacher. This conforms to the actual teaching scenario. Therefore, the apparent incongruity when stereoscopically displaying multiple users U is reduced.

[0074] Generally speaking, as follows. The first user U-1 is a user of the first electronic device 100-1. The first user image UIMG-1 is an image representing the first user U-1. The second user U-2 is a user of the second electronic device 100-2. The second user image UIMG-2 is an image representing the second user U-2. The third user U-3 is a user of the third electronic device 100-3. The controller 150 of the third electronic device 100-3 acquires the first user image UIMG-1 and the second user image UIMG-2. The controller 150 of the third electronic device 100-3 performs a stereoscopic display process of displaying the first user image UIMG-1 and the second user image UIMG-2 on the display 110 (display surface 111) in such a way that the first user U-1 and the second user U-2 appear to emerge from the display surface 111.

[0075] The controller 150 of the third electronic device 100-3 acquires an image IMG of the third user U-3 through the camera 120. The controller 150 of the third electronic device 100-3 calculates the relative positional relationship between the left and right eyes of the third user U-3 and the display surface 111 based on the image IMG of the third user U-3. The controller 150 of the third electronic device 100-3 may also perform stereoscopic display processing based on the relative positional relationship between the left and right eyes of the third user U-3 and the display surface 111. The controller 150 of the third electronic device 100-3 may also perform stereoscopic display processing in such a manner that the apparent positions of the first user U-1 and the second user U-2 are on the line connecting the left and right eyes of the third user U-3 and the display surface 111.

[0076] In the fixed mode in which the first user U-1 is fixed as a fixed user, the controller 150 of the third electronic device 100-3 performs stereoscopic display processing in such a manner that it appears that the first user U-1 is closest to the display surface 111 and the second user U-2 is in front of the first user U-1. In other words, the controller 150 of the third electronic device 100-3 performs stereoscopic display processing so that the apparent position of the first user U-1 is closest to the display surface 111, and the apparent position of the second user U-2 is farther from the display surface 111 than that of the first user U-1. Thereby, since the front-back relationship between the first user U-1 (fixed user) and the other second user U-2 is shown, the apparent incongruity when stereoscopically displaying multiple users U is reduced.

[0077] Figure 6 It is a schematic diagram for explaining another example of the stereoscopic display processing in the fixed mode. In Figure 6 In the example shown, specific information is displayed in the first area 111X of the display surface 111.

[0078] The specific information is, for example, information generated or input by the user U-A who is the fixed user. For example, the specific information is an explanation (blackboard writing) written by the user U-A who is a teacher. For example, the user U-A writes an explanation on a prescribed object, and the camera 120 of the electronic device 100-A captures the writing of the explanation. As another example, the user U-A may also use the user interface 130 of the electronic device 100-A to input an explanation. The image of the writing of the captured explanation or the information of the writing of the input explanation becomes the specific information. The specific information is sent to the other electronic devices 100-B, 100-C, 100-D. The electronic devices 100-B, 100-C, 100-D respectively display the specific information in the first area 111X of the display surface 111 of the display 110.

[0079] If the user U-C or U-D, who appears to emerge from the display surface 111, blocks the first area 111X, the user U-B cannot see the specific information displayed in the first area 111X. Therefore, the display positions of the users U-C and U-D are adjusted. Specifically, the second area 111Y other than the first area 111X in the display surface 111 is considered. No specific information is displayed in the second area 111Y. When specific information is displayed in the first area 111X, stereoscopic display processing is performed in such a way that the apparent positions of the users U-C and U-D are on the line connecting the second area 111Y and the left and right eyes of the user U-B. Thereby, the user U-B can see the specific information displayed in the first area 111X.

[0080] Generally speaking, as follows. In the fixed mode, when specific information is displayed in the first area 111X in the display surface 111, the controller 150 of the third electronic device 100-3 performs stereoscopic display processing in such a way that the apparent position of the second user U-2 is on the line connecting the second area 111Y and the left and right eyes of the third user U-3. Thereby, the third user U-3 can see the specific information displayed in the first area 111X.

[0081] 4. Mode Switching

[0082] The mode of the stereoscopic display processing may also include a "fixed mode" in which a fixed user is set, and a "free mode" in which no fixed user is set. The fixed mode and the free mode can be switched. The user U can also use the user interface 130 to select (specify) one of the fixed mode and the free mode. The controller 150 receives information on the mode selected (specified) by the user U via the user interface 130. Moreover, the controller 150 performs stereoscopic display processing in the mode selected (specified) by the user U.

Claims

1. An electronic device, which is an electronic device for information display, wherein, the electronic device includes: a display having a display surface; and a controller, a first user image is an image representing a user of a first electronic device communicating with the electronic device, i.e., a first user, a second user image is an image representing a user of a second electronic device communicating with the electronic device, i.e., a second user, the controller is configured to: acquire the first user image and the second user image, perform a stereoscopic display process of displaying the first user image and the second user image on the display in such a manner that the first user and the second user appear to emerge from the display surface, in a fixed mode in which the first user is set as a fixed user, perform the stereoscopic display process so that the first user appears to be closest to the display surface and the second user is in front of the first user.

2. The electronic device according to claim 1, wherein, it further includes a camera, the controller is configured to: acquire an image of a third user who is a user of the electronic device through the camera, calculate a positional relationship between the left and right eyes of the third user and the display surface based on the image of the third user, perform the stereoscopic display process based on the positional relationship between the left and right eyes of the third user and the display surface.

3. The electronic device according to claim 2, wherein, in the fixed mode, the controller is configured to: perform the stereoscopic display process in such a manner that the apparent positions of the first user and the second user exist on the line connecting the left and right eyes of the third user and the display surface.

4. The electronic device according to claim 2, wherein, in the fixed mode, when specific information is displayed in a first area on the display surface, the controller is configured to: perform the stereoscopic display process in such a manner that the apparent position of the second user exists on the line connecting a second area outside the first area on the display surface and the left and right eyes of the third user.

5. An information display system, wherein, it includes a plurality of electronic devices interconnected via a communication network, each of the plurality of electronic devices includes: a display having a display surface; and a controller, a first user image is an image representing a user of a first electronic device among the plurality of electronic devices, i.e., a first user, a second user image is an image representing a user of a second electronic device among the plurality of electronic devices, i.e., a second user, the controller of a third electronic device among the plurality of electronic devices is configured to: acquire the first user image and the second user image, perform a stereoscopic display process of displaying the first user image and the second user image on the display in such a manner that the first user and the second user appear to emerge from the display surface, in a fixed mode in which the first user is set as a fixed user, perform the stereoscopic display process so that the first user appears to be closest to the display surface and the second user is in front of the first user.

Images