Display device and display method

Abstract

To properly achieve the coincidence of lines of sight between users in remote conversation accompanied with images.SOLUTION: A display device includes: a display section displaying images; a plurality of imaging sections provided on a display screen of the display section and capturing images of a user located in a direction of the display screen; a display control section allowing the display section to display images of another user talking with the user; a face detection section detecting a face including the eyes of the other user from the images of the other user displayed on the display section; an imaging control section controlling the imaging section to capture images of the user, the imaging section being provided in the vicinity of an area of the eyes of the other user displayed on the display section; and a communication control section transmitting images of the user captured by the imaging section provided in the vicinity of the area of the eyes of the other user.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a display device and a display method. [Background technology]

[0002] There are known techniques for realizing remote conversations involving video such as video conferences. For example, Patent Literature 1 discloses a technique for making users' gazes meet in a video conference or video conversation system. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-37722 A Summary of the Invention [Problem to be solved by the invention]

[0004] In a video conference or the like, for example, a situation is assumed in which a user speaks while moving in front of a whiteboard, etc. In such a case, with the technology described in Patent Document 1, it may be difficult for the users to make eye contact with each other.

[0005] An object of the present invention is to provide a display device and a display method that enable users to appropriately meet each other's gazes during a remote conversation involving video. [Means for solving the problem]

[0006] The display device of the present invention comprises a display unit that displays images, a plurality of imaging units that are provided on the display screen of the display unit and capture images of users positioned in the direction of the display screen, a display control unit that causes images of other users conversing with the user to be displayed on the display unit, a face detection unit that detects the face including the eyes of the other users from the images of the other users displayed on the display unit, an imaging control unit that controls the imaging units provided in the vicinity of the eye areas of the other users displayed on the display unit to capture images of the users, and a communication control unit that transmits the images of the users captured by the imaging units provided in the vicinity of the eye areas of the other users.

[0007] The display method of the present invention includes the steps of: displaying an image on a display unit; causing a plurality of imaging units provided on a display screen of the display unit to capture an image of a user located in the direction of the display screen; displaying an image of another user conversing with the user on the display unit; detecting the face including the eyes of the other user from the image of the other user displayed on the display unit; controlling the imaging units provided near the eye area of ​​the other user displayed on the display unit to capture an image of the user; and transmitting the image of the user captured by the imaging units provided near the eye area of ​​the other user. Effect of the Invention

[0008] According to the present invention, it is possible for users to appropriately meet each other's gazes during a remote conversation accompanied by video. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a display system according to the first embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of the display device according to the first embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of the configuration of a display unit according to the first embodiment. [Figure 4]FIG. 4 is a diagram for explaining an example of the trimming process according to the first embodiment. [Diagram 5] FIG. 5 is a diagram for explaining an outline of the processing contents of the display device according to the first embodiment. [Figure 6] FIG. 6 is a flowchart showing the processing contents of the display device according to the first embodiment. [Figure 7] FIG. 7 is a diagram for explaining an image displayed on the display unit according to the first embodiment. [Figure 8] FIG. 8 is a diagram showing an example of the configuration of a display unit according to a first modified example of the first embodiment. [Figure 9] FIG. 9 is a diagram showing an example of the configuration of a display unit according to a second modified example of the first embodiment. [Figure 10] FIG. 10 is a flowchart showing the processing contents of the display device according to the second embodiment. [Figure 11] FIG. 11 is a diagram for explaining an outline of the processing content of the display device according to the third embodiment. [Figure 12] FIG. 12 is a flowchart showing the processing contents of the display device according to the third embodiment. [Figure 13] FIG. 13 is a diagram for explaining an image displayed on a display unit according to the third embodiment. [Figure 14] FIG. 14 is a block diagram showing an example of the configuration of a display device according to the fourth embodiment. [Figure 15] FIG. 15 is a diagram for explaining an outline of the processing contents of the display device according to the fourth embodiment. [Figure 16] FIG. 16 is a flowchart showing the processing contents of the display device according to the fourth embodiment. [Figure 17] FIG. 17 is a block diagram showing an example of the configuration of a display device according to the fifth embodiment. [Figure 18] FIG. 18 is a diagram for explaining a method of arranging a microphone unit according to the fifth embodiment. [Figure 19] FIG. 19 is a flowchart showing the processing contents of the display device according to the fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to this embodiment, and in the following embodiments, the same components are designated by the same reference numerals, and duplicated explanations will be omitted.

[0011] [First embodiment] (Display System) An example of the configuration of a display system according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of a display system according to the first embodiment.

[0012] As shown in Fig. 1, the display system 1 includes a display device 10-1 and a display device 10-2. The display device 10-1 and the display device 10-2 are communicatively connected via a wired or wireless network N. The network N is, for example, the Internet, but is not limited thereto. The display device 10-1 and the display device 10-2 may be collectively referred to as a display device 10.

[0013] The display device 10-1 is a device used by a user. The display device 10-2 is a device used by another user. The user and the other user can have a remote conversation accompanied by a video by using the display device 10-1 and the display device 10-2. The display device 10-1 and the display device 10-2 of this embodiment can realize a remote conversation accompanied by a video, in which the line of sight of the user and the other user can be aligned with each other.

[0014] (display device) A configuration example of the display device according to the first embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing a configuration example of the display device according to the first embodiment.

[0015] As shown in FIG. 2, the display device 10 includes an input unit 20, an imaging unit 22, a display unit 24, a microphone unit 26, an audio output unit 28, a memory unit 30, a communication unit 32, an encoder 34, a decoder 36, and a control unit 38.

[0016] The input unit 20 accepts various operations for the display device 10. The input unit 20 is realized by a switch, a button, a touch panel, or the like.

[0017] The imaging unit 22 captures an image. A plurality of imaging units 22 are provided on the display screen of the display unit 24. The imaging units 22 are provided so as to capture an image in the direction of the display screen of the display unit 24. The imaging unit 22 captures, for example, a moving image. The imaging unit 22 may be, for example, a camera including an optical element and an imaging element.

[0018] The display unit 24 displays various images. For example, the display unit 24 displays images of other users acquired from other display devices. The display unit 24 is, for example, a large LED (Light Emitting Diode) display.

[0019] FIG. 3 is a diagram showing an example of the configuration of the display unit 24 according to the first embodiment. As shown in FIG. 3, the display unit 24 has a plurality of imaging units 22 arranged facing the direction of the display screen. The plurality of imaging units 22 are arranged at equal intervals along the X direction and the Y direction in the display unit 24. The plurality of imaging units 22 are arranged, for example, by replacing the plurality of LED elements arranged on the display screen of the display unit 24. It is preferable that the plurality of imaging units 22 are arranged, for example, by replacing the plurality of LED elements with blue LED elements. This is because blue is the color that is least likely to contribute to contour recognition for the human eye, and therefore even if the plurality of imaging units 22 are replaced with blue LED elements, the influence on the recognition of the image displayed on the display unit 24 is small.

[0020] The microphone unit 26 detects sound around the display device 10. The microphone unit 26 detects the sound of a user who is using the display device 10. The microphone unit 26 converts the detected sound into an audio signal.

[0021] The audio output unit 28 is a speaker that outputs various types of audio. For example, the audio output unit 28 outputs the audio of another user acquired from another display device 10.

[0022] The storage unit 30 stores various types of information. The storage unit 30 stores information such as the contents of calculations performed by the control unit 38 and programs. The storage unit 30 includes at least one of a RAM (Random Access Memory), a main storage device such as a ROM (Read Only Memory), and an external storage device such as a HDD (Hard Disk Drive).

[0023] The communication unit 32 is a communication interface that performs communication between the display device 10 and an external device. The communication unit 32 executes communication between the display device 10 and another display device 10, for example.

[0024] The encoder 34 compresses the image data captured by the imaging unit 22. The encoder 34 compresses the image data captured by the imaging unit 22 using a known compression method. An example of the compression method is a compression method based on the H.264 standard, but is not limited to this.

[0025] The decoder 36 decodes image data acquired from another display device 10. The decoder 36 decodes image data compressed by a known compression method using a known method to generate image data.

[0026] The control unit 38 controls each unit of the display device 10. The control unit 38 has, for example, an information processing device such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), and a storage device such as a RAM or a ROM. The control unit 38 executes a program that controls the operation of the display device 10 according to the present invention. The control unit 38 may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 38 may be realized by a combination of hardware and software.

[0027] The control unit 38 includes an imaging control unit 50 , an image data acquisition unit 52 , an image processing unit 54 , a face detection unit 56 , a display control unit 58 , an audio output control unit 60 , and a communication control unit 62 .

[0028] The imaging control unit 50 controls the imaging unit 22 to capture an image. The imaging control unit 50 controls the imaging unit 22 to capture an image of a user who uses the display device 10. The imaging control unit 50 acquires, from the imaging unit 22, image data relating to the image captured by the imaging unit 22.

[0029] The imaging control unit 50 controls the imaging unit 22 provided in the vicinity of the eye area of ​​another user displayed on the display unit 24 and detected by the face detection unit 56, for example, to capture an image of the user.

[0030] The image data acquisition unit 52 acquires image data of other users captured by the other display devices 10 from the other display devices 10 via the communication unit 32. The image data acquisition unit 52 may acquire the image data of other users together with the voice data of other users from the other display devices 10, for example.

[0031] The image processing unit 54 controls the encoder 34 to compress the image data acquired by the imaging control unit 50 from the imaging unit 22. The image processing unit 54 controls the decoder 36 to decode the compressed image data acquired from the other display device 10 via the communication unit 32.

[0032] The image processing unit 54 may, for example, execute a trimming process to cut out a predetermined range from the image data acquired by the imaging control unit 50 from the imaging unit 22. In this case, the image processing unit 54 may determine the range of the trimming process according to the position where the imaging unit 22 is provided. The image processing unit 54 determines the range of the trimming process according to the position where the imaging unit 22 is provided so that each imaging unit 22 can capture a similar image.

[0033] FIG. 4 is a diagram for explaining an example of the trimming process according to the first embodiment. In FIG. 4, the range R1 indicates the imaging range of the imaging unit 22. As shown in FIG. 4, the range R1 is wider in the X direction than the X direction of the display unit 24. In the X direction, the outside of the display unit 24 may be assumed to be an unnecessary area where the user using the display device 10 is unlikely to be present. Therefore, the image processing unit 54 performs a trimming process on the image data to delete an area assumed to be an unnecessary area. For example, the image data may be trimmed to match the X direction of the range R1 with the X direction of the display unit 24, and image data of the range R2 may be generated. In this case, the image processing unit 54 may control, for example, the encoder 34 to compress the image data of the range R2. Note that the method of the image processing unit 54 performing the trimming process on the image data is not limited to that shown in FIG. 4, and other methods may be used.

[0034] The face detection unit 56 detects the faces of other users by performing recognition processing on the images of other users displayed on the display unit 24. The method of detecting a human face from an image can be any well-known method, and is not limited thereto.

[0035] The display control unit 58 controls the display unit 24 to display various images on the display unit 24. For example, the display control unit 58 causes the display unit 24 to display image data acquired by the image data acquisition unit 52 from another display device 10. For example, the display control unit 58 causes the display unit 24 to display image data decoded by the decoder 36.

[0036] The audio output control unit 60 controls the audio output unit 28 to output various sounds from the audio output unit 28. The audio output control unit 60 controls the audio output unit 28 to obtain the sounds of other users from the audio output control unit 60.

[0037] The communication control unit 62 controls the communication unit 32 to execute communication between the display device 10 and an external device. The communication control unit 62 controls the communication unit 32 to execute communication between the display device 10 and another display device 10.

[0038] The communication control unit 62 transmits the image data acquired by the imaging control unit 50 from the imaging unit 22 to the other display device 10. The communication control unit 62 transmits image data compressed by the encoder 34 to the other display device 10, for example.

[0039] The communication control unit 62 receives image data from another display device 10. The communication control unit 62 receives compressed image data from another display device 10, for example.

[0040] (Processing contents) The processing contents of the display device according to the first embodiment will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a diagram for explaining the outline of the processing contents of the display device according to the first embodiment. Fig. 6 is a flowchart showing the processing contents of the display device according to the first embodiment.

[0041] As shown in FIG. 5, for example, the display device 10-1 is used by a user U1, and the display device 10-2 is used by a user U2. The display device 10-1 captures an image of the user U1 using the imaging unit 22, and transmits image data including the user U1 to the display device 10-2. The display device 10-2 captures an image of the user U2 using the imaging unit 22, and transmits image data including the user U2 to the display device 10-1. The display device 10-1 displays a user image UI2 that is an image of the user U2 on the display unit 24. The display device 10-2 displays a user image UI1 that is an image of the user U1 on the display unit 24. The display device 10-1 captures an image of the user U1 so that the user U2 can match his / her line of sight with the user image UI1. The display device 10-2 captures an image of the user U2 so that the user U1 can match his / her line of sight with the user image UI2.

[0042] Since the processing performed by the display device 10-1 and the display device 10-2 is the same, the following description will be given taking as an example the processing performed by the display device 10-1 that receives the image data of the user U2 from the display device 10-2.

[0043] 6, first, the imaging control unit 50 controls the initial setting imaging unit 22 among the multiple imaging units 22 to start imaging the user U1 (step S10). The initial setting imaging unit 22 is provided, for example, near the center of the display screen of the display unit 24. The initial setting imaging unit 22 may be arbitrarily set in advance. Then, the process proceeds to step S12.

[0044] The communication control unit 62 transmits image data captured by the initially set imaging unit 22 to the display device 10-2 (step S12). Specifically, the communication control unit 62 transmits image data compressed by the encoder 34 to the display device 10-2. Then, the process proceeds to step S14.

[0045] The image data acquisition unit 52 acquires the image data from the display device 10-2 (step S14), and the process proceeds to step S16.

[0046] The display control unit 58 controls the display unit 24 to display an image related to the image data acquired by the image data acquisition unit 52 (step S16). Specifically, the display control unit 58 causes the display unit 24 to display an image decoded by the decoder 36. FIG. 7 is a diagram for explaining an image displayed on the display unit 24 according to the first embodiment. As shown in FIG. 7, the display control unit 58 causes the display unit 24 to display a user image UI2, which is an image of the user U2. Then, the process proceeds to step S18.

[0047] The face detection unit 56 executes face detection processing (step S18). Specifically, the face detection unit 56 executes face detection processing on the user image UI2 shown in Fig. 7 to detect a face in the user image UI2. Then, the process proceeds to step S20.

[0048] The face detection unit 56 determines whether or not a face is detected (step S20). If it is determined that a face is detected (step S20; Yes), the process proceeds to step S22. If it is determined that a face is not detected (step S20; No), the process proceeds to step S30.

[0049] If the result of the determination in step S20 is Yes, the face detection unit 56 calculates the position coordinates of the detected face (step S22). Specifically, the face detection unit 56 calculates the position coordinates of the face area of ​​the user image UI2 displayed on the display unit 24. Then, the process proceeds to step S24.

[0050] The imaging control unit 50 selects the imaging unit 22 that captures the image of the user U1 based on the position coordinates calculated by the face detection unit 56 (step S24). Specifically, the imaging control unit 50 selects the imaging unit 22 provided near the eyes of the face of the user image UI2 displayed on the display unit 24. In the example shown in FIG. 7, the imaging control unit 50 selects the imaging unit 22a provided in the face area of ​​the user image UI2 as the imaging unit that captures the image of the user U1. By capturing an image of the user U1 with the imaging unit 22a, the display device 10-2 can display the user image UI1 whose line of sight matches that of the user U2. Then, the process proceeds to step S26.

[0051] The imaging control unit 50 starts imaging with the selected imaging unit 22 (step S26). In the example shown in Fig. 7, the imaging control unit 50 controls the imaging unit 22a to start imaging the user U1. Then, the process proceeds to step S28.

[0052] The communication control unit 62 transmits the image data captured by the imaging unit 22a to the display device 10-2 (step S28). Specifically, the communication control unit 62 transmits the image data compressed by the encoder 34 to the display device 10-2. Then, the process proceeds to step S30.

[0053] If the result of the determination in step S20 is No or after step S28, the control unit 38 determines whether or not to end the process (step S30). Specifically, the control unit 38 determines to end the process when the remote conversation between the user U1 and the user U2 ends, for example. If it is determined to end the process (step S30; Yes), the process of FIG. 6 ends. If it is not determined to end the process (step S30; No), the process proceeds to step S12.

[0054] That is, if it is determined as No in step S30, the processes from step S12 to step S28 are repeated. Therefore, when the user image UI2 moves on the display unit 24, the imaging unit 22 that captures the user U1 is switched in accordance with the movement of the user image UI2. As a result, even if the user image UI2 moves on the display unit 24, it is possible to capture an image of the user U1 whose line of sight matches that of the user U2.

[0055] [Modification of the first embodiment] In the first embodiment, a plurality of imaging units 22 are arranged at equal intervals on the display unit 24. However, the present invention is not limited to this.

[0056] A configuration example of a display unit according to a first modified example of the first embodiment will be described with reference to Fig. 8. Fig. 8 is a diagram showing a configuration example of a display unit according to a first modified example of the first embodiment.

[0057] As shown in Fig. 8, the display unit 24A includes a plurality of camera groups 40. The camera groups 40 are arranged at equal intervals along each of the X direction and the Y direction on the display unit 24A. The camera group 40 includes a plurality of imaging units 22. In the example shown in Fig. 8, the camera group 40 includes, but is not limited to, four imaging units 22 arranged closely together. By arranging the camera group 40 in which the plurality of imaging units 22 are arranged closely together, the brightness of the image data can be improved.

[0058] A configuration example of a display unit according to a second modified example of the first embodiment will be described with reference to Fig. 9. Fig. 9 is a diagram showing a configuration example of a display unit according to a second modified example of the first embodiment.

[0059] 9, the display unit 24B includes a plurality of imaging units 22. In the display unit 24B, the imaging units 22 are arranged more densely near the center in the X direction and the Y direction than in other locations. This is because it is expected that a user who uses the display unit 24B will often be near the center of the display unit 24B. By providing the imaging units 22 near the center in the X direction and the Y direction, it becomes possible to appropriately capture an image of the user.

[0060] [Second embodiment] A second embodiment will be described. In the second embodiment, a trimming process is performed to remove unnecessary parts of image data of an image of a user. The configuration of the display device according to the second embodiment is the same as that of the display device shown in FIG. 2, and therefore the description will be omitted.

[0061] (Processing contents) The processing contents of the display device according to the second embodiment will be described with reference to Fig. 10. Fig. 10 is a flowchart showing the processing contents of the display device according to the second embodiment.

[0062] The processes from step S40 to step S56 are the same as the processes from step S10 to step S26 shown in FIG. 6, respectively, and therefore will not be described.

[0063] The image processing unit 54 performs a trimming process on the image data (step S58). Specifically, the image processing unit 54 performs a trimming process on the image data to delete an area that is assumed to be an unnecessary area. Then, the process proceeds to step S60.

[0064] The processes in steps S60 and S62 are the same as those in steps S28 and S30 shown in FIG. 6, respectively, and therefore will not be described.

[0065] As described above, the second embodiment can trim unnecessary parts of image data of the user U1. As a result, the second embodiment can transmit appropriate image data to the display device 10-2 in consideration of the positional relationship between the user U1 and the display device 10-1. Even if the imaging unit 22 that images the user U1 is switched, the change in the range of the image of the user U1 seen by the user U2 and the change in the position of the user U1 in the image seen by the user U2 are suppressed, thereby reducing the sense of discomfort felt by the user U2.

[0066] [Third embodiment] A third embodiment will now be described. In the third embodiment, when a plurality of people are displayed on the display unit 24, a process is executed to identify the people displayed in the image and select a camera to capture the user.

[0067] (Processing contents) The processing contents of the display device according to the third embodiment will be described with reference to Fig. 11 and Fig. 12. Fig. 11 is a diagram for explaining the outline of the processing contents of the display device according to the third embodiment. Fig. 12 is a flowchart showing the processing contents of the display device according to the third embodiment.

[0068] As shown in FIG. 11, for example, the display device 10-1 is used by users U1 and U4, and the display device 10-2 is used by users U2 and U3. The display device 10-1 captures images of the users U1 and U4 using the imaging unit 22, and transmits image data including the users U1 and U4 to the display device 10-2. The display device 10-2 captures images of the users U2 and U3 using the imaging unit 22, and transmits image data including the users U2 and U3 to the display device 10-1. The display device 10-1 displays a user image UI2 that is an image of the user U2, and a user image UI3 that is an image of the user U3, on the display unit 24. The display device 10-2 displays a user image UI1 that is an image of the user U1, and a user image UI4 that is an image of the user U4, on the display unit 24. The display device 10-1 captures images of the users U1 and U4 so that the users U2 and U3 can match their gazes with the user images UI1 and UI4. The display device 10-2 captures images of the user U2 and the user U3 so that the user U1 and the user U4 can make eye contact with the user image UI2 and the user image UI3.

[0069] Since the processing performed by display device 10-1 and display device 10-2 is the same, the following description will be given taking as an example the processing performed by display device 10-1 which receives image data of user U2 and user U3 from display device 10-2.

[0070] In FIG. 12, the processes from step S70 to step S74 are the same as the processes from step S10 to step S14 shown in FIG. 6, respectively, and therefore the description thereof will be omitted.

[0071] The display control unit 58 controls the display unit 24 to display an image related to the image data acquired by the image data acquisition unit 52 (step S76). Specifically, the display control unit 58 causes the display unit 24 to display an image related to the image data decoded by the decoder 36. FIG. 13 is a diagram for explaining an image displayed on the display unit 24 according to the third embodiment. As shown in FIG. 13, the display control unit 58 causes the display unit 24 to display a user image UI2 which is an image of a user U2 and a user image UI3 which is an image of a user U3. Then, the process proceeds to step S80.

[0072] The face detection unit 56 executes face detection processing (step S78). Specifically, the face detection unit 56 executes face detection processing on the user image UI2 and the user image UI3 shown in Fig. 13 to detect faces in the user image UI2 and the user image UI3. Then, the process proceeds to step S80.

[0073] The face detection unit 56 determines whether or not a face is detected (step S80). If it is determined that a face is detected (step S80; Yes), the process proceeds to step S82. If it is determined that a face is not detected (step S80; No), the process proceeds to step S102.

[0074] If it is determined that multiple faces have been detected in step S80 as Yes, the face detection unit 56 determines whether multiple faces have been detected (step S82). If it is determined that multiple faces have been detected (step S82; Yes), the process proceeds to step S84. If it is not determined that multiple faces have been detected (step S82; No), the process proceeds to step S94.

[0075] If it is determined as Yes in step S82, the face detection unit 56 identifies the speaker (step S84). Specifically, in the example shown in Fig. 13, the face detection unit 56 detects the mouth movement included in the user image UI2 and the user image UI3, and if the mouth movement during speech is detected, identifies the user of the user image in which the movement is detected as the speaker. Here, if the mouth movement during speech is not detected, the face detection unit 56 identifies the user who was determined as the speaker immediately before as the current speaker. Then, the process proceeds to step S86.

[0076] The face detection unit 56 calculates the position coordinates of the speaker's face (step S86). Specifically, the face detection unit 56 calculates the position coordinates on the display unit 24 of the face area of ​​the user image UI2 or the user image UI3 displayed on the display unit 24. Then, the process proceeds to step S88.

[0077] The imaging control unit 50 selects the imaging unit 22 for imaging the user U1 and the user U4 based on the position coordinates of the face of the speaker calculated by the face detection unit 56 (step S88). Specifically, the imaging control unit 50 selects the imaging unit 22 provided near the eyes of the face of the user image UI2 or the user image UI3 identified as the speaker by the face detection unit 56, which is displayed on the display unit 24. In the example shown in FIG. 13, the imaging control unit 50 selects the imaging unit 22a provided in the face area of ​​the user image UI2 or the imaging unit 22b provided in the face area of ​​the user image UI3 as the imaging unit for imaging the user U1 and the user U4. By imaging the user U1 and the user U4 with the imaging unit 22a or the imaging unit 22b, the display device 10-2 can display the user image UI1 and the user image UI4 whose line of sight matches that of the user U2 and the user U3. Then, the process proceeds to step S90.

[0078] The imaging control unit 50 starts imaging with the selected imaging unit (step S90). In the example shown in Fig. 13, the imaging control unit 50 controls the imaging unit 22a or the imaging unit 22b to start imaging the users U1 and U4. Then, the process proceeds to step S92.

[0079] The communication control unit 62 transmits the image data captured by the imaging unit 22a or the imaging unit 22b to the display device 10-2 (step S92). Specifically, the communication control unit 62 transmits the image data compressed by the encoder 34 to the display device 10-2. Then, the process proceeds to step S102.

[0080] If it is determined as No in step S82, the face detection unit 56 calculates the position coordinates of the detected face (step S94). Here, in the example shown in FIG. 13, it is assumed that the face of the user image UI2 is detected from the user image UI2 and the user image UI3 in step S80. The face detection unit 56 calculates the position coordinates on the display unit 24 of the face area of ​​the user image UI2 displayed on the display unit 24. Then, the process proceeds to step S96.

[0081] The processes from step S96 to step S100 are the same as the processes from step S24 to step S28 shown in FIG. 6, respectively, and therefore will not be described.

[0082] If it is determined that the process is to be ended (step S102), the control unit 38 determines whether or not to end the process. Specifically, the control unit 38 determines that the process is to be ended when, for example, a remote conversation between the user U1 and the user U4 and the user U2 and the user U3 is ended. If it is determined that the process is to be ended (step S102; Yes), the process of FIG. 12 ends. If it is not determined that the process is to be ended (step S102; No), the process proceeds to step S72.

[0083] That is, if it is determined as No in step S102, the processes from step S72 to step S92 are repeated. Therefore, when the speaker of the user image UI2 and the user image UI3 is changed on the display unit 24, the imaging unit for imaging the user U1 and the user U4 is switched. As a result, even if the speaker of the user image UI2 and the user image UI3 is changed on the display unit 24, it is possible to capture images of the user U1 and the user U4 whose gazes match with the speaker.

[0084] [Fourth embodiment] A fourth embodiment will be described. In the fourth embodiment, the luminance of image data captured by each imaging unit provided in a display unit is analyzed. In the fourth embodiment, when the luminance difference between the image data captured by each imaging unit is equal to or greater than a predetermined value, a process is executed so that the luminance difference between the image data captured by each imaging unit becomes equal to or smaller than a predetermined value.

[0085] A configuration example of the display device according to the fourth embodiment will be described with reference to Fig. 14. Fig. 14 is a block diagram showing a configuration example of the display device according to the fourth embodiment.

[0086] As shown in FIG. 14, the display device 10A differs from the display device 10 shown in FIG.

[0087] The image analysis unit 64 performs image analysis processing on the image data acquired by the imaging control unit 50 from the imaging unit 22, and detects the luminance of the image data. The method for detecting the luminance of the image data can be a well-known method, and is not limited thereto.

[0088] (Processing contents) The processing contents of the display device according to the fourth embodiment will be described with reference to Fig. 15 and Fig. 16. Fig. 15 is a diagram for explaining the outline of the processing contents of the display device according to the fourth embodiment. Fig. 16 is a flowchart showing the processing contents of the display device according to the fourth embodiment.

[0089] As shown in FIG. 15, it is assumed that the user image UI2, which is an image of the user U2 displayed on the display unit 24, moves in the direction of the arrow. In this case, as the user image UI2 moves, the imaging unit that captures the user U1 changes from the imaging unit 22-1 to the imaging unit 22-2, the imaging unit 22-3, the imaging unit 22-4, and the imaging unit 22-5. In this case, if there is a difference in luminance between the image data captured by the imaging units 22-1 to 22-5 that is equal to or greater than a predetermined value, it is assumed that other users who view the other display device 10A on which the image data of the user U1 is displayed when the imaging unit is switched will feel uncomfortable. In addition, since each imaging unit 22 captures the image by following the user U1, for example, if the brightness of the background changes as the user U1 moves, the brightness value of the image data may change. Examples of situations in which the brightness of the background changes as the user moves include, for example, when the user U1 moves from a sunny place to a shaded place outdoors, or when the user moves from a place with strong lighting to a place with weak lighting indoors.

[0090] In this embodiment, a process of adjusting the brightness of the image data captured by the imaging units 22-1 and 22-2 is executed so as not to give a sense of incongruity to other users viewing another display device 10A on which the image data of user U1 is displayed.

[0091] 16, first, the imaging control unit 50 acquires image data from the imaging unit 22 (step S110). In the example shown in Fig. 15, the imaging control unit 50 acquires image data captured by the imaging units 22-1 to 22-5. Then, the process proceeds to step S112.

[0092] The image analysis unit 64 performs image analysis processing on the image data acquired by the imaging control unit 50 (step S112). In the example shown in Fig. 15, the image analysis unit 64 performs image analysis processing on the image data acquired by the imaging control unit 50 from the imaging units 22-1 to 22-5, respectively. Then, the process proceeds to step S114.

[0093] The image analysis unit 64 judges whether the luminance difference between the image data is equal to or greater than a threshold (step S114). In the example shown in FIG. 15, it judges whether the luminance difference between the image data acquired from the imaging units 22-1 to 22-5 is equal to or greater than a threshold. The luminance difference being equal to or greater than a threshold means, for example, a luminance difference that is large enough to cause a user who continuously views images related to the image data acquired from the imaging units 22-1 to 22-5 to feel uncomfortable due to a change in luminance. If it is judged that the luminance difference is equal to or greater than the threshold (step S114; Yes), the process proceeds to step S116. If it is not judged that the luminance difference is equal to or greater than the threshold (step S114; No), the process proceeds to step S118.

[0094] If it is determined as Yes in step S114, the imaging control unit 50 sets the image quality of the imaging unit 22 based on each image data (step S116). In the example shown in FIG. 15, the imaging control unit 50 adjusts the gain of each of the imaging units 22-1 to 22-5 based on each image data acquired from the imaging units 22-1 to 22-5, respectively. For example, the imaging control unit 50 adjusts the gain of each of the imaging units 22-1 to 22-5 so that the luminance of each image data captured by the imaging units 22-1 to 22-5 is the same. For example, the imaging control unit 50 may adjust the gain of each of the imaging units 22-1 to 22-5 so that the difference in luminance value of the image data captured by the adjacent imaging units 22 among the imaging units 22-1 to 22-5 is equal to or less than a predetermined value. For example, the imaging control unit 50 may adjust the gains of the imaging units 22-1 to 22-5 so that the luminance values ​​of the image data acquired from the imaging units 22-1 to 22-5 change gradually in order. Then, the process proceeds to step S120.

[0095] If it is determined as No in step S114, the imaging control unit 50 sets each imaging unit 22 to the same setting so that the image quality of the image displayed on the display unit 24 does not change when the imaging unit 22 is switched (step S118). In the example shown in Fig. 15, the imaging control unit 50 sets the gain of each of the imaging units 22-1 to 22-5 to the same setting. Then, the process proceeds to step S120.

[0096] The control unit 38A determines whether or not to end the process (step S120). For example, the control unit 38A determines to end the process when the remote conversation between the user U1 and the user U2 ends. If it is determined to end the process (step S120; Yes), the control unit 38A ends the process of FIG. 16. If it is not determined to end the process (step S120; No), the control unit 38A proceeds to step S110.

[0097] The control unit 38A may perform the process shown in FIG. 16 constantly while the user U1 and the user U2 are having a remote conversation, or may perform the process at predetermined intervals (for example, every 1 to 10 minutes).

[0098] As described above, the fourth embodiment adjusts the gain of each imaging unit that captures an image of the user U1 constantly or at predetermined intervals. This makes it possible to capture an image of the user U1 that does not give a sense of incongruity to a user U2 or the like who is viewing the image data of the user U1.

[0099] [Fifth embodiment] A fifth embodiment will be described. In the fifth embodiment, the image capturing unit captures the voice of the user, analyzes the captured voice, and identifies the direction from which the voice is emitted. In the fifth embodiment, a process is executed to identify the speaker based on the captured image of the user and the identified direction from which the voice is emitted.

[0100] (display device) A configuration example of the display device according to the fifth embodiment will be described with reference to Fig. 17. Fig. 17 is a block diagram showing a configuration example of the display device according to the fifth embodiment.

[0101] As shown in FIG. 17, a display device 10B differs from the display device 10 shown in FIG. 2 in that a control unit 38B includes a voice data acquisition unit 66, a direction identification unit 68, and a speaker identification unit .

[0102] Fig. 18 is a diagram for explaining a method of arranging microphone units 26 according to the fifth embodiment. As shown in Fig. 18, in the fifth embodiment, for example, a plurality of microphone units 26 are provided in the same plane as the display screen of the display unit 24. In the example shown in Fig. 18, five microphone units 26 are provided at the bottom of the display unit 24. The number and positions of the microphone units 26 are not limited as long as they are provided in the same plane as the display screen of the display unit 24. The plurality of microphone units 26 detect, for example, a voice uttered by a user U1.

[0103] In the fifth embodiment, even if it is not possible to identify whether the user U2 is the speaker only from the image because the mouth is covered by a mask, etc., the display device 10B determines whether the user U2 is the speaker based on a user image UI2 that is an image of the user U2 using another display device 10B, and voice data related to the voice of the user U2.

[0104] The voice data acquisition unit 66 acquires voice data relating to a voice detected by the microphone unit 26 of the other display device 10B from the other display device 10B. The voice data acquisition unit 66 acquires, for example, voice data relating to the voice of the user U2 acquired by the other display device 10B.

[0105] The direction identifying unit 68 identifies the direction from which the voice related to the voice data is emitted, based on the voice data detected by the voice data acquiring unit 66. The direction identifying unit 68 identifies the direction of a speaker who emitted the voice, for example, based on the voice data related to the voice of the speaker acquired by the voice data acquiring unit 66. In this embodiment, the microphone unit 26 is provided on the same plane as the display unit 24, so the direction identifying unit 68 identifies the direction with respect to the display unit 24.

[0106] The speaker identification unit 70 identifies the speaker who uttered the voice. The speaker identification unit 70 identifies the speaker based on the image data acquired by the image data acquisition unit 52 from the other display device 10B and the result of the direction identification by the direction identification unit 68. Specifically, the speaker identification unit 70 identifies the user U2 as the speaker when the direction of the user image UI2 acquired by the image data acquisition unit 52 matches the direction identified by the direction identification unit 68.

[0107] (Processing contents) The processing contents of the display device according to the fifth embodiment will be described with reference to Fig. 19. Fig. 19 is a flowchart showing the processing contents of the display device according to the fifth embodiment.

[0108] The processes from step S130 to step S136 are the same as the processes from step S10 to step S16 shown in FIG. 6, respectively, and therefore will not be described.

[0109] The voice data acquisition unit 66 acquires voice data from the other display device 10B (step S138), and the process proceeds to step S140.

[0110] The processes in steps S140 and S142 are the same as those in steps S18 and S20 shown in FIG. 6, respectively, and therefore will not be described.

[0111] If the answer to step S142 is Yes, the direction identification unit 68 performs a voice analysis process on the voice data acquired by the voice data acquisition unit 66 to identify the direction from which the voice is emitted (step S144). Then, the process proceeds to step S146.

[0112] The speaker identification unit 146 determines whether or not the speaker has been identified (step S146). If it is determined that the speaker has been identified (step S146; Yes), the process proceeds to step S148. If it is determined that the speaker has not been identified (step S146; No), the process proceeds to step S156.

[0113] The processes from step S148 to step S164 are the same as the processes from step S86 to step S102 shown in FIG. 12, respectively, and therefore will not be described.

[0114] In the description of this embodiment, the display device identifies the direction from which the voice is emitted based on the voice data acquired from the other display device, but the other display device may identify the direction from which the voice is emitted based on the voice detected by the microphone unit 26. In this case, the display device acquires data on the direction from which the voice is emitted from the other display device, and the speaker identification unit 70 identifies the user U2 as the speaker when the direction of the user image UI2 matches the direction from which the voice indicated by the acquired data is emitted.

[0115] As described above, in the fifth embodiment, even if the speaker cannot be identified from the image, the speaker can be identified based on the image and the voice. As a result, the fifth embodiment can capture an image of the user U1 so that the line of sight of the user U2 is appropriately matched.

[0116] Each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc. This distribution and integration configuration may be performed dynamically.

[0117] Although the embodiments of the present invention have been described above, the present invention is not limited to the contents of these embodiments. In the embodiments of the present invention, an LED display has been described as a display device, but the display device is not limited to this. For example, a projector (projection device) may be used as the display device, and multiple imaging units may be arranged on or behind a screen onto which the image is projected. In addition, in the case of a display device in which some display elements cannot be replaced with imaging units, such as a liquid crystal display or an organic EL (Electro-Luminescence), multiple imaging units may be arranged in front of the display unit. In addition, the above-mentioned components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-mentioned components can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of the components can be made within the scope of the gist of the above-mentioned embodiments. [Explanation of symbols]

[0118] 1 Display System 10 Display device 20 Input section 22 Imaging unit 24 Display 26 Microphone section 28 Audio output section 30 Storage section 32 Communications Department 34 Encoder 36 Decoder 38 Control Unit 50 Imaging control section 52 Image data acquisition unit 54 Image processing section 56 Face detection section 58 Display control section 60 Audio output control unit 62 Communication control section 64 Image Analysis Unit 66 Audio data acquisition unit 68 Direction identification part 70 Speaker Identification Unit

Claims

1. A display unit that is an LED display for displaying an image, a plurality of imaging units provided in place of blue pixels included in the LED display on the display screen of the display unit, for imaging an image of a user located in the direction of the display screen, a display control unit for causing the display unit to display an image of another user who converses with the user, a face detection unit for detecting a face including eyes of the other user from the image of the other user displayed on the display unit, an imaging control unit for controlling the imaging unit provided in the vicinity of the eye region of the other user displayed on the display unit to image the image of the user, a communication control unit for transmitting the image of the user imaged by the imaging unit provided in the vicinity of the eye region of the other user, A display device comprising:

2. An image processing unit that trims a range corresponding to the position where the imaging unit that images the user is provided, from among the images of the user imaged by the imaging unit provided in the vicinity of the eye region of the other user displayed on the display unit, The display device according to Claim 1.

3. An image analysis unit for analyzing the luminance of the images imaged by each of the plurality of imaging units provided in the display unit, The imaging control unit adjusts the gain of each of the plurality of imaging units according to the result by the image analysis unit. The display device according to Claim 1 or 2.

4. At least one microphone provided in the vicinity of the display screen of the display unit, a voice data acquisition unit for acquiring voice data regarding the voice of the other user detected by the microphone of another display device, a direction identification unit for identifying the direction in which the voice of the other user was emitted, based on the voice data acquired by the voice data acquisition unit, a speaker identification unit for identifying the other user as a speaker when the direction of the other user imaged by the imaging unit of the other display device matches the direction identified by the direction identification unit, The imaging control unit controls the imaging unit provided in the vicinity of the eye region of the other user who is the speaker identified by the speaker identification unit and is displayed on the display unit, to perform imaging. The display device according to Claim 1 or 2.

5. The step of causing an image to be displayed on a display unit that is an LED display, A step of replacing the blue pixels included in the LED display on the display screen of the display unit and causing a plurality of imaging units to image an image of a user located in the direction of the display screen; A step of causing the display unit to display an image of another user who is conversing with the user; A step of detecting a face including the eyes of the other user from the image of the other user displayed on the display unit; A step of controlling the imaging unit provided in the vicinity of the eye region of the other user displayed on the display unit to image the image of the user; A step of transmitting the image of the user captured by the imaging unit provided in the vicinity of the eye region of the other user; A display method including the above.

Images