Display device and display method

Abstract

To properly achieve the coincidence of lines of sight between users in remote conversation accompanied with images.SOLUTION: A display device includes: a display section displaying images; a plurality of imaging sections provided on a display screen of the display section and capturing images of a user located in a direction of the display screen; a display control section allowing the display section to display images of another user talking with the user; a face detection section detecting a face including the eyes of the other user from the images of the other user displayed on the display section; an imaging control section controlling the imaging section to capture images of the user, the imaging section being provided in the vicinity of an area of the eyes of the other user displayed on the display section; and a communication control section transmitting images of the user captured by the imaging section provided in the vicinity of the area of the eyes of the other user.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a display device and a display method.

Background Art

[0002] Techniques for realizing remote conversations with video such as video conferences are known. For example, Patent Document 1 discloses a technique for aligning the line of sight of users in a video conference or a video conversation system.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a video conference or the like, for example, a scene is assumed where a user speaks while moving in front of a whiteboard or the like. In such a case, with the technique as described in Patent Document 1, it may be difficult to align the lines of sight of the users.

[0005] An object of the present invention is to provide a display device and a display method capable of appropriately aligning the lines of sight of users in a remote conversation with video.

Means for Solving the Problems

[0006] The display device of the present invention comprises: a display unit for displaying images; a plurality of imaging units provided on the display screen of the display unit for capturing images of users located in the direction of the display screen; a display control unit for causing the display unit to display images of other users conversing with the user; a face detection unit for detecting the face, including the eyes of the other user, from the images of the other user displayed on the display unit; an imaging control unit for controlling the imaging units provided near the eye area of ​​the other user displayed on the display unit to capture images of the user; and a communication control unit for transmitting images of the user captured by the imaging units provided near the eye area of ​​the other user.

[0007] The display method of the present invention includes the steps of: displaying an image on a display unit; causing a plurality of imaging units provided on the display screen of the display unit to capture an image of a user located in the direction of the display screen; displaying an image of another user conversing with the user on the display unit; detecting a face including the eyes of the other user from the image of the other user displayed on the display unit; controlling the imaging unit provided near the eye area of ​​the other user displayed on the display unit to capture an image of the user; and transmitting the image of the user captured by the imaging unit provided near the eye area of ​​the other user. [Effects of the Invention]

[0008] According to the present invention, it is possible to appropriately align the gazes of users in remote conversations accompanied by video. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is a block diagram showing an example configuration of the display system according to the first embodiment. [Figure 2] Figure 2 is a block diagram showing an example configuration of a display device according to the first embodiment. [Figure 3] Figure 3 shows an example of the configuration of the display unit according to the first embodiment. [Figure 4]Figure 4 is a diagram illustrating an example of the trimming process according to the first embodiment. [Figure 5] Figure 5 is a diagram illustrating the overview of the processing content of the display device according to the first embodiment. [Figure 6] Figure 6 is a flowchart showing the processing details of the display device according to the first embodiment. [Figure 7] Figure 7 is a diagram illustrating the image displayed on the display unit according to the first embodiment. [Figure 8] Figure 8 shows an example of the configuration of the display unit according to the first modified example of the first embodiment. [Figure 9] Figure 9 shows an example of the configuration of the display unit according to a second modified example of the first embodiment. [Figure 10] Figure 10 is a flowchart showing the processing details of the display device according to the second embodiment. [Figure 11] Figure 11 is a diagram illustrating the overview of the processing content of the display device according to the third embodiment. [Figure 12] Figure 12 is a flowchart showing the processing details of the display device according to the third embodiment. [Figure 13] Figure 13 is a diagram illustrating the image displayed on the display unit according to the third embodiment. [Figure 14] Figure 14 is a block diagram showing an example configuration of a display device according to the fourth embodiment. [Figure 15] Figure 15 is a diagram illustrating the overview of the processing content of the display device according to the fourth embodiment. [Figure 16] Figure 16 is a flowchart showing the processing details of the display device according to the fourth embodiment. [Figure 17] Figure 17 is a block diagram showing an example configuration of a display device according to the fifth embodiment. [Figure 18] Figure 18 is a diagram illustrating the arrangement method of the microphone unit according to the fifth embodiment. [Figure 19] Figure 19 is a flowchart showing the processing details of the display device according to the fifth embodiment.

Best Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited by this embodiment, and in the following embodiments, the same parts are denoted by the same reference numerals and redundant descriptions are omitted.

[0011] [First Embodiment] (Display System) Using FIG. 1, a configuration example of the display system according to the first embodiment will be described. FIG. 1 is a block diagram showing a configuration example of the display system according to the first embodiment.

[0012] As shown in FIG. 1, the display system 1 includes a display device 10-1 and a display device 10-2. The display device 10-1 and the display device 10-2 are communicably connected via a wired or wireless network N. The network N is, for example, the Internet, but is not limited thereto. The display device 10-1 and the display device 10-2 may be collectively referred to as the display device 10.

[0013] The display device 10-1 is a device used by a user. The display device 10-2 is a device used by another user. The user and the other user can perform a remote conversation with video by using the display device 10-1 and the display device 10-2. The display devices 10-1 and 10-2 of the present embodiment can realize a remote conversation with video in which the lines of sight of the user and the other user can be made to coincide with each other.

[0014] (Display Device) Using FIG. 2, a configuration example of the display device according to the first embodiment will be described. FIG. 2 is a block diagram showing a configuration example of the display device according to the first embodiment.

[0015] As shown in Figure 2, the display device 10 includes an input unit 20, an imaging unit 22, a display unit 24, a microphone unit 26, an audio output unit 28, a storage unit 30, a communication unit 32, an encoder 34, a decoder 36, and a control unit 38.

[0016] The input unit 20 receives various operations on the display device 10. The input unit 20 can be implemented as a switch, button, touch panel, or the like.

[0017] The imaging unit 22 captures images. Multiple imaging units 22 are provided on the display screen of the display unit 24. The imaging units 22 are configured to capture the direction of the display screen of the display unit 24. The imaging unit 22 captures, for example, moving images. The imaging unit 22 may be, for example, a camera including an optical element and an image sensor.

[0018] The display unit 24 displays various images. For example, the display unit 24 displays images of other users obtained from other display devices. The display unit 24 is, for example, a large LED (Light Emitting Diode) display.

[0019] Figure 3 is a diagram showing an example configuration of the display unit 24 according to the first embodiment. As shown in Figure 3, the display unit 24 has a plurality of imaging units 22 arranged facing the direction of the display screen. The plurality of imaging units 22 are arranged at equal intervals along the X and Y directions in the display unit 24. The plurality of imaging units 22 are arranged to replace, for example, a plurality of LED elements arranged on the display screen of the display unit 24. Preferably, the plurality of imaging units 22 are arranged to replace, for example, blue LED elements among the plurality of LED elements. This is because blue is the color that contributes least to contour recognition to the human eye, so replacing the blue LED elements has little effect on the recognition of the image displayed on the display unit 24.

[0020] The microphone unit 26 detects sounds in the vicinity of the display device 10. The microphone unit 26 detects the voice of the user using the display device 10. The microphone unit 26 converts the detected sounds into an audio signal.

[0021] The audio output unit 28 is a speaker that outputs various types of audio. For example, the audio output unit 28 outputs the audio of another user acquired from another display device 10.

[0022] The memory unit 30 stores various types of information. The memory unit 30 stores the calculation contents of the control unit 38 and information such as programs. The memory unit 30 includes, for example, at least one of the following: RAM (Random Access Memory), main memory such as ROM (Read Only Memory), and external memory such as HDD (Hard Disk Drive).

[0023] The communication unit 32 is a communication interface that performs communication between the display device 10 and an external device. For example, the communication unit 32 performs communication between the display device 10 and another display device 10.

[0024] The encoder 34 compresses the image data captured by the imaging unit 22. The encoder 34 compresses the image data captured by the imaging unit 22 using a well-known compression method. Examples of compression methods include, but are not limited to, a compression method based on the H.264 standard.

[0025] The decoder 36 decodes image data acquired from another display device 10. The decoder 36 decodes image data compressed using a well-known compression method using a well-known method to generate image data.

[0026] The control unit 38 controls each part of the display device 10. The control unit 38 includes, for example, an information processing device such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), and a storage device such as RAM or ROM. The control unit 38 executes a program that controls the operation of the display device 10 according to the present invention. The control unit 38 may be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 38 may be implemented by a combination of hardware and software.

[0027] The control unit 38 includes an imaging control unit 50, an image data acquisition unit 52, an image processing unit 54, a face detection unit 56, a display control unit 58, an audio output control unit 60, and a communication control unit 62.

[0028] The imaging control unit 50 controls the imaging unit 22 to capture an image. The imaging control unit 50 controls the imaging unit 22 to capture an image of the user using the display device 10. The imaging control unit 50 acquires image data related to the image captured by the imaging unit 22 from the imaging unit 22.

[0029] The imaging control unit 50 controls, for example, the imaging unit 22 located near the eye area of ​​another user displayed on the display unit 24, which has been detected by the face detection unit 56, to capture an image of the user.

[0030] The image data acquisition unit 52 acquires image data of other users captured by other display devices 10 from the other display device 10 via the communication unit 32. The image data acquisition unit 52 may, for example, acquire image data of other users together with the other user's voice data from the other display device 10.

[0031] The image processing unit 54 controls the encoder 34 to compress the image data acquired by the imaging control unit 50 from the imaging unit 22. The image processing unit 54 controls the decoder 36 to decode the compressed image data acquired from another display device 10 via the communication unit 32.

[0032] The image processing unit 54 may, for example, perform a trimming process to extract a predetermined range from the image data acquired by the imaging control unit 50 from the imaging unit 22. In this case, the image processing unit 54 may determine the range of the trimming process according to the location where the imaging unit 22 is installed. For example, the image processing unit 54 determines the range of the trimming process according to the location where the imaging unit 22 is installed so that each imaging unit 22 can capture a similar image.

[0033] Figure 4 is a diagram illustrating an example of the trimming process according to the first embodiment. In Figure 4, range R1 indicates the imaging range of the imaging unit 22. As shown in Figure 4, range R1 is wider in the X direction than the X direction of the display unit 24. In the X direction, the area outside the display unit 24 can be assumed to be an unnecessary area where it is unlikely that a user of the display device 10 will be present. Therefore, the image processing unit 54 performs a trimming process to remove the area assumed to be unnecessary from the image data. For example, the image data may be trimmed so that the X direction of range R1 matches the X direction of the display unit 24, and image data for range R2 may be generated. In this case, the image processing unit 54 may, for example, control the encoder 34 to compress the image data for range R2. Note that the method by which the image processing unit 54 performs the trimming process on the image data is not limited to Figure 4, and other methods may be used.

[0034] The face detection unit 56 performs recognition processing on images of other users displayed on the display unit 24 and detects the faces of other users. The method for detecting a person's face from an image is not limited to known methods and can be used.

[0035] The display control unit 58 controls the display unit 24 to display various types of images on the display unit 24. For example, the display control unit 58 displays image data acquired by the image data acquisition unit 52 from another display device 10 on the display unit 24. For example, the display control unit 58 displays image data decoded by the decoder 36 on the display unit 24.

[0036] The audio output control unit 60 controls the audio output unit 28 to output various types of audio to the audio output unit 28. The audio output control unit 60 also controls the audio output unit 28 to acquire the audio of other users from the audio output control unit 60.

[0037] The communication control unit 62 controls the communication unit 32 to perform communication between the display device 10 and an external device. The communication control unit 62 controls the communication unit 32 to perform communication between the display device 10 and another display device 10.

[0038] The communication control unit 62 transmits the image data acquired by the imaging control unit 50 from the imaging unit 22 to another display device 10. The communication control unit 62 transmits, for example, the image data compressed by the encoder 34 to the other display device 10.

[0039] The communication control unit 62 receives image data from other display devices 10. For example, the communication control unit 62 receives compressed image data from other display devices 10.

[0040] (Processing details) The processing content of the display device according to the first embodiment will be explained using Figures 5 and 6. Figure 5 is a diagram illustrating the overview of the processing content of the display device according to the first embodiment. Figure 6 is a flowchart showing the processing content of the display device according to the first embodiment.

[0041] As shown in Figure 5, for example, let's assume that display device 10-1 is used by user U1 and display device 10-2 is used by user U2. Display device 10-1 captures user U1 with its imaging unit 22 and transmits the image data including user U1 to display device 10-2. Display device 10-2 captures user U2 with its imaging unit 22 and transmits the image data including user U2 to display device 10-1. Display device 10-1 displays user image UI2, which is an image of user U2, on its display unit 24. Display device 10-2 displays user image UI1, which is an image of user U1, on its display unit 24. Display device 10-1 captures user U1 so that user U2 can align their gaze with user image UI1. Display device 10-2 captures user U2 so that user U1 can align their gaze with user image UI2.

[0042] Since the processing performed by display device 10-1 and display device 10-2 is the same, the following explanation will use the processing performed by display device 10-1 after receiving image data from user U2 from display device 10-2 as an example.

[0043] As shown in Figure 6, first, the imaging control unit 50 controls the initial-setting imaging unit 22 among the multiple imaging units 22 to start imaging for user U1 (step S10). The initial-setting imaging unit 22 is, for example, located near the center of the display screen of the display unit 24. The initial-setting imaging unit 22 may be arbitrarily set in advance. Then, the process proceeds to step S12.

[0044] The communication control unit 62 transmits the image data captured by the initial imaging unit 22 to the display device 10-2 (step S12). Specifically, the communication control unit 62 transmits the image data compressed by the encoder 34 to the display device 10-2. Then, the process proceeds to step S14.

[0045] The image data acquisition unit 52 acquires image data from the display device 10-2 (step S14). Then, step S16 is performed.

[0046] The display control unit 58 controls the display unit 24 to display an image related to the image data acquired by the image data acquisition unit 52 (step S16). Specifically, the display control unit 58 displays the image decoded by the decoder 36 on the display unit 24. Figure 7 is a diagram illustrating the image displayed on the display unit 24 according to the first embodiment. As shown in Figure 7, the display control unit 58 displays the user image UI2, which is an image of user U2, on the display unit 24. Then, the process proceeds to step S18.

[0047] The face detection unit 56 performs face detection processing (step S18). Specifically, the face detection unit 56 performs face detection processing on the user image UI2 shown in Figure 7 to detect faces in the user image UI2. Then, the process proceeds to step S20.

[0048] The face detection unit 56 determines whether or not a face has been detected (step S20). If it is determined that a face has been detected (step S20; Yes), the process proceeds to step S22. If it is determined that no face has been detected (step S20; No), the process proceeds to step S30.

[0049] If the result in step S20 is determined to be Yes, the face detection unit 56 calculates the position coordinates of the detected face (step S22). Specifically, the face detection unit 56 calculates the position coordinates of the face area of ​​the user image UI2 displayed on the display unit 24. Then, the process proceeds to step S24.

[0050] The imaging control unit 50 selects an imaging unit 22 to image user U1 based on the position coordinates calculated by the face detection unit 56 (step S24). Specifically, the imaging control unit 50 selects an imaging unit 22 located near the eyes of the face in the user image UI2 displayed on the display unit 24. In the example shown in Figure 7, the imaging control unit 50 selects the imaging unit 22a located within the face region of the user image UI2 as the imaging unit to image user U1. By imaging user U1 with the imaging unit 22a, the display device 10-2 can display the user image UI1 such that its gaze matches that of user U2. Then, the process proceeds to step S26.

[0051] The imaging control unit 50 starts imaging with the selected imaging unit 22 (step S26). In the example shown in Figure 7, the imaging control unit 50 controls the imaging unit 22a to start imaging of user U1. Then, the process proceeds to step S28.

[0052] The communication control unit 62 transmits the image data captured by the imaging unit 22a to the display device 10-2 (step S28). Specifically, the communication control unit 62 transmits the image data compressed by the encoder 34 to the display device 10-2. Then, the process proceeds to step S30.

[0053] If the result in step S20 is No, or after step S28, the control unit 38 determines whether or not to terminate the process (step S30). Specifically, the control unit 38 determines to terminate the process when, for example, the remote conversation between user U1 and user U2 ends. If it is determined that the process should be terminated (step S30; Yes), the process in Figure 6 ends. If it is not determined that the process should be terminated (step S30; No), the process proceeds to step S12.

[0054] In other words, if the result in step S30 is determined to be No, the processing from step S12 to step S28 is repeated. Therefore, when the user image UI2 moves on the display unit 24, the imaging unit 22 that images user U1 switches in accordance with the movement of the user image UI2. As a result, even when the user image UI2 moves on the display unit 24, an image of user U1 can be captured in which the line of sight of user U2 coincides.

[0055] [Modified version of the first embodiment] In the first embodiment, the display unit 24 was described as having multiple imaging units 22 arranged at equal intervals, but the present invention is not limited thereto.

[0056] Using Figure 8, an example of the configuration of the display unit according to the first modified example of the first embodiment will be described. Figure 8 is a diagram showing an example of the configuration of the display unit according to the first modified example of the first embodiment.

[0057] As shown in Figure 8, the display unit 24A includes a plurality of camera groups 40. The camera groups 40 are arranged at equal intervals along the X and Y directions in the display unit 24A. The camera groups 40 include a plurality of imaging units 22. In the example shown in Figure 8, the camera group 40 includes, but is not limited to, four densely arranged imaging units 22. By arranging a camera group 40 with a plurality of densely arranged imaging units 22, the brightness of the image data can be improved.

[0058] Using Figure 9, an example of the configuration of the display unit according to a second modification of the first embodiment will be described. Figure 9 is a diagram showing an example of the configuration of the display unit according to a second modification of the first embodiment.

[0059] As shown in Figure 9, the display unit 24B includes multiple imaging units 22. In the display unit 24B, the imaging units 22 are densely arranged near the center in the X and Y directions compared to other areas. This is because it is assumed that users of the display unit 24B are often located near the center of the display unit 24B. By providing the imaging units 22 near the center in the X and Y directions, it becomes possible to properly image the user.

[0060] [Second Embodiment] A second embodiment will now be described. In the second embodiment, a trimming process is performed to remove unnecessary parts of the image data of the user. The configuration of the display device according to the second embodiment is the same as the configuration of the display device shown in Figure 2, so its description will be omitted.

[0061] (Processing details) The processing details of the display device according to the second embodiment will be explained using Figure 10. Figure 10 is a flowchart showing the processing details of the display device according to the second embodiment.

[0062] The processes from step S40 to step S56 are the same as the processes from step S10 to step S26 shown in Figure 6, so their explanation will be omitted.

[0063] The image processing unit 54 performs a trimming process on the image data (step S58). Specifically, the image processing unit 54 performs a trimming process on the image data to remove areas that are considered unnecessary. Then, the process proceeds to step S60.

[0064] The processes in steps S60 and S62 are the same as those in steps S28 and S30 shown in Figure 6, respectively, so their explanation will be omitted.

[0065] As described above, the second embodiment can trim unnecessary parts of the image data of user U1. This allows the second embodiment to transmit appropriate image data to the display device 10-2, taking into account the positional relationship between user U1 and the display device 10-1. Even if the imaging unit 22 that captures user U1 is switched, changes in the range of the image of user U1 seen by user U2 and changes in the position of user U1 in the image seen by user U2 are suppressed, thereby reducing the sense of discomfort experienced by user U2.

[0066] [Third Embodiment] A third embodiment will now be described. In the third embodiment, when multiple people are displayed on the display unit 24, the system performs a process to identify the person displayed in the image and select a camera to capture the user.

[0067] (Processing details) The processing content of the display device according to the third embodiment will be explained using Figures 11 and 12. Figure 11 is a diagram illustrating the overview of the processing content of the display device according to the third embodiment. Figure 12 is a flowchart showing the processing content of the display device according to the third embodiment.

[0068] As shown in Figure 11, for example, display device 10-1 is used by users U1 and U4, and display device 10-2 is used by users U2 and U3. Display device 10-1 captures images of users U1 and U4 with its imaging unit 22 and transmits the image data including users U1 and U4 to display device 10-2. Display device 10-2 captures images of users U2 and U3 with its imaging unit 22 and transmits the image data including users U2 and U3 to display device 10-1. Display device 10-1 displays user image UI2, which is an image of user U2, and user image UI3, which is an image of user U3, on its display unit 24. Display device 10-2 displays user image UI1, which is an image of user U1, and user image UI4, which is an image of user U4, on its display unit 24. Display device 10-1 captures images of users U1 and U4 so that users U2 and U3 can align their gaze with user images UI1 and user image UI4. The display device 10-2 captures images of users U2 and U3 so that users U1 and U4 can align their gaze with user images UI2 and user images UI3.

[0069] Since the processing performed by display device 10-1 and display device 10-2 is the same, the following explanation will use the processing performed by display device 10-1, which receives image data from user U2 and user U3 from display device 10-2, as an example.

[0070] In Figure 12, the processes from steps S70 to S74 are the same as the processes from steps S10 to S14 shown in Figure 6, so their explanation is omitted.

[0071] The display control unit 58 controls the display unit 24 to display an image related to the image data acquired by the image data acquisition unit 52 (step S76). Specifically, the display control unit 58 displays an image related to the image data decoded by the decoder 36 on the display unit 24. Figure 13 is a diagram illustrating the image displayed on the display unit 24 according to the third embodiment. As shown in Figure 13, the display control unit 58 displays user image UI2, which is an image of user U2, and user image UI3, which is an image of user U3, on the display unit 24. Then, the process proceeds to step S80.

[0072] The face detection unit 56 performs face detection processing (step S78). Specifically, the face detection unit 56 performs face detection processing on user image UI2 and user image UI3 shown in Figure 13 to detect faces in user image UI2 and user image UI3. Then, the process proceeds to step S80.

[0073] The face detection unit 56 determines whether or not a face has been detected (step S80). If it is determined that a face has been detected (step S80; Yes), the process proceeds to step S82. If it is determined that no face has been detected (step S80; No), the process proceeds to step S102.

[0074] If the result in step S80 is Yes, the face detection unit 56 determines whether multiple faces have been detected (step S82). If it is determined that multiple faces have been detected (step S82; Yes), the process proceeds to step S84. If it is determined that multiple faces have not been detected (step S82; No), the process proceeds to step S94.

[0075] If the result in step S82 is "Yes", the face detection unit 56 identifies the speaker (step S84). Specifically, in the example shown in Figure 13, the face detection unit 56 detects the mouth movements included in user image UI2 and user image UI3, and if it detects mouth movements during speech, it identifies the user in the user image in which the movement was detected as the speaker. If no mouth movements during speech are detected, the face detection unit 56 identifies the user previously identified as the speaker as the current speaker. Then, the process proceeds to step S86.

[0076] The face detection unit 56 calculates the position coordinates of the speaker's face (step S86). Specifically, the face detection unit 56 calculates the position coordinates of the face area of ​​the user image UI2 or user image UI3 displayed on the display unit 24. Then, the process proceeds to step S88.

[0077] The imaging control unit 50 selects an imaging unit 22 to image users U1 and U4 based on the position coordinates of the speaker's face calculated by the face detection unit 56 (step S88). Specifically, the imaging control unit 50 selects an imaging unit 22 located near the eyes of the face of user image UI2 or user image UI3, which the face detection unit 56 has identified as the speaker, as displayed on the display unit 24. In the example shown in Figure 13, the imaging control unit 50 selects either the imaging unit 22a located within the face region of user image UI2, or the imaging unit 22b located within the face region of user image UI3, as the imaging unit to image users U1 and U4. By imaging users U1 and U4 with the imaging unit 22a or imaging unit 22b, the display device 10-2 can display user images UI1 and user image UI4 such that their gazes coincide with those of users U2 and U3. Then, the process proceeds to step S90.

[0078] The imaging control unit 50 starts imaging with the selected imaging unit (step S90). In the example shown in Figure 13, the imaging control unit 50 controls either the imaging unit 22a or the imaging unit 22b to start imaging for users U1 and U4. Then, the process proceeds to step S92.

[0079] The communication control unit 62 transmits the image data captured by the imaging unit 22a or imaging unit 22b to the display device 10-2 (step S92). Specifically, the communication control unit 62 transmits the image data compressed by the encoder 34 to the display device 10-2. Then, the process proceeds to step S102.

[0080] If the result in step S82 is determined to be No, the face detection unit 56 calculates the position coordinates of the detected face (step S94). Here, in step S80, in the example shown in Figure 13, it is assumed that the face of user image UI2 was detected out of user image UI3 and user image UI2. The face detection unit 56 calculates the position coordinates of the face area of ​​user image UI2 displayed on the display unit 24. Then, the process proceeds to step S96.

[0081] The processes from step S96 to step S100 are the same as the processes from step S24 to step S28 shown in Figure 6, so their explanation will be omitted.

[0082] If the result in step S80 is No, the control unit 38 determines whether or not to terminate the process (step S102). Specifically, the control unit 38 determines to terminate the process when the remote conversation between users U1 and U4 and users U2 and U3 ends. If it is determined that the process should be terminated (step S102; Yes), the process in Figure 12 ends. If it is not determined that the process should be terminated (step S102; No), the process proceeds to step S72.

[0083] In other words, if the result in step S102 is determined to be No, the process from steps S72 to S92 is repeated. Therefore, when the speaker changes among user images UI2 and UI3 in the display unit 24, the imaging unit that captures user U1 and user U4 switches. As a result, even if the speaker changes among user images UI2 and UI3 in the display unit 24, images of user U1 and user U4 can be captured in a way that their gaze matches that of the speaker.

[0084] [Fourth Embodiment] A fourth embodiment will now be described. In the fourth embodiment, the brightness of image data captured by each imaging unit provided on the display unit is analyzed. In the fourth embodiment, if the brightness difference between image data captured by each imaging unit is greater than or equal to a predetermined value, processing is performed so that the brightness difference between image data captured by each imaging unit becomes less than or equal to the predetermined value.

[0085] An example of the configuration of the display device according to the fourth embodiment will be described using Figure 14. Figure 14 is a block diagram showing an example of the configuration of the display device according to the fourth embodiment.

[0086] As shown in Figure 14, the display device 10A differs from the display device 10 shown in Figure 2 in that the control unit 38A includes an image analysis unit 64.

[0087] The image analysis unit 64 performs image analysis processing on the image data acquired by the imaging control unit 50 from the imaging unit 22 and detects the brightness of the image data. The method for detecting the brightness of the image data is not limited to known methods and can be used.

[0088] (Processing details) The processing content of the display device according to the fourth embodiment will be explained using Figures 15 and 16. Figure 15 is a diagram illustrating the overview of the processing content of the display device according to the fourth embodiment. Figure 16 is a flowchart showing the processing content of the display device according to the fourth embodiment.

[0089] As shown in Figure 15, suppose the user image UI2, which is an image of user U2 displayed on the display unit 24, moves in the direction of the arrow. In this case, as the user image UI2 moves, the imaging unit that captures user U1 changes from imaging unit 22-1, imaging unit 22-2, imaging unit 22-3, imaging unit 22-4, to imaging unit 22-5. At this time, if there is a difference in brightness greater than a predetermined amount between the image data captured by imaging unit 22-1 to imaging unit 22-5, it is expected that when the imaging unit switches, other users viewing other display devices 10A that display the image data of user U1 will feel a sense of discomfort. Also, since each imaging unit 22 captures images while following user U1, for example, if the brightness of the background changes as user U1 moves, the brightness value of the image data may change. Examples of situations in which the brightness of the background changes as movement include when user U1 moves from a sunny area to a shady area outdoors, or when user U1 moves from a brightly lit area to a dimly lit area indoors.

[0090] In this embodiment, in order to avoid causing discomfort to other users who are viewing another display device 10A that displays user U1's image data, the imaging unit 22-1 performs a process to adjust the brightness of the image data captured by the imaging unit 22-2.

[0091] As shown in Figure 16, first, the imaging control unit 50 acquires image data from the imaging unit 22 (step S110). In the example shown in Figure 15, the imaging control unit 50 acquires image data captured by imaging unit 22-5 from imaging unit 22-1. Then, the process proceeds to step S112.

[0092] The image analysis unit 64 performs image analysis processing on the image data acquired by the imaging control unit 50 (step S112). In the example shown in Figure 15, the image analysis unit 64 performs image analysis processing on the image data acquired by the imaging control unit 50 from imaging units 22-1 to 22-5. Then, the process proceeds to step S114.

[0093] The image analysis unit 64 determines whether the brightness difference between image data is greater than or equal to a threshold (step S114). In the example shown in Figure 15, it determines whether the brightness difference between image data acquired from imaging unit 22-1 to imaging unit 22-5 is greater than or equal to a threshold. A brightness difference greater than or equal to a threshold means, for example, a brightness difference that causes discomfort to a user who has viewed images related to image data acquired from imaging unit 22-1 to imaging unit 22-5 consecutively. If it is determined that the brightness difference is greater than or equal to a threshold (step S114; Yes), the process proceeds to step S116. If it is determined that the brightness difference is not greater than or equal to a threshold (step S114; No), the process proceeds to step S118.

[0094] If Yes is determined in step S114, the imaging control unit 50 sets the image quality of the imaging unit 22 based on each image data (step S116). In the example shown in Figure 15, the imaging control unit 50 adjusts the gain of each imaging unit 22-1 to imaging unit 22-5 based on each image data acquired from imaging units 22-1 to 22-5. For example, the imaging control unit 50 adjusts the gain of each imaging unit 22-1 to imaging unit 22-5 so that the brightness of each image data captured by imaging units 22-1 to 22-5 is the same. For example, the imaging control unit 50 may adjust the gain of each imaging unit 22-1 to imaging unit 22-5 so that the difference in brightness values ​​of image data captured by adjacent imaging units 22 is less than or equal to a predetermined value. For example, the imaging control unit 50 may adjust the gains of imaging units 22-1 to 22-5 so that the brightness values ​​of the image data acquired from imaging units 22-1 to 22-5 change smoothly in sequence. Then, the process proceeds to step S120.

[0095] If the result in step S114 is determined to be No, the imaging control unit 50 sets each imaging unit 22 to the same setting so that the image quality displayed on the display unit 24 does not change when the imaging unit 22 is switched (step S118). In the example shown in Figure 15, the imaging control unit 50 sets the gain of each imaging unit 22-1 to imaging unit 22-5 to the same setting. Then, the process proceeds to step S120.

[0096] The control unit 38A determines whether or not to terminate the process (step S120). For example, the control unit 38A determines to terminate the process when the remote conversation between user U1 and user U2 ends. If it is determined that the process should be terminated (step S120; Yes), the process shown in Figure 16 is terminated. If it is not determined that the process should be terminated (step S120; No), the process proceeds to step S110.

[0097] The control unit 38A may perform the process shown in Figure 16 continuously while user U1 and user U2 are having a remote conversation, or it may perform it at predetermined intervals (for example, every 1 to 10 minutes).

[0098] As described above, the fourth embodiment adjusts the gain of each imaging unit that images user U1 either continuously or at predetermined intervals. This allows the fourth embodiment to capture images of user U1 that do not cause discomfort to users U2 or others who view the image data of user U1.

[0099] [Fifth Embodiment] A fifth embodiment will now be described. In the fifth embodiment, the imaging unit acquires the user's voice captured in the image, analyzes the acquired voice, and identifies the direction from which the voice was emitted. Based on the captured image of the user and the identified direction from which the voice was emitted, the fifth embodiment performs a process to identify the speaker.

[0100] (display device) An example of the configuration of the display device according to the fifth embodiment will be described using Figure 17. Figure 17 is a block diagram showing an example of the configuration of the display device according to the fifth embodiment.

[0101] As shown in Figure 17, the display device 10B differs from the display device 10 shown in Figure 2 in that the control unit 38B includes an audio data acquisition unit 66, a direction identification unit 68, and a speaker identification unit 70.

[0102] Figure 18 is a diagram illustrating the arrangement method of the microphone unit 26 according to the fifth embodiment. As shown in Figure 18, in the fifth embodiment, for example, multiple microphone units 26 are provided in the same plane as the display screen of the display unit 24. In the example shown in Figure 18, five microphone units 26 are provided at the bottom of the display unit 24. The number and position of the microphone units 26 are not limited as long as they are provided in the same plane as the display screen of the display unit 24. The multiple microphone units 26 detect, for example, the voice emitted by the user U1.

[0103] In the fifth embodiment, even if it is not possible to determine whether user U2 is the speaker from the image alone because the mouth is covered with a mask or the like, the display device 10B determines whether user U2 is the speaker. Specifically, the display device 10B determines whether user U2 is the speaker based on a user image UI2, which is an image of user U2 using another display device 10B, and audio data relating to user U2's voice.

[0104] The audio data acquisition unit 66 acquires audio data related to the voice detected by the microphone unit 26 of the other display device 10B from the other display device 10B. For example, the audio data acquisition unit 66 acquires audio data related to the voice of user U2 acquired by the other display device 10B.

[0105] The direction identification unit 68 identifies the direction from which the sound related to the sound data was emitted, based on the sound data detected by the sound data acquisition unit 66. For example, the direction identification unit 68 identifies the direction of the speaker who emitted the sound, based on the sound data related to the speaker's voice acquired by the sound data acquisition unit 66. In this embodiment, since the microphone unit 26 is provided on the same plane as the display unit 24, the direction identification unit 68 identifies the direction relative to the display unit 24.

[0106] The speaker identification unit 70 identifies the speaker who made the sound. The speaker identification unit 70 identifies the speaker based on the image data acquired by the image data acquisition unit 52 from another display device 10B and the direction identification result by the direction identification unit 68. Specifically, the speaker identification unit 70 identifies user U2 as the speaker when the direction of the user image UI2 acquired by the image data acquisition unit 52 matches the direction identified by the direction identification unit 68.

[0107] (Processing details) The processing details of the display device according to the fifth embodiment will be explained using Figure 19. Figure 19 is a flowchart showing the processing details of the display device according to the fifth embodiment.

[0108] The processes from step S130 to step S136 are the same as the processes from step S10 to step S16 shown in Figure 6, so their explanation will be omitted.

[0109] The audio data acquisition unit 66 acquires audio data from another display device 10B (step S138). Then, the process proceeds to step S140.

[0110] The processes in steps S140 and S142 are the same as those in steps S18 and S20 shown in Figure 6, respectively, so their explanation will be omitted.

[0111] If the result in step S142 is "Yes", the direction identification unit 68 performs voice analysis processing on the voice data acquired by the voice data acquisition unit 66 to determine the direction from which the voice was emitted (step S144). Then, the process proceeds to step S146.

[0112] The speaker identification unit 146 determines whether or not the speaker has been identified (step S146). If it is determined that the speaker has been identified (step S146; Yes), the process proceeds to step S148. If it is determined that the speaker has not been identified (step S146; No), the process proceeds to step S156.

[0113] The processes from steps S148 to S164 are identical to the processes from steps S86 to S102 shown in Figure 12, so their explanation will be omitted.

[0114] In this embodiment, the display device identifies the direction from which the sound was emitted based on audio data acquired from another display device. However, the other display device may identify the direction from which the sound was emitted based on the sound detected by the microphone unit 26. In this case, the display device acquires data regarding the direction from which the sound was emitted from the other display device, and the speaker identification unit 70 identifies user U2 as the speaker when the direction of the user image UI2 matches the direction from which the sound was emitted indicated by the acquired data.

[0115] As described above, the fifth embodiment can identify the speaker based on the image and audio, even when the speaker cannot be identified from the image alone. This allows the fifth embodiment to capture an image of user U1 in such a way that the gaze of user U2 is properly aligned with that of user U1.

[0116] Each component of the illustrated device is a functional concept and does not necessarily have to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. Furthermore, this distribution and integration configuration may be performed dynamically.

[0117] Although embodiments of the present invention have been described above, the present invention is not limited to the content of these embodiments. In the embodiments of the present invention, an LED display was described as a display device, but the display device is not limited to this. For example, a projector may be used as a display device, and a plurality of imaging units may be arranged on or behind the screen onto which the image is projected. In the case of a display device in which some display elements cannot be replaced with imaging units, such as a liquid crystal display or an organic EL (electro-luminescence), a plurality of imaging units may be arranged in front of the display unit. Furthermore, the aforementioned components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are equivalent. Furthermore, the aforementioned components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the gist of the embodiments described above. [Explanation of Symbols]

[0118] 1 Display System 10 Display device 20 Input section 22 Imaging Department 24 Display section 26 Microphone section 28 Audio output section 30 Storage section 32 Communications Department 34 encoders 36 Decoders 38 Control Unit 50 Imaging control unit 52 Image Data Acquisition Unit 54 Image Processing Unit 56 Face detection unit 58 Display Control Unit 60 Audio Output Control Unit 62 Communication Control Unit 64 Image Analysis Department 66 Audio data acquisition unit 68 Direction identification part 70. Speaker Identification Section

Claims

1. The display unit is an LED display that shows images, A plurality of imaging units are provided in place of the blue pixels included in the LED display on the display screen of the display unit, and capture images of the user positioned in the direction of the display screen, A display control unit that displays images of other users conversing with the aforementioned user on the display unit, A face detection unit that detects the face, including the eyes of the other user, from the image of the other user displayed on the display unit, An imaging control unit controls the imaging unit, which is located near the eye area of ​​the other user displayed on the display unit, to capture an image of the user. A communication control unit that transmits an image of the user captured by an imaging unit located near the eye area of ​​the other user, A display device equipped with the following features.

2. The image processing unit includes a function that crops an image of the user captured by the imaging unit located near the eye area of ​​the other user displayed on the display unit, to a range corresponding to the position where the imaging unit that captured the user is located. The display device according to claim 1.

3. The display unit includes an image analysis unit that analyzes the brightness of the image captured by each of the plurality of imaging units provided on the display unit, The imaging control unit adjusts the gain of each of the multiple imaging units according to the results obtained by the image analysis unit. The display device according to claim 1 or 2.

4. At least one microphone is provided near the display screen of the aforementioned display unit, A voice data acquisition unit that acquires voice data relating to the voice of the other user detected by the microphone of the other display device, A direction identification unit identifies the direction from which the other user's voice was emitted, based on the voice data acquired by the voice data acquisition unit. The system includes a speaker identification unit that identifies the other user as the speaker when the direction of the other user captured by the imaging unit of the other display device matches the direction identified by the direction identification unit, The imaging control unit controls the imaging unit, which is located near the eye area of ​​the other user who is the speaker identified by the speaker identification unit and whose image is displayed on the display unit, to take an image. The display device according to claim 1 or 2.

5. The steps of displaying an image on a display unit which is an LED display, The steps include: providing a plurality of imaging units in place of the blue pixels included in the LED display on the display screen of the display unit, and causing the imaging units to capture an image of the user positioned in the direction of the display screen; The steps include displaying images of other users who are conversing with the aforementioned user on the display unit, The steps include detecting the face of the other user, including its eyes, from the image of the other user displayed on the display unit, The steps include controlling the imaging unit, which is located near the eye area of ​​the other user displayed on the display unit, to capture an image of the user, The steps include transmitting an image of the user captured by an imaging unit located near the eye area of ​​the other user, Display methods, including those mentioned above.

Images