Display method, display device and display system

Abstract

A display method, a display device, and a display system are provided, which enable a conference to be conducted smoothly even when multiple speakers are speaking at the same time. The display method includes: displaying a first image corresponding to a first terminal (1A) and a second image corresponding to a second terminal (1B) in parallel in a first area; when a first voice detected by the first terminal (1A) overlaps with a second voice detected by the second terminal (1B), displaying a first text image representing the content of the first voice in correspondence with the first image in the first area, and displaying a second text image representing the content of the second voice in correspondence with the second image in the first area; and displaying the first text image in correspondence with the second image in the first area when an operation of moving the first text image to a second area different from the first area is accepted.

Description

Technical Field

[0001] The invention relates to a display method, a display device and a display system. Background Art

[0002] Conventionally, there are known voice conferencing systems and video conferencing systems using computers. For example, Patent Document 1 discloses a structure in which the voice level is adjusted according to the priority of the speaker in order to solve the problem that the voice becomes unclear when multiple participants speak at the same time in a voice conferencing system.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2007-96555

[0004] As described above, if the speeches of multiple speakers overlap, the voice becomes unclear and it is difficult to hear the content of the speech clearly. In response to this problem, in the structure disclosed in Patent Document 1, it is easy to hear the speech of the speaker with a high priority, but it is more difficult to hear the speeches of other speakers clearly. Therefore, when multiple speakers speak at the same time, it is necessary to ask a speaker to repeat his speech, which reduces the efficiency of the meeting. Summary of the invention

[0005] One embodiment of the present disclosure is a display method, comprising: displaying a first image corresponding to a first terminal and a second image corresponding to a second terminal side by side in a first area; in a case where a first voice detected by the first terminal overlaps with a second voice detected by the second terminal, displaying a first text image representing the content of the first voice in correspondence with the first image in the first area, and displaying a second text image representing the content of the second voice in correspondence with the second image in the first area; and in a case where an operation to move the first text image to a second area different from the first area is accepted, displaying the first text image in the second area.

[0006] Another embodiment of the present disclosure is a display device, comprising: a display; and a control circuit that controls the display, wherein the control circuit performs the following control: by controlling the display, a first image corresponding to a first terminal and a second image corresponding to a second terminal are displayed side by side in a first area; when a first voice detected by the first terminal overlaps with a second voice detected by the second terminal, by controlling the display, a first text image representing the content of the first voice is displayed in the first area in correspondence with the first image, and a second text image representing the content of the second voice is displayed in the first area in correspondence with the second image; and when an operation of moving the first text image to a second area different from the first area is accepted, the display is controlled to display the first text image in the second area.

[0007] Another embodiment of the present disclosure is a display system, comprising: a first terminal having a first microphone; a second terminal having a second microphone; and a third terminal having a display, wherein the third terminal is configured to: display a first image corresponding to the first terminal and a second image corresponding to the second terminal side by side in a first area of ​​the display, display a first text image representing the content of a first voice detected by the first terminal through the first microphone in the first area in correspondence with the first image, and display a second text image representing the content of a second voice detected by the second terminal through the second microphone in correspondence with the second image, and display the first text image in the second area when an operation to move the first text image from the first area to the second area of ​​the display is accepted. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 This is a diagram showing an example of the configuration of a conference system according to an embodiment.

[0009] Figure 2 This is an explanatory diagram showing an overview of the operation of the conference system.

[0010] Figure 3 It is an explanatory diagram showing an example of the execution status of a conference.

[0011] Figure 4 This is a block diagram of devices constituting a conference system.

[0012] Figure 5 It is a flowchart showing the operation of the server.

[0013] Figure 6 This is a diagram showing an example of a conference screen.

[0014] Figure 7 This is a diagram showing an example of a conference screen.

[0015] Figure 8 This is a diagram showing an example of a conference screen.

[0016] Fig. 9 This is a diagram showing an example of a conference screen.

[0017] Description of symbols

[0018] 1: terminal; 1A: first terminal; 1B: second terminal; 1C: third terminal (display device); 11: first control circuit; 12: processor; 13: memory; 14: first display (display); 15: input device; 15a: keyboard; 15b: mouse; 16: camera; 17: microphone (first microphone); 18: speaker; 19: communication device; 21: second control circuit; 22: processor; 23: memory; 24: second display (display); 25: input device 25a: keyboard; 26: camera; 27: microphone (second microphone); 28: speaker; 29: communication device; 31: third control circuit (control circuit); 32: processor; 33: memory; 34: third display (display); 35: input device; 35a: keyboard; 36: camera; 37: microphone; 38: speaker; 39: communication device; 50: server; 51: server control circuit; 52: processor; 53: memory; 54: communication device ; 61: conference screen; 62: conference screen; 64: first area; 65: second area; 100: conference system (display system); 532: conference record data; 621, 621c, 621d: user image; 621a: user image (first image); 621b: user image (second image); 631, 631a, 631b, 631c, 631d: logo (first text image, second text image); 651, 651a, 651b: reservation label (first text image); text image, second text image); D1, D1A, D1B, D1C: image data; D2, D2A, D2B, D2C: voice data; D3, D3A, D3B, D3C: operation data; D4, D4A, D4B, D4C: display data; D5, D5A, D5B, D5C: integrated voice data; OP1: input; OP2: input (select operation); S, S1, S2, S3: base; U, U1, U2A, U2B, U3: user; V1: speech. DETAILED DESCRIPTION

[0019] [1. Conference System Overview]

[0020] Hereinafter, the present embodiment will be described with reference to the drawings.

[0021] Figure 1 It is a diagram showing an example of the configuration of the conference system 100 according to the present embodiment.

[0022] The conference system 100 is a system in which a plurality of users U use computers to share voices and hold conferences. The conference system 100 includes a plurality of terminals 1 and a server 50 used by the plurality of users U. The terminals 1 and the server 50 are connected via a communication network 7 so as to be able to communicate data with each other. The conference system 100 corresponds to an example of a display system.

[0023] There is no limit to the number of terminals 1 included in the conference system 100. In addition, the server 50 may be a single computer, may be composed of a plurality of computers, or may be a cloud server.

[0024] In this embodiment, if Figure 1 As shown, an example of holding a conference using three terminals 1 is described. In the following description, the three terminals 1 are referred to as the first terminal 1A, the second terminal 1B, and the third terminal 1C. When they are not distinguished, they are recorded as terminal 1. The first terminal 1A, the second terminal 1B, and the third terminal 1C are computers with communication functions. Specifically, they are desktop PCs (Personal Computers), tablet PCs, smartphones, etc. The third terminal 1C corresponds to an example of a display device. The same applies to the first terminal 1A and the second terminal 1B.

[0025] exist Figure 1 In the example shown, the first terminal 1A is set at the base S1, the second terminal 1B is set at the base S2, and the third terminal 1C is set at the base S3. The geographical relationship between the bases S1, S2, and S3 is not limited. The bases S1, S2, and S3 can be separate places, places in the same building, or places separated from each other in the same room. In the following description, the base S is recorded without distinguishing the bases S1, S2, and S3.

[0026] There is no limit to the number of users U who use the terminal 1. For example, multiple users U may use one terminal 1 to attend a meeting. If the terminal 1 is a portable computer, the user U may carry the terminal 1 to use it. In this embodiment, one user U1 uses the first terminal 1A, two users U2A and U2B use the second terminal 1B, and one user U3 uses the third terminal 1C. When the users U1, U2A, U2B, and U3 are not distinguished, they are recorded as user U.

[0027] The communication network 7 may be a LAN (Local Area Network) or a WAN (Wide Area Network), or may be a global network including a dedicated line, a public line network, the Internet, and the like.

[0028] The first terminal 1A includes a first display 14, a keyboard 15a, a mouse 15b, a camera 16, a microphone 17, and a speaker 18. Each of these devices is connected to the main body of the first terminal 1A by wire or wirelessly. At least one of each device may also be integrally assembled in the main body of the first terminal 1A. In some cases, these devices are sometimes referred to as the first display 14 of the first terminal 1A, the microphone 17 of the first terminal 1A, etc. The first display 14 is a display device having a display panel such as a liquid crystal display panel, an organic EL (Electro-Luminescence) panel, or a plasma display panel. The keyboard 15a and the mouse 15b are input devices used by the user U1 in input operations. The camera 16 takes a picture of the user U1. The microphone 17 collects the voice of the user U1. The speaker 18 outputs the voice of the meeting. The user U1 uses these devices to participate in the meeting. The microphone 17 corresponds to an example of the first microphone.

[0029] The second terminal 1B includes a second display 24, a keyboard 25a, a camera 26, a microphone 27, and a speaker 28. Each of these devices is connected to the main body of the second terminal 1B by wire or wirelessly. The second display 24, the keyboard 25a, the camera 26, the microphone 27, and the speaker 28 are respectively configured similarly to the first display 14, the keyboard 15a, the camera 16, the microphone 17, and the speaker 18. The microphone 27 corresponds to an example of a second microphone.

[0030] The camera 26 takes pictures of the users U2A and U2B. The microphone 27 collects the voices of the users U2A and U2B. The speaker 28 outputs the voices of the conference. The users U2A and U2B participate in the conference using these devices.

[0031] The third terminal 1C includes a third display 34, a keyboard 35a, a camera 36, ​​a microphone 37, and a speaker 38. These devices are connected to the main body of the third terminal 1C by wire or wirelessly. The third display 34, the keyboard 35a, the camera 36, ​​the microphone 37, and the speaker 38 are configured similarly to the first display 14, the keyboard 15a, the camera 16, the microphone 17, and the speaker 18, respectively.

[0032] The camera 36 takes a picture of the user U3. The microphone 37 collects the voice of the user U3. The speaker 38 outputs the voice of the meeting. The user U3 uses these devices to participate in the meeting.

[0033] Figure 2 It is an explanatory diagram showing an overview of the operation of the conference system 100 .

[0034] The conference performed by the conference system 100 is conducted by multiple users U sharing at least each other's voice. In addition, the conference system 100 may also be a structure for sharing images and videos captured by the cameras 16, 26, and 36 to conduct the conference. In this embodiment, the conference system 100 shares the videos captured by the cameras 16, 26, and 36 to conduct the conference.

[0035] exist Figure 2 , as data transmitted from the first terminal 1A, the second terminal 1B, and the third terminal 1C to the server 50, image data D1A, D1B, and D1C, voice data D2A, D2B, and D2C, and operation data D3A, D3B, and D3C are shown. In the following description, when the image data D1A, D1B, and D1C are not distinguished, they are described as image data D1. Similarly, when the voice data D2A, D2B, and D2C are not distinguished, they are described as voice data D2, and when the operation data D3A, D3B, and D3C are not distinguished, they are described as operation data D3.

[0036] In addition, the server 50 sends display data D4A, D4B, D4C and integrated voice data D5A, D5B, D5C to the first terminal 1A, the second terminal 1B, and the third terminal 1C. In the following description, when display data D4A, D4B, D4C are not distinguished, they are recorded as display data D4, and when integrated voice data D5A, D5B, D5C are not distinguished, they are recorded as integrated voice data D5.

[0037] The first terminal 1A transmits image data D1A based on the image captured by the camera 16 and voice data D2A based on the voice collected by the microphone 17 to the server 50. In addition, when the first terminal 1A receives an operation from the keyboard 15a or the mouse 15b, it transmits operation data D3A based on the received operation to the server 50.

[0038] The second terminal 1B transmits the video data D1B, the audio data D2B, and the operation data D3B to the server 50 in the same manner as the first terminal 1A. The third terminal 1C transmits the video data D1C, the audio data D2C, and the operation data D3C to the server 50.

[0039] The server 50 distributes the voice of the conference of the conference system 100 to each terminal 1. The server 50 generates integrated voice data D5 based on the voice data D2A, D2B, and D2C. The integrated voice data D5 includes the voice detected by the first terminal 1A through the microphone 17, the voice detected by the second terminal 1B through the microphone 27, and the voice detected by the third terminal 1C through the microphone 37. The first terminal 1A, the second terminal 1B, and the third terminal 1C output the voice from the speakers 18, 28, and 38 based on the integrated voice data D5. As a result, all users U using the conference system 100 can hear the voices of other users U and can conduct a conference based on voice.

[0040] In order to prevent howling at the base S, the server 50 may also make the integrated voice data D5 sent to the first terminal 1A, the second terminal 1B, and the third terminal 1C different data. For example, the server 50 may also send the integrated voice data D5A including the voice based on the voice data D2B and D2C but not including the voice based on the voice data D2A to the first terminal 1A. In this case, the first terminal 1A outputs the voice detected by the second terminal 1B and the voice detected by the third terminal 1C from the speaker 18. The user U1 can hear the voices uttered by the users U2A, U2B, and U3. In addition, the first terminal 1A does not output the voice uttered by the user U1 himself from the speaker 18, so the user U1 will not feel a sense of disharmony or the voice of the user U1 will not echo, and the meeting can be carried out smoothly.

[0041] In this case, the server 50 may also send the integrated voice data D5B including the voice based on the voice data D2A and D2C but not including the voice based on the voice data D2B to the second terminal 1B. In addition, the server 50 may also send the integrated voice data D5C including the voice based on the voice data D2A and D2B but not including the voice based on the voice data D2C to the third terminal 1C.

[0042] The server 50 sends the display data D4A to the first terminal 1A. The first terminal 1A displays an image on the first display 14 based on the display data D4A. Similarly, the server 50 sends the display data D4B to the second terminal 1B and sends the display data D4C to the third terminal 1C. The second terminal 1B displays an image on the second display 24 based on the display data D4B, and the third terminal 1C displays an image on the third display 34 based on the display data D4C.

[0043] Figure 3 2 is an explanatory diagram showing an example of the execution status of a meeting, and shows the execution status of the meeting at the base S3. Figure 3 In the example shown, a conference screen 61 is displayed on the third display 34 . The conference screen 61 is a screen arranged in the display area 34 a of the third display 34 .

[0044] The conference screen 61 is provided with a display frame 611a and a display frame 611b. The display frames 611a and 611b are regions for displaying images related to the terminal 1 other than the third terminal 1C. The display frame 611a is a region corresponding to the first terminal 1A, and the display frame 611b is a region corresponding to the second terminal 1B. When the display frames 611a and 611b and the display frame 611c described later are not distinguished, they are described as the display frame 611.

[0045] The user image 621a is displayed in the display frame 611a. The user image 621a is an image corresponding to the user U1 who uses the first terminal 1A. The user image 621a is not limited to the image captured by the camera 16. The user image 621a, as an image corresponding to the user U1, may also be an image stored in the first terminal 1A or the server 50. The user image 621a may also be an image generated by the server 50 based on the image data D1A.

[0046] User images 621b and 621c are displayed in the display frame 611b. The user image 621b is an image corresponding to the user U2A using the second terminal 1B, and the user image 621c is an image corresponding to the user U2B. The user images 621b and 621c may be images stored in the second terminal 1B or the server 50. The user images 621b and 621c may be images generated by the server 50 based on the image data D1B. In the case where the user images 621a, 621b, 621c and the user image 621d described later are not distinguished, they are recorded as user images 621.

[0047] In this embodiment, a label 631a is configured in the display frame 611a. The label 631a is a text indicating the content of the voice collected by the microphone 17. For example, the server 50 performs voice recognition processing on the voice data D2A, converts the voice data D2A into text data, and thereby generates the label 631a. The conference system 100 updates the content of the label 631a, for example, every time the user U1 speaks. In this case, the label 631a is a text image indicating the content of one speech of the user U1.

[0048] In addition, labels 631b and 631c are arranged in the display frame 611b. Labels 631b and 631c are texts indicating the content of the voice collected by the microphone 27. For example, the server 50 performs voice recognition processing on the voice data D2B, converts the voice data D2B into text data, and thereby generates labels 631b and 631c. The microphone 27 collects the voices of the users U2A and U2B. Therefore, the voices uttered by multiple characters are included in the voice data D2B. It is recorded as label 631 without distinguishing between labels 631a, 631b, 631c and label 631d described later. Label 631 corresponds to an example of a first text image and a second text image.

[0049] In the conference system 100 , the speech included in the speech data D2B is classified for each user U. Figure 3 The labels 631b and 631c shown represent the content of the speech collected by the microphone 27. Label 631b represents the content of the speech of one of the two users U2A and U2B, and label 631c represents the content of the speech uttered by the user U2 different from label 631b. Therefore, the speakers corresponding to labels 631b and 631c are different, so labels 631b and label 631c are displayed in different display modes. That is, label 631b is displayed in a first visual representation, and label 631c is displayed in a second visual representation. In this case, label 631a is an example of a first text image representing the content of the first speech, and label 631b is an example of a third text image representing the content of the third speech.

[0050] The visual representation of label 631 refers to features that can be recognized by the vision of user U. Specifically, the visual representation includes at least one of the attributes of the font, the shape of the frame, and the decoration. Examples of font attributes include font, font size, and font color. The shape of the frame can include the presence or absence of the frame, the color of the frame, the thickness of the frame, and the size of the frame. The decoration can include the background color of the font, the decoration on the outside of the frame, and the decoration displayed temporarily. Label 631b and label 631c are displayed with different visual representations in a manner that can be visually distinguished.

[0051] The third display 34 displays the conference screen 61, and the user U3 can view the images of the users U1, U2A, and U2B who are other participants in the conference. The user U3 can have a conversation with each other by giving a speech V1 while viewing the conference screen 61. In addition, when the user images 621a, 621b, and 621c are images based on the image data D1A and D1B, the user U3 can view the status of the users U1, U2A, and U2B who are other participants in the conference by viewing the display frames 611a and 611b.

[0052] Figure 3 The conference screen 61 shown is an example, and for example, a display frame 611 corresponding to the user U3 may also be configured. Specifically, in the conference screen 61, a display frame 611 corresponding to the user U3 is configured at a position that does not overlap with the display frames 611a and 611b. Then, a user image 621 corresponding to the user U3 and a label 631 indicating the content of the speech of the user U3 are displayed in the display frame 611.

[0053] The labels 631a, 631b, and 631c may be displayed at all times or only when a predetermined condition is satisfied. The predetermined condition may be, for example, that the user U3 sets the labels 631a, 631b, and 631c to be displayed, or that the speeches of multiple users U overlap. When the timing of speeches of multiple users U overlap, it is difficult to hear the speech output from the speaker 38 clearly. In such a case, by displaying the labels 631a, 631b, and 631c, the content of the speech can be visually confirmed, thereby facilitating a smooth meeting.

[0054] During the conference being performed by the conference system 100, the first terminal 1A and the second terminal 1B display the conference screen 61 similarly to the third terminal 1C. For example, the first terminal 1A displays the conference screen 61 having the display frames 611 corresponding to the second terminal 1B and the third terminal 1C through the first display 14. Also, for example, the second terminal 1B displays the conference screen 61 having the display frames 611 corresponding to the first terminal 1A and the third terminal 1C through the second display 24.

[0055] [2. Structure of devices constituting the conference system]

[0056] Figure 4 It is a block diagram of devices constituting the conference system 100 .

[0057] like Figure 4 As shown, the first terminal 1A has a first control circuit 11 , and the first control circuit 11 is connected to a first display 14 , an input device 15 , a camera 16 , a microphone 17 , a speaker 18 and a communication device 19 .

[0058] The input device 15 is a device used by the user U for input operations, such as a keyboard 15 a and a mouse 15 b . The input device 15 may be a touch sensor provided to overlap with the display panel of the first display 14 , or may be other devices.

[0059] The communication device 19 is a device that is connected to the communication network 7 and performs data communication with the server 50 via the communication network 7. The communication device 19 includes, for example, a connector for connecting a communication cable and a communication interface circuit. The communication device 19 may also be a structure that includes an antenna and a wireless communication circuit and is connected to the communication network 7 via a wireless communication line.

[0060] The first control circuit 11 includes a processor 12 and a memory 13. The processor 12 is composed of a CPU (Central Processing Unit), an MPU (Micro-processing unit), etc. The processor 12 controls each unit of the first terminal 1A by executing a program.

[0061] The memory 13 is a storage device that stores programs and data executed by the processor 12 in a non-volatile manner. The memory 13 is composed of a semiconductor storage element such as a magnetic storage device, a flash ROM (Read Only Memory), or other types of non-volatile storage devices. The memory 13 may also include a RAM (Random Access Memory) that constitutes a work area of ​​the processor 12. The memory 13 stores data processed by the first control circuit 11 and a control program executed by the processor 12.

[0062] The processor 12 obtains the image data from the camera 16 and generates the image data D1A based on the image data. The processor 12 converts the voice collected by the microphone 17 into digital voice data. The processor 12 generates the voice data D2A based on the digital voice data. The processor 12 receives the input from the input device 15 and generates the operation data D3A based on the received input.

[0063] The processor 12 controls the communication device 19 to transmit the image data D1A, the audio data D2A, and the operation data D3A to the server 50 .

[0064] Processor 12 receives display data D4A and integrated voice data D5A transmitted from server 50 via communication device 19. Processor 12 controls first display 14 based on display data D4A, thereby causing first display 14 to display conference screen 61. Processor 12 causes speaker 18 to output voice based on integrated voice data D5A.

[0065] The second terminal 1B and the third terminal 1C are configured similarly to the first terminal 1A. The second terminal 1B includes a second control circuit 21 connected to a second display 24 , an input device 25 , a camera 26 , a microphone 27 , a speaker 28 , and a communication device 29 .

[0066] The input device 25 is a device used by the user U for input operations, and is, for example, a keyboard 25 a . The input device 25 may be a touch sensor provided to overlap with the display panel of the second display 24 , or may be other devices.

[0067] The communication device 29 is a device having the same structure as the communication device 19, and is connected to the communication network 7 to perform data communication with the server 50 via the communication network 7. The communication device 29 includes, for example, a connector for connecting a communication cable and a communication interface circuit. The communication device 29 may also include an antenna and a wireless communication circuit and be connected to the communication network 7 via a wireless communication line.

[0068] The second control circuit 21 includes a processor 22 and a memory 23. The processor 22 is composed of a CPU, an MPU, etc. The processor 22 controls each part of the second terminal 1B by executing a program. The specific structure of the processor 22 is the same as that of the processor 12, and the specific structure of the memory 23 is the same as that of the memory 13.

[0069] Processor 22 obtains the image data from camera 26 and generates image data D1B based on the image data. Processor 22 converts the voice collected by microphone 27 into digital voice data. Processor 22 generates voice data D2B based on the digital voice data. Processor 22 receives input from input device 25 and generates operation data D3B based on the received input.

[0070] The processor 22 controls the communication device 29 to transmit the image data D1B, the voice data D2B, and the operation data D3B to the server 50 .

[0071] Processor 22 receives display data D4B and integrated voice data D5B transmitted from server 50 via communication device 29. Processor 22 controls second display 24 based on display data D4B, thereby causing second display 24 to display conference screen 61. Processor 22 causes speaker 28 to output voice based on integrated voice data D5B.

[0072] The third terminal 1C has a third control circuit 31 connected to a third display 34, an input device 35, a camera 36, ​​a microphone 37, a speaker 38, and a communication device 39. The third display 34 corresponds to an example of a display, and the third control circuit 31 corresponds to an example of a control circuit.

[0073] The input device 35 is a device used by the user U for input operations, and is, for example, a keyboard 35 a . The input device 35 may be a touch sensor provided to overlap with the display panel of the third display 34 , or may be other devices.

[0074] The communication device 39 is a device having the same structure as the communication device 19, and is connected to the communication network 7 to perform data communication with the server 50 via the communication network 7. The communication device 39 includes, for example, a connector for connecting a communication cable and a communication interface circuit. The communication device 39 may also be a structure having an antenna and a wireless communication circuit and connected to the communication network 7 via a wireless communication line.

[0075] The third control circuit 31 includes a processor 32 and a memory 33. The processor 32 is composed of a CPU, an MPU, etc. The processor 32 controls each part of the third terminal 1C by executing a program. The specific structure of the processor 32 is the same as that of the processor 12, and the specific structure of the memory 33 is the same as that of the memory 13.

[0076] The processor 32 obtains the image data from the camera 36 and generates the image data D1C based on the image data. The processor 32 converts the voice collected by the microphone 37 into digital voice data. The processor 32 generates the voice data D2C based on the digital voice data. The processor 32 receives the input from the input device 35 and generates the operation data D3C based on the received input.

[0077] The processor 32 controls the communication device 39 to transmit the image data D1C, the voice data D2C, and the operation data D3C to the server 50 .

[0078] Processor 32 receives display data D4C and integrated voice data D5C transmitted from server 50 via communication device 39. Processor 32 controls third display 34 based on display data D4C, thereby causing third display 34 to display conference screen 61. Processor 32 causes speaker 38 to output voice based on integrated voice data D5C.

[0079] The server 50 has a server control circuit 51, and the server control circuit 51 is connected to a communication device 54. The communication device 54 is connected to the communication network 7, and performs data communication with the terminal 1 via the communication network 7. The communication device 54, for example, includes a connector for connecting a communication cable and a communication interface circuit. The communication device 54 may also be a structure that includes an antenna and a wireless communication circuit and is connected to the communication network 7 through a wireless communication line.

[0080] The server control circuit 51 includes a processor 52 and a memory 53. The processor 52 is composed of a CPU, an MPU, etc. The processor 52 controls each part of the third terminal 1C by executing a program. The specific structure of the processor 52 is the same as that of the processor 12, and the specific structure of the memory 53 is the same as that of the memory 13.

[0081] The processor 52 executes the control program 531 stored in the memory 53 to function as a communication control unit 521 , a terminal identification unit 522 , a speech analysis unit 523 , a display data generation unit 524 , and a data processing unit 525 .

[0082] The communication control unit 521 controls the communication device 54 to perform data communication with the terminal 1. The communication control unit 521 receives the image data D1, the voice data D2, and the operation data D3 transmitted from the terminal 1 through the communication device 54. In addition, as described later, the communication control unit 521 transmits the display data D4 and the integrated voice data D5 generated by the processor 52 to the terminal 1.

[0083] When receiving the image data D1 from the terminal 1, the terminal identification unit 522 identifies the terminal 1 that sent the image data D1. Similarly, when receiving the voice data D2, the terminal identification unit 522 identifies the terminal 1 that sent the voice data D2, and when receiving the operation data D3 from the terminal 1, the terminal identification unit 522 identifies the terminal 1 that sent the operation data D3.

[0084] The speech analysis unit 523 analyzes the speech data D2 received by the communication device 54, and converts the speech of the user U included in the speech data D2 into text data representing the speech content of the user U. The speech analysis unit 523 determines whether the speech included in the speech data D2 is the speech of one user U or the speech of multiple users U. When the speech data D2 includes the speech of multiple users U, the speech analysis unit 523 classifies the speech included in the speech data D2 according to the user U, and generates text data representing the speech content of each user U.

[0085] The speech analysis unit 523 generates integrated speech data D5 based on the speech data D2 received from the plurality of terminals 1. The speech analysis unit 523 may also generate integrated speech data D5 common to all the terminals 1. In addition, the speech analysis unit 523 may also generate integrated speech data D5A, D5B, and D5C including mutually different speech. For example, the speech analysis unit 523 may also generate integrated speech data D5A that does not include the speech collected by the microphone 17, integrated speech data D5B that does not include the speech collected by the microphone 27, and integrated speech data D5C that does not include the speech collected by the microphone 37.

[0086] The display data generation unit 524 generates the display data D4. The display data generation unit 524 generates user images 621 corresponding to each user U participating in the conference. For example, the display data generation unit 524 generates a user image 621a corresponding to the user U1 based on the image captured by the camera 16. In this case, the display data generation unit 524 generates the user image 621a by performing resolution adjustment of the image data D1A, etc. In addition, the display data generation unit 524 may use a static image stored in advance corresponding to the user U1 as the user image 621a. The display data generation unit 524 similarly generates user images 621 corresponding to the users U2A, U2B, and the user U3 using the conference system 100, respectively.

[0087] The display data generation unit 524 generates display data D4 for displaying the conference screen 61 using the user images 621 corresponding to the users U participating in the conference using the conference system 100. The display data generation unit 524 arranges the display frames 611 corresponding to the terminals 1 participating in the conference and arranges the user images 621 in the display frames 611, thereby generating the display data D4.

[0088] The display data generation unit 524 can generate display data D4 including the text data generated by the voice analysis unit 523. In this case, the display data generation unit 524 generates the display data D4 by superimposing the text data generated by the voice analysis unit 523 on the user image 621 of the user U corresponding to the text data. In addition, the display data generation unit 524 can also generate the display data D4 for displaying the text data in a specific visual expression.

[0089] The display data generation unit 524 may generate display data D4 common to all terminals 1 participating in the conference, or may generate display data D4 different for each terminal 1. Specifically, the display data generation unit 524 may generate display data D4A, D4B, and D4C different from each other. In this case, the display data D4A includes the user images 621 of the users U other than the user U1, but does not include the user image 621a corresponding to the user U1.

[0090] Furthermore, the display data generating unit 524 generates display data D4 for displaying a conference screen 62 including a first area 64 and a second area 65 described later, based on the operation data D3. The conference screen 62, the first area 64, and the second area 65 will be described later.

[0091] The data processing unit 525 generates the meeting record data 532 or updates the meeting record data 532 based on the operation data D3 , and stores the meeting record data 53 in the memory 53 .

[0092] [3. How the conference system works]

[0093] Figure 5 is a flow chart showing the operation of the server 50. In this embodiment, Figure 2 1 and 2 will now be described an example in which the server 50 receives the image data D1 , the voice data D2 , and the operation data D3 from the terminal 1 , and generates and transmits the display data D4 and the integrated voice data D5 . Figure 5 The operations shown are executed by the server control circuit 51 included in the server 50 .

[0094] Figure 51 shows a process of generating one display data D4. This process is executed when any one of the display data D4A, D4B, and D4C is generated. Therefore, when the server 50 generates the display data D4A, D4B, and D4C, the display data D4 is executed. Figure 5 In addition, when the server 50 sends the common display data D4 to all terminals 1, Figure 5 The processing generates common display data D4.

[0095] In step ST11, the server control circuit 51 detects speech based on the speech data D2 received from the terminal 1, and determines whether the user U speaks. If it is determined that there is no speech (step ST11 is No), the server control circuit 51 returns to step ST11 and executes the determination of step ST11 after a predetermined time. If it is determined that there is speech (step ST11 is Yes), the server control circuit 51 transfers to step ST12.

[0096] In step ST12, the server control circuit 51 analyzes the voice detected from the voice data D2. For example, the server control circuit 51 generates text data representing the content of the voice data D2 in step ST12. In step ST13, the server control circuit 51 determines the user U who spoke and the terminal 1 used by the user U who spoke. In step ST13, the server control circuit 51 detects at least the voice detected in step ST11 to determine the terminal 1 that sent the voice data D2, and establishes a correspondence between the voice detected in step ST11 and the terminal 1. For example, the server control circuit 51 identifies the terminal 1 that sent the voice data D2 based on the identification information of the terminal 1 included in the voice data D2, and establishes a correspondence between the identified terminal 1 and the voice included in the voice data D2.

[0097] In step ST14, the server control circuit 51 determines whether the detected voices are overlapped by the voices of two or more users U. If it is determined that there is no overlap of the voices of two or more users U (No in step ST14), the server control circuit 51 transfers to step ST15. In step ST15, the server control circuit 51 determines whether the text display setting is set to "ON" in the terminal 1 that receives the display data D4.

[0098] By operating the input device 35 by the user U3, the third terminal 1C can set the text display on and off. When the text display is set to "on", the third terminal 1C displays a conference screen 61 including a label 631. In addition, when the text display is turned off, the label 631 is not displayed. That is, the conference screen 61 without the label 631 is displayed on the third display 34. When the text display in the third terminal 1C is set to "on" or "off", the third terminal 1C sends operation data D3C including data indicating the setting state of the text display to the server 50. The same is true for the first terminal 1A and the second terminal 1B. Therefore, the server control circuit 51 can determine the setting state of the text display based on the operation data D3 sent from the terminal 1.

[0099] When it is determined in step ST15 that the text display setting is not “ON” (NO in step ST15 ), the server control circuit 51 proceeds to step ST19 described later.

[0100] If it is determined in step ST15 that the text display setting is "ON" (step ST15: YES), the server control circuit 51 transfers to step ST16. In step ST16, the server control circuit 51 generates an image of the label 631 based on the text data representing the speech content of the user U. In step ST16, the server control circuit 51 further determines the arrangement between the label 631 and the user image 621 so that the label 631 overlaps with the user image 621 corresponding to the terminal 1 determined in step ST13, and transfers to step ST19.

[0101] When the server control circuit 51 determines in step ST14 that the voices of two or more users U overlap (step ST14: Yes), the process proceeds to step ST17. In step ST17, the server control circuit 51 determines whether the voice data D2 transmitted from one terminal 1 includes the voices of two or more users U.

[0102] When the server control circuit 51 determines that the voice data D2 sent from one terminal 1 contains the voices of more than two users U (step ST17 is yes), it transfers to step ST18. In step ST18, the server control circuit 51 generates a label 631 based on the text data representing the content of the voice data D2 sent from one terminal 1. In step ST18, the server control circuit 51 generates a plurality of labels 631 corresponding to the speech of more than two users U. Furthermore, the server control circuit 51 gives different visual expressions to the plurality of labels 631 generated in step ST18 and determines the configuration of each label 631. In detail, the server control circuit 51 determines the display position of each label 631 in such a way that the plurality of labels 631 do not overlap each other. Then, the configuration of the label 631 is determined in such a way that these plurality of labels 631 overlap with the user image 621 corresponding to the terminal 1 identified in step ST13.

[0103] After the processing of step ST18, the server control circuit 51 shifts to step ST19.

[0104] When the server control circuit 51 determines in step ST17 that the voice data D2 transmitted from one terminal 1 does not include the voices of two or more users U (No in step ST17 ), the process proceeds to step ST16 .

[0105] In step ST19, the server control circuit 51 generates or updates the display data D4 based on the arrangement between the user image 621 and the tag 631 determined in step ST16 or step ST18, and transmits it to the terminal 1. Thus, the conference screen is displayed on the terminal 1 or the conference screen is updated.

[0106] In step ST20 , the server control circuit 51 analyzes the operation data D3 transmitted from the terminal 1 , and determines whether an input for moving the tab 631 outside the first area 64 has been performed while the conference screen is displayed.

[0107] Here, refer to Figure 6 to Figure 9 The details of the input to the tag 631 will be described.

[0108] Figure 6 , Figure 7 , Figure 8 as well as Fig. 9 FIG. 6 is a diagram showing an example of the conference screen 62. Figure 6 to Figure 9 2 shows an example in which the third terminal 1C displays the conference screen 62 on the third display 34 .

[0109] like Figure 6As shown, the conference screen 62 includes a first area 64 and a second area 65. A user image 621a corresponding to the user U1, and user images 621b and 621c corresponding to the users U2A and U2B are arranged in the first area 64. The user image 621a is arranged in a display frame 611a corresponding to the first terminal 1A. In addition, the user images 621b and 621c are arranged in a display frame 611b corresponding to the second terminal 1B. The user image 621a corresponds to an example of the first image, and the image corresponding to the second terminal 1B including the user image 621b and the user image 621c corresponds to an example of the second image.

[0110] exist Figure 6 , a conference screen 62 is shown in the figure when the user U2 sets the text display to "on" in the third terminal 1C, and when the timing of the speeches of multiple users U overlap. Labels 631a, 631b, and 631c are configured on the conference screen 62. Label 631a is an image representing the content of the voice collected by the first terminal 1A through the microphone 17, and is therefore configured to overlap with the user image 621a. Labels 631b and 631c are images representing the content of the voice collected by the second terminal 1B through the microphone 27, and are therefore configured to overlap with the user images 621b and 621c. Label 631b and label 631c are displayed with different visual expressions. The server control circuit 51 displays the labels 631b and 631c in the display frame 611b in a manner that does not determine which of the user images 621b and 621c the labels 631b and 631c correspond to, respectively. In addition, the server control circuit 51 may also determine which of the user images 621b and 621c the labels 631b and 631c correspond to, respectively.

[0111] A display frame 611c corresponding to the third terminal 1C is arranged on the conference screen 62. A user image 621d corresponding to the user U3 using the third terminal 1C and a label 631d indicating the content of the voice collected by the third terminal 1C through the microphone 37 are arranged on the display frame 611c. Whether to arrange the display frame 611c can be changed by inputting the input device 35, for example. That is, it can be changed by inputting the input device 35. Figure 3 The conference screen displayed on the third display 34 does not include a display corresponding to the third terminal 1C, and Figure 6 In this way, switching is performed between the structures including the display corresponding to the third terminal 1C.

[0112] In this way, the first area 64 is an area where images related to the user U participating in the conference using the conference system 100 are displayed.

[0113] In the conference screens 61 and 62, when the speeches of multiple users U overlap, labels 631a to 631d are displayed as text images indicating the content of the speeches. Thus, even if it is difficult to hear the speeches in the conference clearly, the user U can easily know the content of the speeches. Therefore, there are fewer scenes of repeated questions about the content of the speeches or repeated speeches, so the conference can proceed smoothly.

[0114] Furthermore, the user U can display the labels 631 a to 631 d even when the speech does not overlap by setting the text display to “ON”, thereby allowing the user to visually check the content of the speech.

[0115] The second area 65 is an area where the selected tag 631 among the tags 631a to 631d displayed in the first area 64 is displayed. Figure 6 As shown in the middle mark OP1, when an input is made to move the label 631a to the second area 65, a reserved label 651 having the same content as the label 631a is displayed in the second area 65. Figure 7 The state in which the reservation label 651 is displayed in the second area 65 is shown. The input OP1 corresponds to an example of an operation of moving the text image to the second area.

[0116] One or more reservation labels 651 can be displayed in the second area 65. Figure 7 In the example, a text image representing the same content as the label 631a is displayed as a reserved label 651a in the second area 65. When the reserved label 651a is not distinguished from the reserved label 651b described later, it is recorded as a reserved label 651. The reserved label 651 corresponds to an example of the first text image and the second text image.

[0117] The label 631 displayed in the first area 64 is updated every time the user U speaks. In contrast, the reserved label 651 continues to be displayed in the second area 65 regardless of the update of the label 631. Therefore, by displaying the reserved label 651 in the second area 65, the user U can continue to visually see the content of the label 631 selected by the input OP1.

[0118] The terminal 1 that accepts the input OP1 is not limited, and the input OP1 can be performed in all the terminals 1 participating in the conference. Figure 6 In the conference screen 62 , when an input OP1 is input through the input device 15 of the first terminal 1A and when an input OP1 is input through the input device 25 of the second terminal 1B, the reservation label 651 is also displayed in the second area 65 .

[0119] The display of the second area 65 is displayed on all the terminals 1 participating in the conference of the conference system 100. Therefore, all the users U participating in the conference can visually see the display of the second area 65. For example, when the input OP1 is performed by the operation of the user U3, the first terminal 1A, the second terminal 1B, and the third terminal 1C all display the second area 65. This operation is realized by the server control circuit 51 configuring the second area 65 and the reservation tag 651 in all the display data D4A, D4B, and D4C.

[0120] The terminal 1 may not display the second area 65 during the period when the input OP1 is not performed. Specifically, the terminal 1 may not display the second area 65 but only display the first area 64 before the input OP1 is performed in any of the first terminal 1A, the second terminal 1B, and the third terminal 1C. In this case, the conference screen 62 becomes a screen having the first area 64 but not the second area 65. This display method can be realized by the server control circuit 51 generating display data D4 containing only the content of the first area 64 and sending it to the terminal 1.

[0121] In this case, when the input OP1 is accepted in any of the first terminal 1A, the second terminal 1B, and the third terminal 1C, the display of the second area 65 starts. The method of input OP1 is not limited to the operation of moving the label 631 from the first area 64 to the second area 65, as long as it is an operation of moving the label 631 outside the first area 64. In addition, when the label 631 is moved outside the first area 64 by the input OP1, the server control circuit 51 generates display data D4 including the first area 64 and the second area 65 and provided with the reserved label 651 corresponding to the selected label 631, and transmits it to the terminal 1.

[0122] Figure 8 as well as Fig. 9 A display example when an input to the second area 65 is accepted is shown.

[0123] In terminal 1, such as Figure 8 As shown, an input OP2 for the reservation label 651 displayed in the second area 65 can be accepted. The input OP2 is an input for displaying other speeches in the second area 65 in correspondence with the reservation label 651. The input OP2 is, for example, an operation using a pointing device such as the mouse 15b or a touch panel. Specifically, the input OP2 is an operation of clicking the reservation label 651, an operation of selecting the reservation label 651 using a menu screen for the reservation label 651, and the like. The input OP2 corresponds to an example of a selection operation.

[0124] For example, when the second terminal 1B receives the input OP2 through the input device 25, Fig. 9As shown, a retention label 651b indicating the content of the voice collected by the microphone 27 after the input OP2 is displayed in the second area 65. Fig. 9 , a label 631b indicating the content of the speech collected by the microphone 27 is displayed in the first area 64. Furthermore, a retention label 651b which is a text image indicating the same content as the label 631b is displayed in the second area 65. In this way, when the terminal 1 accepts an input OP2 for the retention label 651a, the retention label 651b indicating the content of the speech collected after the input OP2 is displayed in the second area 65 in correspondence with the retention label 651a.

[0125] In this way, after the reservation tag 651a is arranged in the second area 65, the reservation tag 651b indicating the content of the speech of the user U is arranged in the second area 65 in correspondence with the reservation tag 651a. By inputting OP2, the user U can make the content of his / her own speech be displayed as the reservation tag 651b in the second area 65. Similarly, other users U can make the text image indicating the content of the speech of the user U be displayed as a new reservation tag 651 in the second area 65 by inputting OP2 to the reservation tag 651a or the reservation tag 651b.

[0126] All users U participating in the meeting can make input OP2. For example, user U1 can make input OP2 to the tag 631 corresponding to all users U including user U1. The same applies to users U2A, U2B, and U3. Thus, all users U participating in the meeting can select the reservation tag 651 to display their speech in the second area 65 in association with the reservation tag 651.

[0127] By displaying a specific speech as a reservation label 651 in the second area 65 during the meeting, the conversation related to the specific topic can be continued. For example, only a part of the users U participating in the meeting can continue the discussion on a specific topic. In this way, the meeting dealing with many topics can be carried out more smoothly.

[0128] In addition, the user image 621 and the label 631 corresponding to the user U using the terminal 1 may be displayed in the first area 64. For example, the third terminal 1C may display the user image 621d and the label 631d of the user U3 in the display frame 611c. In this case, the user U3 can visually confirm the content of the speech of the user U3. Furthermore, the user U3 can make the content of the speech of the user U3 a reserved label 651 by performing the input OP1 to move the label 631d to the second area 65.

[0129] exist Figure 5In step ST20, the server control circuit 51 determines whether the input OP1 has been performed. When the server control circuit 51 determines that an operation to move the label 631 has been performed, that is, when it is determined that the input OP1 has been performed (step ST20 is yes), the server control circuit 51 transfers to step ST21. In step ST21, the server control circuit 51 configures the label 631 that will be the object of the input OP1 in the second area 65. Next, in step ST22, the server control circuit 51 adds the content of the label 631 moved to the second area 65 to the meeting record data 532, updates the meeting record data 532, and transfers to step ST23. In addition, when the server control circuit 51 determines that an operation to move the label 631 has not been performed (step ST20 is no), the server control circuit 51 transfers to step ST23.

[0130] The conference record data 532 is data recording the content of the conference. The server control circuit 51 saves the text data included in the reservation tag 651 as a record of the conference conducted by the conference system 100. In steps ST21 to ST22, the server control circuit 51 adds the text data included in the newly arranged reservation tag 651 to the conference record data 532 and stores it in the memory 53 every time the reservation tag 651 is arranged in the second area 65.

[0131] In step ST23, the server control circuit 51 determines whether the speech analyzed in step ST12 is speech uttered after input OP2 of the selection reservation tag 651. If it is determined that the speech is not uttered after input OP2 (step ST23: No), the server control circuit 51 moves to step ST26 described later.

[0132] If it is determined that the speech is uttered after the input OP2 (Yes in step ST23), the server control circuit 51 transfers to step ST24. In step ST24, the server control circuit 51 arranges the reservation tag 651 having the same content as the tag 631 arranged in step ST16 or step ST18 in parallel with the reservation tag 651 selected in the input OP2 in the second area 65. In the next step ST25, the server control circuit 51 adds the content of the reservation tag 651 arranged in step ST24 to the conference record data 532, updates the conference record data 532, and transfers to step ST26.

[0133] In step ST26 , the server control circuit 51 updates the display data D4 , and transmits the updated display data D4 to the terminal 1 .

[0134] exist Figure 5In steps ST23 to ST24 of the embodiment, an example is described in which the server control circuit 51 displays the reservation label 651 indicating the content of the speech in the second area 65 in response to the speech uttered after the input OP2 of selecting the reservation label 651, but this operation can also be canceled. For example, in the first terminal 1A, the second terminal 1B, and the third terminal 1C, a cancel operation of canceling the selection of the reservation label 651 can be accepted after the input OP2. The cancel operation is, for example, an operation performed by the user U using the input device 15, 25, or 35. Specifically, the cancel operation can include an operation of the ESC key of the keyboard 15a and an operation similar to the input OP2.

[0135] When the first terminal 1A receives the release operation, the speech of the user U1 detected by the microphone 17 is not displayed as the hold label 651 in the second area 65. The content of the speech of the user U1 may also be displayed as the label 631 in the first area 64. In addition, when the text display setting is "off", the content of the speech of the user U1 is not displayed in the first area 64. The same is true for the second terminal 1B and the third terminal 1C.

[0136] When the selection of the reservation label 651 based on the input OP2 can be cancelled by accepting the cancellation operation, during the meeting, it is possible to switch between an operation of displaying the content of the speech as the reservation label 651 and an operation of not displaying the content as the reservation label 651. Thus, the discussion on specific content and the overall discussion of the meeting can be used separately, and further improvement in convenience can be achieved.

[0137] [4. Effects of implementation methods]

[0138] As described above, the display method performed in the conference system 100 includes: displaying the user image 621a corresponding to the first terminal 1A and the user image 621b corresponding to the second terminal 1B in parallel in the first area 64. The display method performs a specific display when the first voice detected by the first terminal 1A overlaps with the second voice detected by the second terminal 1B. The specific display includes: displaying the label 631a representing the content of the first voice in the first area 64 in correspondence with the user image 621a, and displaying the label 631b representing the content of the second voice in the first area 64 in correspondence with the user image 621b. In addition, it also includes: when an operation of moving the label 631a to the second area 65 different from the first area 64 is accepted, the label 631a is displayed as a reserved label 651a in the second area 65.

[0139] According to the display method, when the voice collected by the first terminal 1A overlaps with the voice collected by the second terminal 1B, by displaying the labels 631a and 631b, the user U can visually recognize the content of the speech even when the speech overlaps and becomes difficult to hear clearly. Therefore, the phenomenon of delaying the meeting due to the request for re-speech, etc. can be suppressed. In addition, according to the operation, the label 631a indicating the content of the speech is displayed in the second area 65 different from the first area 64 displaying the user image 621. Therefore, other users U can focus on specific content in the content of the speech in the meeting. As a result, the meeting using the conference system 100 can be carried out smoothly. For example, by allowing the user U participating in the meeting to see the label 631, the content of the meeting can be accurately identified, and by displaying the reservation label 651a in the second area 65, the discussion on the specific content can be deepened.

[0140] In the above display method, the first terminal 1A displays the first area 64 and the second area 65 in the display area of ​​the first display 14. Furthermore, when the first terminal 1A receives an input OP1 for moving the label 631a displayed in the first area 64 of the first display 14 to the second area 65, the label 631a may be displayed as a reserved label 651a in the second area 65 through the first display 14. In this case, the user U1 using the first terminal 1A can move the label 631a indicating the content of the speech of the user U1 to the second area 65, so that other users U2A, U2B, and U3 can pay attention to the content of the speech of the user U1. As a result, the user U1 can continue to discuss the content of the speech without repeating the content of the speech of the user U1, so that the meeting can proceed more smoothly.

[0141] In the above display method, displaying the label 631a and the label 631b in the first area 64 includes displaying the first area 64 in the display area of ​​the third display 34 of the third terminal 1C. Displaying the label 631a in the second area 65 includes: when the third terminal 1C receives an operation to move the label 631a to the second area 65, displaying the label 631a as a reserved label 651a in the second area 65.

[0142] In this case, the user U3 using the third terminal 1C can move the label 631a indicating the content of the speech of the user U1 to the second area 65, so that other users U1, U2A, and U2B can pay attention to the content of the speech of the user U1. Thus, even if the user U1 or the user U3 does not repeat the content of the speech of the user U1, the discussion of the content can continue, so that the meeting can proceed more smoothly.

[0143] In the above display method, when the first voice detected by the first terminal 1A through the microphone 17 and the second voice detected by the second terminal 1B through the microphone 27 do not overlap, the labels 631a and 631b may not be displayed in the first area 64. In this case, by simplifying the display in the first area 64, it can be expected that the user U can easily concentrate on listening to the voice and pay attention to the user image 621.

[0144] In the above display method, displaying the label 631a in correspondence with the user image 621a includes: displaying the label 631a in an overlapping manner on the user image 621a. In addition, displaying the label 631b in correspondence with the user image 621b includes: displaying the label 631b in an overlapping manner on the user image 621b. Since the user image 621 and the label 631 are displayed in an overlapping manner, it is easy to understand the correspondence between the label 631 indicating the content of the speech and the user image 621 of the user U who made the speech, so that the meeting can be carried out more smoothly.

[0145] The display method includes: after displaying the label 631a, when the first terminal 1A detects a new first voice, updating the label 631a displayed in the first area 64 to a new label 631a based on the content of the new first voice; and not updating the reserved label 651a displayed in the second area 65. Thus, by updating the label 631 of the first area 64 according to the speech of the user U, the display of the first area 64 can follow the progress of the meeting. Therefore, the meeting can proceed more smoothly. In addition, since the reserved label 651 of the second area 65 is not updated according to the speech, the user U can continue to see the content of the specific speech. Therefore, it can be expected that the effect of easily continuing to discuss the specific content can be expected.

[0146] The display method includes: receiving a selection operation, i.e., inputting OP2, of selecting the reservation label 651a displayed in the second area 65 by the second terminal 1B. Furthermore, when the second terminal 1B receives the input OP2, the reservation label 651b indicating the content of the second voice detected by the second terminal 1B through the microphone 27 after the input OP2 is displayed in the second area 65. Thus, the user U can make a speech in association with the reservation label 651 displayed in the second area 65, and the content of the speech can be displayed in the second area 65. Therefore, it is easy to discuss specific content, and thus the meeting can be conducted more smoothly.

[0147] In the above display method, after the selection operation, the second terminal 1B accepts a release operation to release the selection of the reservation label 651a displayed in the second area 65. When the release operation is accepted by the second terminal 1B, the label 631b indicating the content of the second voice detected by the second terminal 1B after the release operation is displayed in the first area 64 in correspondence with the user image 621b. Thus, the selection of the reservation label 651 can be released. Therefore, during the meeting, it is possible to switch between the action of displaying the content of the speech as the reservation label 651 and the action of not displaying the content as the reservation label 651. Thus, the discussion on the specific content and the overall discussion of the meeting can be used separately, and further improvement of convenience can be achieved.

[0148] In the above display method, after the selection operation, the second terminal 1B accepts a release operation to release the selection of the first text image displayed in the second area 65. The above display method may also be a method in which, when the release operation is accepted by the second terminal 1B, the text image representing the content of the second speech detected by the second terminal 1B after the release operation is not displayed. In this case, the selection of the reservation label 651 can be released. Therefore, during the meeting, it is possible to switch between the action of displaying the content of the speech as the reservation label 651 and the action of not displaying the reservation label 651. In addition, the display of the conference screen 62 after the selection of the reservation label 651 is released can be simplified.

[0149] In the above display method, the display of the second area 65 may be started when the operation of moving the tab 631 to the outside of the first area 64, that is, the input OP1 is received. In this case, before the input OP1 is performed, the display area of ​​the first display 14, the second display 24, or the third display 34 can be used for the display of the first area 64. Therefore, the meeting can be conducted more smoothly.

[0150] The display method includes: the second terminal 1B obtains the first voice and the third voice through the microphone 27. In addition, the display method includes: displaying the label 631b representing the content of the first voice in a first visual representation, and displaying the label 631c representing the content of the third voice in a second visual representation different from the first visual representation. Therefore, the labels 631 corresponding to the voices of multiple users U are displayed in different visual representations for each user U. Therefore, other users U can visually know whether the multiple labels 631 are the contents of the speeches of different users U or the contents of the speech of one user U.

[0151] The third terminal 1C as a display device includes a third display 34 and a third control circuit 31 for controlling the third display 34. The third control circuit 31 performs the following control: by controlling the third display 34, the user image 621a corresponding to the first terminal 1A and the user image 621b corresponding to the second terminal 1B are displayed in parallel in the first area 64; when the first voice detected by the first terminal 1A and the second voice detected by the second terminal 1B overlap, the third display 34 is controlled to display a first text image, i.e., a label 631a, which is a content of the first voice, in correspondence with the user image 621a, and a second text image, i.e., a label 631b, which is a content of the second voice, in correspondence with the user image 621b, in the first area 64; and when an input OP1 for moving the label 631a to a second area 65 different from the first area 64 is accepted, the third display 34 is controlled to display a reserved label 651a corresponding to the label 631a in the second area 65.

[0152] Thus, when the voice collected by the first terminal 1A overlaps with the voice collected by the second terminal 1B, the third display 34 of the third terminal 1C displays the label 631a in correspondence with the user image 621a, and displays the label 631b in correspondence with the user image 621b. Therefore, even when the speech overlaps and becomes difficult to hear clearly, the user U3 can visually recognize the content of the speech, so it is possible to suppress the phenomenon of delaying the meeting due to the request for re-speaking, etc. In addition, according to the input OP1, the label 631a indicating the content of the speech is displayed in the second area 65 different from the first area 64 displaying the user image 621. Therefore, by the user U3 inputting OP1, it is possible to make other users U1, U2A, and U2B pay attention to the specific content of the speech in the meeting. Thus, the meeting using the conference system 100 can be smoothly conducted. For example, by making the user U participating in the meeting see the label 631, the content of the meeting can be accurately recognized, and by displaying the reservation label 651a in the second area 65, the discussion on the specific content can be deepened.

[0153] The conference system 100 includes a first terminal 1A having a microphone 17, a second terminal 1B having a microphone 27, and a third terminal 1C having a third display 34. The third terminal 1C displays a user image 621a corresponding to the first terminal 1A and a user image 621b corresponding to the second terminal 1B in parallel in a first area 64 of the third display 34. The third terminal 1C displays a label 631a indicating the content of a first voice detected by the first terminal 1A through the microphone in correspondence with the user image 621a in the first area 64, and displays a label 631b indicating the content of a second voice detected by the second terminal 1B through the microphone 27 in correspondence with the user image 621b. When an operation of moving the label 631a from the first area 64 to the second area 65 of the third display 34, i.e., input OP1, is accepted, a reservation label 651a is displayed in the second area 65.

[0154] Thus, when the voice collected by the first terminal 1A overlaps with the voice collected by the second terminal 1B, the label 631a is displayed in correspondence with the user image 621a in the first area 64 of the third display 34, and the label 631b is displayed in correspondence with the user image 621b. Therefore, even when the speech overlaps and becomes difficult to hear clearly, the user U3 can visually recognize the content of the speech, so it is possible to suppress the phenomenon of delaying the meeting due to the request for re-speech, etc. In addition, according to the input OP1, the label 631a indicating the content of the speech is displayed in the second area 65 different from the first area 64 displaying the user image 621. Thus, the meeting using the conference system 100 can be smoothly conducted. For example, by allowing the user U participating in the meeting to see the label 631, the content of the meeting can be accurately recognized, and by displaying the reservation label 651a in the second area 65, the discussion on the specific content can be deepened.

[0155] In the conference system 100, when the third terminal 1C displays a label 631a and a user image 621a in correspondence with each other in the first area 64, and displays a label 631b and a user image 621b indicating the content of a second voice detected by the second terminal 1B through the microphone 27 in correspondence with each other, the second terminal 1B can also display the user image 621a in the first area 64 of the second display 24, and display a label 631a and a user image 621a in correspondence with each other in the first area 64.

[0156] In this case, the user image 621a and the label 631a are displayed in the first area 64 of the third display 34, and the user image 621a and the label 631a are displayed in correspondence in the first area 64 of the second display 24. As a result, multiple users U participating in a conference based on the conference system 100 can visually see the user U who has spoken and the content of the speech through the user image 621a and the label 631a displayed in the first area 64 of the display. Therefore, the content of the speech in the meeting can be shared with other users U, so that the meeting can proceed smoothly.

[0157] In the conference system 100, when the third terminal 1C receives an operation to move the label 631a from the first area 64 to the second area 65 of the third display 34, i.e., input OP1, the third terminal 1C may display the reserved label 651a in the second area 65, and the second terminal 1B may display the reserved label 651a corresponding to the label 631a in the second area 65 of the second display 24.

[0158] In this case, according to the input OP1, the label 631a indicating the content of the speech is displayed in the second area 65 of the third display 34, and the label 631a is displayed in the second area 65 of the second display 24. Therefore, the input OP1 can be used as an opportunity for multiple users U participating in the conference of the conference system 100 to share the specific content of the speech in the conference and make them pay attention to each other. As a result, the conference using the conference system 100 can be smoothly carried out. For example, by allowing the user U participating in the conference to see the label 631, the content of the conference can be accurately recognized, and by displaying the reservation label 651a in the second area 65, the discussion on the specific content can be deepened.

[0159] [5. Other embodiments]

[0160] The above-described embodiments are specific examples to which the present invention is applied, and the present invention is not limited thereto.

[0161] For example, in the above embodiment, if Figure 2As shown, an example is described in which the server 50 performs the function of receiving the image data D1, the voice data D2 and the operation data D3 from the terminal 1 and generating and sending the display data D4 and the integrated voice data D5. The structure of the present disclosure is not limited to this. For example, any one of the first terminal 1A, the second terminal 1B and the third terminal 1C can also perform the function of the server 50. That is, it can also be a structure in which any one of the first terminal 1A, the second terminal 1B and the third terminal 1C has a communication control unit 521, a terminal identification unit 522, a voice analysis unit 523, a display data generation unit 524 and a data processing unit 525 that are equivalent to the functional units of the server control circuit 51. In addition, it can also be configured that the first terminal 1A has the same function as the voice analysis unit 523, analyzes the voice detected by the microphone 17 to generate text data, includes the text data in the voice data D2A and sends it to the server 50. In this case, the server 50 can omit the process of analyzing the voice data D2A to generate text data. In addition, the second terminal 1B and the third terminal 1C can also form the same structure.

[0162] In addition, the second terminal 1B may be configured to have the same function as the speech analysis unit 523, and analyze and determine whether the speech detected by the microphone 27 includes the speech of the user U2A and the speech of the user U2B. In this case, the second terminal 1B generates text data based on the speech detected by the microphone 27, and associates the generated text data with the user U2A and the user U2B who are the speakers. The second terminal 1B causes the speech data D2B to include the text data of the speech detected by the microphone 27 and data indicating the association between each text data and the users U2A and U2B, and transmits the speech data D2B to the server 50. In this case, the server 50 can omit the process of analyzing the speech data D2B to identify the user U who is speaking.

[0163] The content and data format of the display data D4 sent by the server 50 to the terminal 1 are not limited. For example, the display data D4A may be data of the conference screens 61 and 62 displayed on the first display 14. In addition, the display data D4A is data required for the first terminal 1A to display the conference screens 61 and 62, and the first terminal 1A may also perform processing to generate data of the conference screens 61 and 62 based on the display data D4A. The same is true for the display data D4B and the display data D4C.

[0164] in addition, Figure 4The functional units shown show functional structures and do not limit specific installation methods. For example, the server 50 does not need to install hardware corresponding to the functional units of the server control circuit 51, and can of course be configured to implement the functions of multiple functional units by executing a program through a processor. In addition, in the above-mentioned embodiments, a part of the functions implemented by software can also be implemented by hardware, or a part of the functions implemented by hardware can also be implemented by software. In addition, the specific detailed structures of other parts of the conference system 100 can also be arbitrarily changed within the scope of not departing from the main purpose.

[0165] In addition, for example, Figure 5 The step units of the actions shown are divided according to the main processing contents in order to facilitate understanding of the actions of the conference system 100. The present disclosure is not limited to the division method and name of the processing units. Depending on the processing contents, it can also be divided into multiple step units. In addition, it can also be divided into one step unit containing multiple processes. In addition, the order of the steps can also be appropriately changed within the scope that does not hinder the main purpose of the present disclosure.

Claims

1. A display method, wherein: The display method comprises: Displaying a first image corresponding to the first terminal and a second image corresponding to the second terminal in parallel in the first area; When a first voice detected by the first terminal overlaps with a second voice detected by the second terminal, a first text image representing the content of the first voice is displayed in the first area in correspondence with the first image, and a second text image representing the content of the second voice is displayed in the first area in correspondence with the second image; When an operation of moving the first text image to a second area different from the first area is accepted, displaying the first text image in the second area; and When the second terminal receives a selection operation for selecting the first text image displayed in the second area, a text image indicating the content of the second voice detected by the second terminal after the selection operation is displayed in the second area.

2. The display method according to claim 1, wherein: Displaying the first text image and the second text image in the first area includes: displaying the first area on a display of a third terminal, Displaying the first text image in the second area includes displaying the first text image in the second area when the third terminal accepts an operation to move the first text image to the second area.

3. The display method according to claim 1 or 2, wherein: Displaying the first text image and the first image in correspondence includes: displaying the first text image and the first image in an overlapping manner, Displaying the second text image in correspondence with the second image includes: displaying the second text image and the second image in an overlapping manner.

4. The display method according to claim 1 or 2, wherein: The display method comprises: After the first text image is displayed in the first area, if the first terminal detects a new first voice, updating the first text image displayed in the first area to a new first text image based on the content of the new first voice; and The first text image displayed in the second area is not updated.

5. The display method according to claim 1, wherein: After the selection operation, when the second terminal receives a cancellation operation for canceling the selection of the first text image displayed in the second area, a text image indicating the content of the second audio is displayed in the first area in association with the second image.

6. The display method according to claim 1, wherein: After the selection operation, when the second terminal receives a release operation for releasing the selection of the first text image displayed in the second area, the text image representing the content of the second audio is not displayed.

7. The display method according to claim 1 or 2, wherein: When an operation to move the first text image outside the first area is accepted, display of the second area starts.

8. The display method according to claim 1 or 2, wherein: The display method further includes: The first terminal obtains the first voice and the third voice through a microphone; and The first text image representing the content of the first speech is displayed in a first visual representation, and a third text image representing the content of the third speech is displayed in a second visual representation different from the first visual representation.

9. A display device, wherein: The display device comprises: Display; and a control circuit for controlling the display, The control circuit performs the following control: By controlling the display, a first image corresponding to the first terminal and a second image corresponding to the second terminal are displayed in parallel in the first area; When a first voice detected by the first terminal overlaps with a second voice detected by the second terminal, the display is controlled to display a first text image representing the content of the first voice in correspondence with the first image in the first area, and a second text image representing the content of the second voice in correspondence with the second image in the first area; When an operation of moving the first text image to a second area different from the first area is accepted, controlling the display to display the first text image in the second area; and When the second terminal receives a selection operation for selecting the first text image displayed in the second area, the display is controlled to display a text image indicating the content of the second voice detected by the second terminal after the selection operation in the second area.

10. A display system, wherein: The display system comprises: a first terminal having a first microphone; a second terminal having a second microphone; and a third terminal having a display, The third terminal is constituted as follows: Displaying a first image corresponding to the first terminal and a second image corresponding to the second terminal in parallel in a first area of ​​the display, A first text image representing the content of a first voice detected by the first terminal through a first microphone is displayed in the first area in correspondence with the first image, and a second text image representing the content of a second voice detected by the second terminal through a second microphone is displayed in the first area in correspondence with the second image, When an operation to move the first text image from the first area to the second area of ​​the display is accepted, the first text image is displayed in the second area; When the second terminal receives a selection operation for selecting the first text image displayed in the second area, a text image indicating the content of the second voice detected by the second terminal after the selection operation is displayed in the second area.

Images