Display control device and program

Abstract

To suitably communicate with a speaker who speaks a language different from that of the speaker.SOLUTION: A display control device includes an utterance text acquirer that acquires text uttered by each of two or more speakers who speak different languages, and a display controller that displays, for each of the speakers, an utterance display group including a self-utterance display section in which text uttered by the speaker is displayed in a language used by the speaker, and an other-speaker utterance display section in which text uttered by another speaker is displayed in a language used by the other speaker.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a display control device and a program. [Background technology]

[0002] In recent years, multilingual automatic translation (machine translation) technology has advanced and is used in a variety of situations. For example, Patent Document 1 discloses a technology for communicating between multiple languages ​​by translating the speech content of a speaker located on one side of a transparent display and displaying it to a speaker located on the other side, and translating the speech content of the speaker located on the other side and displaying it to a speaker located on the one side. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2024-40187 Summary of the Invention [Problem to be solved by the invention]

[0004] Machine translation functions can make translation errors, resulting in inaccurate translation of spoken content, depending on the length of the sentence being translated, differences in language systems, and the presence or absence of prerequisite knowledge for the conversation. In conversations between speakers of different languages, if a translation error occurs and the conversation continues without noticing the translation error, it can lead to major misunderstandings in communication between the two parties.

[0005] However, the technology described in Patent Document 1 has a problem in that the translation result of the user's own speech is displayed in reversed horizontally, making it difficult to check for translation errors.

[0006] The present invention has been made in consideration of the above-mentioned points, and aims to provide a technology that enables a person to communicate favorably with a speaker who speaks a language different from the person's own. [Means for solving the problem]

[0007] One aspect of the present invention is a display control device that includes a speech content acquisition unit that acquires the content spoken by each of two or more speakers who speak different languages, and a display control unit that displays a speech display group that includes, for each speaker, a self-speech display unit that displays the content spoken by that speaker in sentences in the speaker's own language, and an other speaker's speech display unit that displays the content spoken by other speakers in sentences translated into the speaker's own language.

[0008] In addition, in one aspect of the present invention, the speech display group has a similar area within each speech display group, and the own speech display unit and the other speaker speech display unit are displayed at predetermined positions within the area.

[0009] In one aspect of the present invention, at least one of the self-utterance display unit and the other speaker's utterance display unit is provided with a display that identifies whether it is the self-utterance display unit or the other speaker's utterance display unit.

[0010] In addition, in one aspect of the present invention, the display control device further includes an image acquisition unit that acquires an image of a speaker, and the display control unit displays the speech display group for the speaker superimposed on the image of the speaker.

[0011] In addition, in one aspect of the present invention, the display control device further includes an image acquisition unit that acquires an image of a speaker, and the display control unit displays the image of the speaker in the vicinity of the speech display group for that speaker among each of the speech display groups.

[0012] In addition, in one aspect of the present invention, the speech content acquisition unit acquires the speech content of each speaker based on the sound picked up by a microphone provided for each speaker, and the display control unit displays the speech display group for each speaker using the microphone.

[0013] In one aspect of the present invention, the microphones are arranged in a row in a predetermined direction, and the display control unit displays the speech display group for each speaker in the direction in the order of the microphones used by each speaker.

[0014] In one aspect of the present invention, the different languages ​​include dialects.

[0015] In one aspect of the present invention, the speech contents acquired by the speech content acquisition unit are in different languages, that is, in three or more languages.

[0016] Another aspect of the present invention is a program that causes a computer to execute a speech content acquisition step of acquiring the content spoken by each of two or more speakers who speak different languages, and a display control step of displaying a group of speech displays for each speaker, including a self-speech display section in which the content spoken by that speaker is displayed in sentences in the speaker's own language, and an other speaker's speech display section in which the content spoken by other speakers is displayed in sentences translated into the speaker's own language. [Effects of the Invention]

[0017] According to the present invention, it is possible to preferably communicate with a speaker who speaks a language different from one's own. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 2 is a diagram illustrating an example in which a plurality of speakers who speak different languages ​​converse in an embodiment. [Figure 2] FIG. 3 is a diagram for explaining an example of a first display screen displayed on the display device according to the embodiment. [Figure 3] 1 is a schematic diagram for explaining an aspect of a simultaneous display system according to an embodiment. [Figure 4] FIG. 1 is a block diagram illustrating an example of a functional configuration of a display control device according to an embodiment. [Figure 5]FIG. 10 is a diagram for explaining an example of a second display screen displayed on the display device according to the embodiment. [Figure 6] FIG. 10 is a diagram for explaining an example of a third display screen displayed on the display device according to the embodiment. [Figure 7] 10 is a flowchart illustrating an example of a processing flow of a display control device according to an embodiment. [Figure 8] 1 is a block diagram showing an example of an internal configuration of a display control device according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0019] [Embodiment] A display control device and a program according to the present embodiment will be described in detail below with reference to the accompanying drawings, showing preferred embodiments. In the drawings, identical or similar parts are designated by identical or similar reference numerals. Note that the present embodiment is not limited to these embodiments and includes various modifications or improvements. In other words, the components described below include those that can be easily imagined by a person skilled in the art and those that are substantially identical, and the components described below can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components of the present embodiment may be made without departing from the spirit of the present invention.

[0020] First, the technology according to the embodiment will be briefly described with reference to FIGS.

[0021] 1 is a diagram illustrating an example of a conversation between multiple speakers S who speak different languages ​​in an embodiment. A first speaker S1 and a second speaker S2 are examples of the multiple speakers S. Hereinafter, when there is no need to distinguish between the first speaker S1 and the second speaker S2, they may be simply referred to as speakers S.

[0022] Multiple speakers S converse using different languages. Different languages ​​are languages ​​with different linguistic systems. The term "language" used in this specification may include official languages ​​designated for official use in each country and official state languages ​​used in each state, as well as languages ​​other than official languages ​​that are not designated as official languages ​​and are used only in specific regions or ethnic groups (hereinafter referred to as "non-official languages"). For example, "language" may include a standard language (or common language) that is used throughout the country, a dialect (accent) that is used in certain local communities within the country and differs from the standard language in terms of phonology, grammar, or vocabulary, or a language used by certain local communities or ethnic groups within the country and whose linguistic system is significantly different from the standard language (such as the Ainu language in Japan). Hereinafter, different languages ​​may be collectively referred to as "multilingual languages."

[0023] In other words, multilingual languages ​​may include cases where two or more languages ​​have different names, such as an official language and an official language, or an official language and a non-official language. Multilingual languages ​​may also include cases where two or more languages ​​have the same name but differ in part in phonology, grammar, or vocabulary depending on the country or region, such as British English and American English, or standard Japanese and a dialect.

[0024] 1, a first speaker S1 and a second speaker S2 are side by side conversing while looking at content displayed on a display device 40 such as a monitor. Hereinafter, directions may be described using a three-dimensional coordinate system in which the viewing direction of speaker S in a horizontal plane with the ground is the x-direction, the direction perpendicular to the x-direction in the horizontal plane is the y-direction from the first speaker S1 to the second speaker S2, and the normal direction to the horizontal plane is the z-direction.

[0025] The speakers S speak into a microphone 10 provided for each speaker S. The microphone 10 used by the first speaker S1 may be referred to as the first microphone 10-1, and the microphone 10 used by the second speaker S2 may be referred to as the second microphone 10-2. The voices picked up by the respective microphones 10 are converted into text and then translated into the language spoken by the first speaker S1 and the language spoken by the second speaker S2. The translated text is output to a display device 40. Hereinafter, the language spoken by the speaker S may be referred to as the language used.

[0026] The camera 20 captures images of the first speaker S1 and the second speaker S2 who are in the −x direction. The captured images of the first speaker S1 and the second speaker S2 are output to the display device 40.

[0027] The display device 40 is, for example, a display. The display device 40 displays the translated text and images of the first speaker S1 and the second speaker S2. In the following description, the image of the speaker S displayed on the display device 40 may be referred to as a speaker image SI. Furthermore, the speaker image SI of the first speaker S1 may be referred to as a first speaker image SI1, and the speaker image SI of the second speaker S2 may be referred to as a second speaker image SI2.

[0028] 1, the image captured by camera 20 is displayed in a horizontally inverted manner. When the image is displayed in a horizontally inverted manner, the second speaker image SI2 is displayed on display device 40 to the right (y direction) of the first speaker image SI1 as seen from the first speaker S1. This makes the direction from which the voice of second speaker S2 is heard the same as the direction in which the second speaker image SI2 is projected, allowing the first speaker S1 to have a conversation without feeling uncomfortable, as if they were having a conversation while looking into a mirror.

[0029] 2 is a diagram illustrating an example of a first display screen D1 displayed on the display device 40 according to the embodiment. The utterance display group D10 has a screen configuration including a first speaker image SI1, a second speaker image SI2, a first utterance display group D11, and a second utterance display group D12.

[0030] An utterance display group D10 is displayed for each speaker S. The first utterance display group D11 is the utterance display group D10 for the first speaker S1, and the second utterance display group D12 is the utterance display group D10 for the second speaker S2. The first utterance display group D11 includes a self-utterance display section D111, another speaker's utterance display section D112, and an identification display D113. The second utterance display group D12 also includes a self-utterance display section D111, another speaker's utterance display section D112, and an identification display D113.

[0031] The own utterance display section D111 displays the contents of the speaker's own utterance in a sentence in the speaker's own language. The speaker is the speaker S of the utterance display group D10. That is, the own utterance display section D111 displays a sentence in which the contents of the utterance of the speaker S of the display target are transcribed. For example, the own utterance display section D111 in the first utterance display group D11 displays the contents of the utterance of the first speaker S1 in the language of the first speaker S1 (hereinafter, may be referred to as the first language).

[0032] The other speaker's utterance display section D112 displays the content of an utterance made by another speaker S in a sentence translated into the speaker's own language. The other speaker S is one or more speakers other than the speaker S to be displayed in the utterance display group D10. For example, the other speaker's utterance display section D112 in the first utterance display group D11 displays the content of an utterance made in the language used by the second speaker S2 (hereinafter, sometimes referred to as the second language) in a sentence translated into the first language. That is, the first utterance display group D11 displays the content of the conversation between the first speaker S1 and the second speaker S2 in the first language, and the second utterance display group D12 displays the content of the conversation between the first speaker S1 and the second speaker S2 in the second language. Note that in FIG. 2, the sentences displayed in the own utterance display section D111 and the other speaker's utterance display section D112 are abbreviated.

[0033] The own utterance display section D111 and the other speaker utterance display section D112 are displayed in order of utterance time. For example, the own utterance display section D111 and the other speaker utterance display section D112 may be displayed in such a manner that the lower they are on the first display screen D1, the more recent the utterance content, and the higher they are, the older the utterance content. Furthermore, the own utterance display section D111 and the other speaker utterance display section D112 are displayed superimposed on the speaker image SI of the speaker S to be displayed (overlay display).

[0034] The identification display D113 is a display that distinguishes between an own utterance display section D111 that shows the content of an utterance by the first speaker S1 and an other speaker utterance display section D112 that shows the content of an utterance by another speaker S, in the first utterance display group D11. The identification display D113 is, for example, a red circle attached to the own utterance display section D111.

[0035] [Display control device details] Next, the display control device 30 that realizes the above-mentioned technology will be described in detail with reference to FIG. 3 and subsequent figures.

[0036] 3 is a schematic diagram illustrating an aspect of a simultaneous display system 1 according to an embodiment. The simultaneous display system 1 includes a plurality of microphones 10, a camera 20, a display control device 30, and a display device 40. A first microphone 10-1 and a second microphone 10-2 exemplify the plurality of microphones 10.

[0037] The microphone 10, the camera 20, the display control device 30, and the display device 40 may be connected to each other via a network NW, or may be connected to each other by wire. The network NW is a predetermined communication network configured using lines. The network NW is, for example, the Internet.

[0038] It is desirable that the camera 20 captures images of multiple speakers S in high image quality (for example, 4K) and at a high frame rate (for example, 60 fps). By capturing images in high image quality, the speaker S looking at the display device 40 tends to feel as if he or she is talking to the speaker S behind the display device 40, as if in a mirror, and communication can be carried out smoothly. It is also desirable that the face and upper body of the speaker S are captured so that the other speakers S can see the facial expressions and body language of the speaker S.

[0039] The display device 40 may be an image display device such as a display or a projector, or may be an information processing device such as a smartphone, a tablet, or a personal computer. The display screen of the display device 40 is arranged horizontally in a direction perpendicular to the viewing direction of the speaker S (y direction).

[0040] 4 is a block diagram illustrating an example of a functional configuration of a display control device 30 according to an embodiment. The display control device 30 includes a voice acquisition unit 31, a voice processing unit 32, a speech content acquisition unit 33, a translation unit 34, a captured image acquisition unit 35, and a display control unit 36. Each of these functional units is realized, for example, using a computer including a central processing unit (CPU) and memory, and software. Each functional unit may also be realized using an electronic circuit, if necessary. Furthermore, each functional unit does not have to be included in a single device, and the display control device 30 may be configured from multiple devices.

[0041] The voice acquisition unit 31 acquires the voice picked up by the microphone 10 for each speaker S using the microphone 10. The voice acquisition unit 31 acquires first voice information indicating the voice of the first speaker S1 from the first microphone 10-1, and acquires second voice information indicating the voice of the second speaker S2 from the second microphone 10-2.

[0042] The voice processing unit 32 acquires voice information of the speaker S from the voice acquisition unit 31. The voice processing unit 32 performs processing such as voice recognition on the acquired voice information to generate sentences indicating the speech content of the speaker S. The voice processing unit 32 generates sentences indicating the speech content in a first language used by the first speaker S1 based on the first voice information, and generates sentences indicating the speech content in a second language used by the second speaker S2 based on the second voice information. Note that the voice processing unit 32 can prevent the generation of sentences in a language unintended by the speaker S by setting the language used by the speaker S in advance. The voice processing unit 32 may, for example, output the voice information to a machine learning model such as LLM (Large Language Models) and acquire sentences speech-recognized by the LLM.

[0043] The speech content acquisition unit 33 acquires from the voice processing unit 32 sentences indicating the speech content of each of two or more speakers S who speak different languages.

[0044] The translation unit 34 translates the sentence indicating the utterance content acquired by the utterance content acquisition unit 33 into the language used by the other speaker S. Specifically, the translation unit 34 acquires the utterance content of the first speaker S1 and translates it into the language used by each speaker S other than the first speaker S1 (for example, the second language used by the second speaker S2). The translation unit 34 also acquires the utterance content of the second speaker S2 and translates it into the language used by each speaker S other than the second speaker S2 (for example, the first language used by the first speaker S1). The translation unit 34 may, for example, output the sentence before translation to the LLM and acquire the sentence translated by the LLM.

[0045] The captured image acquisition unit 35 acquires a captured image captured by the camera 20. The captured image may be a still image or a video (image). Note that a video is preferable in order to visually recognize changes in the other person's facial expression and body language.

[0046] The display control unit 36 ​​acquires sentences indicating the utterance content of each speaker S in the language used by the speaker S who made the utterance from the utterance content acquisition unit 33. The display control unit 36 ​​also acquires sentences obtained by translating the utterance content of each speaker S into a language used by a speaker S other than that of the speaker S who made the utterance from the translation unit 34. That is, the display control unit 36 ​​acquires sentences indicating the utterance content of each speaker S in the language used by that speaker S. Furthermore, the display control unit 36 ​​acquires captured images from the captured image acquisition unit 35. The display control unit 36 ​​controls the display device 40 to display captured images of each speaker S and an utterance display group D10 for each speaker S.

[0047] [Display control of speech content] The display control unit 36 ​​displays an utterance display group D10 for each speaker S. That is, the display control unit 36 ​​displays a plurality of utterance display groups D10 having similar displayed content but different display languages. In addition, the utterance display group D10 displays the content uttered by each speaker S in order of the time of utterance.

[0048] The display control unit 36 ​​may synchronize the display positions of the own utterance display section D111 and the other speaker utterance display section D112 displayed in the utterance display group D10 for each utterance display group D10. Fig. 5 is a diagram for explaining an example of the second display screen D2 displayed on the display device 40 according to the embodiment. An example of synchronizing the display positions will be described with reference to Fig. 5.

[0049] The utterance display group D10 has a plurality of display areas D20. The display area D20 is an area set for each of the user's own utterance display area D111 and the other speaker's utterance display area D112. The size of the display area D20 may be, for example, according to the amount of text in the utterance content of the user's own utterance display area D111 or the other speaker's utterance display area D112. Furthermore, when the utterance content is the same in each utterance display group D10, the position within the utterance display group D10, the proportion and shape of the display area D20, etc. are the same in each utterance display group D10. In other words, the size of the display area D20 does not change depending on the language used for display. FIG. 5 shows an example in which the utterance display group D10 has three display areas D20: a first utterance content display area D21, a second utterance content display area D22, and a third utterance content display area D23.

[0050] The own speech display section D111 and the other speaker's speech display section D112 are displayed at predetermined positions within the display area D20. The predetermined positions are positions according to a predetermined standard. The predetermined standard determines the top and bottom positions within the utterance display group D10, and may be, for example, top-aligned, center-aligned, bottom-aligned, etc. The predetermined position when top-aligned is used as the standard may be, for example, a position where the upper edge of the display area D20 meets the upper edge of the own speech display section D111 or the other speaker's utterance display section D112.

[0051] 5 shows an example in which the first utterance display group D11 and the second utterance display group D12 are arranged in the horizontal direction (y direction). However, the present embodiment is not limited to this example, and they may be arranged in the vertical direction (z direction), for example.

[0052] [Identification display control] The display control unit 36 ​​displays an identification indicator D113 that serves as an indicator for distinguishing between the own speech display unit D111 and the other speaker's speech display unit D112. The identification indicator D113 may be attached only to the own speech display unit D111, or only to the other speaker's speech display unit D112, or may be attached to both the own speech display unit D111 and the other speaker's utterance display unit D112. The identification indicator D113 is, for example, an indicator indicating that it is the own speech display unit D111 or the other speaker's utterance display unit D112, and may be a symbol, graphic, or the like, such as a letter or a number that is not confused with the text displayed on the own speech display unit D111 and the other speaker's utterance display unit D112. Furthermore, the identification indicator D113 may be an indicator that makes the outline shape, color, pattern, or the like different between the own speech display unit D111 and the other speaker's utterance display unit D112. The identification display D113 is, for example, a display for identifying each speaker S who has spoken, and may be the name of the speaker S, an icon such as a symbol or figure, an illustration, an image, or the like.

[0053] The identification display D113 according to this embodiment is a red circle added to the left side of the self-utterance display section D111. When the identification display D113 is simple and has a small area like this, the speaker image SI is less likely to be obstructed by the identification display D113, and the speaker S can accurately recognize the face and movements of the speaker image SI.

[0054] [Display control of captured images] For example, the display control unit 36 ​​may superimpose an utterance display group D10, in which utterance content is displayed in the language used by the speaker S, at a position where the speaker S is captured in the captured image. Specifically, the display control unit 36 ​​superimposes a first utterance display group D11, which is displayed in a first language spoken by a first speaker S1, on an area where the first speaker S1 is displayed in the captured image (sometimes referred to as a first speaker image SI1). Furthermore, the display control unit 36 ​​superimposes a second utterance display group D12, which is displayed in a second language spoken by a second speaker S2, on an area where the second speaker S2 is displayed in the captured image (sometimes referred to as a second speaker image SI2). When the speaker image SI and the utterance display group D10 are superimposed and displayed, the utterance display group D10 is displayed semi-transparently.

[0055] When there are two speakers S, the display control unit 36 ​​may display the first utterance display group D11 superimposed on the second speaker image SI2, and the second utterance display group D12 superimposed on the first speaker image SI1. This allows the speaker S to check the content of the utterance while visually recognizing the speaker image SI of the other speaker S.

[0056] Further, the display control unit 36 ​​may, for example, display the utterance display group D10 near the speaker image SI of the speaker S to be displayed in the utterance display group D10. Nearby means, for example, a position in each utterance display group D10 that is closest to (e.g., adjacent to) the speaker image SI of the speaker S to be displayed. FIG. 6 is a diagram for explaining an example of a third display screen D3 displayed on the display device 40 according to the embodiment. For example, the display control unit 36 ​​may display the first utterance display group D11 at a position where the distance from the first utterance display group D11 to the first speaker image SI1 is shorter than the distance from the second utterance display group D12 to the first speaker image SI1. Further, the display control unit 36 ​​may display the other speaker's utterance display unit D112 at a position where the distance from the other speaker's utterance display unit D112 to the second speaker image SI2 is shorter than the distance from the own utterance display unit D111 to the second speaker image SI2.

[0057] The display control unit 36 ​​may determine the position at which the utterance display group D10 is displayed depending on the position of the speaker image SI, or may display the utterance display group D10 at a predetermined position. When the utterance display group D10 is displayed at a predetermined position, the position at which the speaker image SI is displayed needs to be adjusted depending on the display position of the utterance display group D10. For example, in order to guide the speaker image SI to be displayed at a predetermined position, a microphone 10 may be provided for each speaker S, and the microphone 10 may be installed at a position to which the speaker S is to be guided. Specifically, the first microphone 10-1 and the second microphone 10-2 are arranged side by side in a predetermined direction (e.g., the y direction). The display control unit 36 ​​displays the utterance display group D10 for each speaker S in the order in which the microphones 10 are arranged in the direction in which the microphones 10 are arranged (the y direction). This allows the first utterance display group D11 to be displayed at the position of the first speaker image SI1, and the second utterance display group D12 to be displayed at the position of the second speaker image SI2.

[0058] 7 is a flowchart illustrating an example of a processing flow of the display control device 30 according to the embodiment. The display control device 30 acquires audio information indicating audio picked up by each microphone 10 provided for each speaker S (step S101). The display control device 30 performs speech recognition on the acquired audio information to generate a sentence indicating the content of the speech by each speaker S (step S102). The display control device 30 acquires the sentence indicating the content of the speech (step S103). The display control device 30 translates the acquired sentence into the language used by each speaker S (step S104). The display control device 30 acquires a captured image (e.g., video) of each speaker S from the camera 20 (step S105). The display control device 30 controls the display device 40 to display, for each speaker S, an utterance display group D10 in chronological order of the utterance, which displays a self-utterance display section D111 in which the speaker's own utterance content is displayed in the speaker's own language and a different speaker's utterance display section D112 in which the other speaker S's utterance content is translated into the speaker's own language and displayed (step S106).

[0059] In the above description, an example is shown in which the simultaneous display system 1 has the same number of microphones 10 as the number of speakers S. However, the present embodiment is not limited to this example, and the simultaneous display system 1 may have only one microphone 10. In this case, the audio processing unit 32 identifies the speaker S who spoke based on audio information from multiple speakers S through one microphone 10 and separates the audio information.

[0060] Furthermore, in the above description, an example has been shown in which the simultaneous display system 1 includes one camera 20. However, the present embodiment is not limited to this example, and the simultaneous display system 1 may include a plurality of cameras 20, for example, the same number as the number of speakers S. In this case, the display control device 30 may associate the video acquired from the camera 20 capturing the image of the speaker S with an utterance display group D10 based on information acquired from the microphone 10 collecting the voice of the speaker S, and display the images on the display device 40.

[0061] In the above description, an example has been shown in which the simultaneous display system 1 is used for communication between a first speaker S1 and a second speaker S2 who are in the same room. However, the present embodiment is not limited to this example, and the simultaneous display system 1 may also be used for communication between speakers S who are in remote locations. For example, the simultaneous display system 1 may be used for online interviews or meetings. In this case, the microphone 10, the camera 20, and the display device 40 may be provided for each speaker S. Furthermore, the speaker S may have an information processing device such as a smartphone, tablet, or personal computer that has the functions of the microphone 10, the camera 20, and the display device 40.

[0062] Furthermore, in the above description, an example has been shown in which the simultaneous display system 1 is used for a conversation between two or more speakers S. However, the present embodiment is not limited to this example, and the simultaneous display system 1 may be used for a conversation between three or more speakers S, i.e., a conversation in three or more languages.

[0063] In the above description, an example has been shown in which the simultaneous display system 1 is used in a conversation between only speakers S who speak different languages. However, this embodiment is not limited to this example, and speakers S who speak the same language may also be included. For example, the simultaneous display system 1 may be used in a conversation between a first speaker S1 who speaks Japanese, a second speaker S2 who speaks Japanese, and a third speaker S3 who speaks English.

[0064] Furthermore, in the above description, an example has been shown in which the simultaneous display system 1 is used for a conversation between speakers S via their voices. However, this embodiment is not limited to this example and may be used for conversations via methods other than voice. That is, speech may be either audible or non-audio. For example, speaker S's speech may be via a non-audio method, such as text input or sign language. When speech is via text input, the simultaneous display system 1 may further include a terminal device such as a smartphone, tablet, or PC, and an input device such as a keyboard, mouse, controller, or pen tablet connected to the terminal device. By using the simultaneous display system 1 according to the embodiment for speech via text input, speaker S can accurately convey to other speakers S words that may be misrecognized or mistranslated, such as technical terms. Note that speaker S may use both speech and non-audio speech. That is, speaker S usually speaks via voice, but may use text input when misrecognition or mistranslation occurs or is likely to occur.

[0065] In the above description, an example is shown in which the display control unit 36 ​​displays the utterance display group D10 and the captured image. However, the present embodiment is not limited to this example, and the utterance display group D10 may be displayed without displaying the captured image.

[0066] [Summary of the embodiment] According to the above-described embodiment, the display control device 30 includes a speech content acquisition unit 33 that acquires the speech content of each of two or more speakers S who speak different languages, and a display control unit 36 ​​that displays, for each speaker S, an utterance display group D10 including a self-utterance display unit D111 in which the speech content of the speaker S is displayed in the speaker's own language as a sentence, and an other-speaker utterance display unit D112 in which the speech content of the other speakers S is displayed in the speaker's own language as a sentence translated into the speaker's own language. According to the display control device 30 of the embodiment, the speaker S can understand how the speech content of the speaker S was recognized and correct any errors. Furthermore, the speaker S can understand the translation result of the speech content of the speaker S and confirm whether it has been translated accurately. If the translation result is inappropriate, the speaker S may speak again while referring to the translation result to avoid incorrect translation. Furthermore, by viewing the utterance display group D10 displayed in the speaker S's own language, the speaker S can understand past exchanges. In other words, the information can be used for minutes of meetings.

[0067] Furthermore, the technology of Patent Document 1, which uses a transparent display for communication, displays the translated content on the back side of the display, making it difficult to use for communication in three or more languages. The display control device 30 of this embodiment can easily support three or more languages ​​by increasing the number of utterance display groups D10 according to the speaker S. Furthermore, the display control device 30 of this embodiment displays the translated text without flipping it horizontally, making it easy to determine whether there are any translation errors. Furthermore, the simultaneous display system 1 of this embodiment can realize the above technology using an information processing device such as a general display, smartphone, or tablet, making it easier and less expensive to implement than the technology of Patent Document 1.

[0068] Furthermore, according to the above-described embodiment, the utterance display group D10 has an area (display area D20) that is positioned similarly in each utterance display group D10, and the own utterance display area D111 and the other speaker utterance display area D112 are displayed at predetermined positions in the display area D20. By aligning the display positions of the own utterance display area D111 and the other speaker utterance display area D112 in each utterance display group D10, the speaker S can easily recognize sentences that show the same utterance content among the sentences displayed in the speaker S's own language and the sentences displayed in the language used by the other speaker S.

[0069] Furthermore, according to the above-described embodiment, at least one of the own speech display section D111 or the other speaker's speech display section D112 has a display (identification display D113) that identifies whether it is the own speech display section D111 or the other speaker's speech display section D112. By displaying the identification display D113, it is possible to easily distinguish between what has been spoken by the speaker S himself and what has been spoken by another speaker S. Therefore, when the speaker S reviews the exchange, he or she can easily determine which speaker S has spoken what.

[0070] Moreover, according to the above-described embodiment, the display control device 30 further includes a captured image acquisition unit 35 that acquires captured images of the speaker S, and the display control unit 36 ​​displays the utterance display group D10 for the speaker S by superimposing it on the captured image of the speaker S. Moreover, the display control device 30 further includes a captured image acquisition unit 35 that acquires captured images of the speaker S, and the display control unit 36 ​​may display the captured image of the speaker S near the utterance display group D10 for the speaker S in each utterance display group D10. By displaying the utterance content of each speaker S and the captured image of each speaker S by superimposing them on each other or displaying them near each other, the speaker S can easily see the facial expressions and body movements of the other speakers S while reading the translated utterance content. By visually observing facial expressions and body movements in addition to the text indicating the content of the speech, speaker S can easily read the reactions and nuances of other speakers S, enabling smooth communication with other speakers S.

[0071] Furthermore, according to the above-described embodiment, the speech content acquisition unit 33 acquires the speech content of each speaker S based on the voice picked up by the microphone 10 provided for each speaker S, and the display control unit 36 ​​displays the speech display group D10 for each speaker S using the microphone 10. In this way, the display control device 30 performs voice recognition for each microphone 10 and can identify the speech content with high accuracy. Therefore, translation errors can be reduced.

[0072] Furthermore, according to the above-described embodiment, the microphones 10 are arranged in a predetermined direction, and the display control unit 36 ​​displays the utterance display group D10 for each speaker S in a predetermined direction in the order of the microphones 10 used by each speaker S. This allows the speaker S to be guided to a desired position, and the positions of the speaker image SI and the utterance display group D10 to be easily adjusted.

[0073] FIG. 8 is a block diagram showing an example of the internal configuration of the display control device 30 according to the embodiment. At least some of the functions of the display control device 30 can be implemented using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be implemented using existing technology. The central processing unit 901 executes instructions contained in a program read from the RAM 902 or the like. In accordance with each instruction, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with external input / output devices. The input / output devices 904 and 905 are input / output devices. The input / output devices 904 and 905 exchange data with the central processing unit 901 via the input / output port 903. The bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from and to the RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port via the bus 906. All or part of the functional units provided in the display control device 30 may be realized using hardware such as an ASIC, a PLD, or an FPGA. All or part of the functional units may be realized by a combination of software and hardware.

[0074] Note that all or part of the functions of each unit of the display control device 30 in the above-described embodiment may be realized by recording a program for realizing these functions on a computer-readable recording medium, and reading and executing the program recorded on the recording medium into a computer system. Note that the term "computer system" here includes hardware such as an OS and peripheral devices.

[0075] Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage units such as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs over networks like the Internet or communication lines like telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within computer systems that serve as servers or clients in such cases. Furthermore, the above-mentioned programs may be programs that realize some of the aforementioned functions, or may be programs that can realize the aforementioned functions in combination with programs already stored in the computer system.

[0076] Although one embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design modifications can be made without departing from the spirit of the present invention. Furthermore, the configurations described in the above-described embodiments and examples can be combined. [Explanation of symbols]

[0077] S...speaker, SI...speaker image, 1...simultaneous display system, 10...microphone, 20...camera, 31...voice acquisition unit, 32...voice processing unit..., 33...speech content acquisition unit..., 34...translation unit, 35...captured image acquisition unit..., 36...display control unit, 30...display control device, 40...display device, D10...speech display group, D111...own speech display unit, D112...other speaker's speech display unit, D113...identification display

Claims

1. a speech content acquisition unit that acquires speech content from two or more speakers who speak different languages; a display control unit that displays, for each of the speakers, a speech display group including a self-speech display unit in which the content of the speaker's speech is displayed in a sentence in the speaker's language, and a different speaker's speech display unit in which the content of the other speakers' speech is displayed in a sentence translated into the speaker's language; A display control device comprising:

2. The utterance display group has regions whose positions within the utterance display group are similar in each of the utterance display groups, the user's own speech display unit and the other speaker's speech display unit are displayed at predetermined positions within the area. The display control device according to claim 1 .

3. At least one of the self-utterance display unit and the other speaker's utterance display unit is displayed to identify whether it is the self-utterance display unit or the other speaker's utterance display unit. The display control device according to claim 1 .

4. further comprising a captured image acquisition unit that acquires a captured image of the speaker; the display control unit displays the utterance display group for the speaker in a superimposed manner on a captured image of the speaker. The display control device according to claim 1 .

5. further comprising a captured image acquisition unit that acquires a captured image of the speaker; the display control unit displays a captured image of a speaker near the utterance display group for the speaker among the utterance display groups. The display control device according to claim 1 .

6. the speech content acquisition unit acquires speech content for each speaker based on a voice picked up by a microphone provided for each speaker; the display control unit displays the speech display group for each speaker who uses the microphone. The display control device according to claim 4 or 5.

7. The microphones are arranged in a row in a predetermined direction, the display control unit displays the utterance display group for each speaker in the direction in the order of the microphones used by each speaker. The display control device according to claim 6 .

8. Different languages ​​include dialects, The display control device according to claim 4 or 5.

9. the speech contents acquired by the speech content acquisition unit are in different languages, i.e., three or more languages; The display control device according to claim 4 or 5.

10. On the computer, a speech content acquisition step of acquiring speech content of each of two or more speakers who speak different languages; a display control step of displaying, for each of the speakers, a speech display group including a self-speech display section in which the content of the speaker's speech is displayed in a sentence in the speaker's language, and an other speaker's speech display section in which the content of the other speakers' speech is displayed in a sentence translated into the speaker's language; A program that executes the following.

Images