Conversation assistance device, conversation assistance method, and program

Abstract

A conversation assistance device, a conversation assistance method, and a program are provided that are capable of realizing two-way communication with a person who has difficulty speaking. [Solution] A conversation assistance device comprising: a display processing unit that displays a first operation area operated by the first user and a second operation area operated by the second user on a first operation terminal shared by a first user on a first main surface side where a first main surface of a transparent display is located, and a second user on a second main surface side where a second main surface is located, which is the main surface opposite to the first main surface; and an input selection processing unit that accepts, in the first operation area, a selection of an input means, either voice input or keyboard input, and a selection of a language to be used with the selected input means, wherein, when keyboard input is selected as the input means, the display processing unit displays, in the first operation area where the selection was accepted, a digital keyboard capable of input using the selected language.

Description

[Technical field]

[0001] The present invention relates to a conversation assistance device, a conversation assistance method, and a program. [Background technology]

[0002] 2. Description of the Related Art Conventionally, various techniques have been proposed for translating and outputting speech inputted by voice as a means for enabling communication between people who speak different languages.

[0003] For example, Patent Document 1 listed below discloses a technology in which, in face-to-face conversations such as at a local government office, a company reception desk, a restaurant, or a cash register, when one speaker speaks into a microphone provided in a conversation assistance device placed between the two parties, the content of the speech is translated into the other person's language and displayed on a display provided in the conversation assistance device. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2024-40187 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, while the technology described in Patent Document 1 enables people who speak different languages ​​to speak and communicate with each other, it makes it difficult to communicate with people who have difficulty speaking, such as those who are hard of hearing or have speech disabilities, because the only input method available is voice input.

[0006] In view of the above-mentioned problems, an object of the present invention is to provide a conversation assistance device, a conversation assistance method, and a program that are capable of realizing two-way communication with a person who has difficulty speaking. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, a conversation assistance device according to one embodiment of the present invention includes a display processing unit that displays a first operation area operated by the first user and a second operation area operated by the second user on a first operation terminal shared by a first user on a first main surface side where a first main surface of a transparent display is located and a second user on a second main surface side where a second main surface is located, the second main surface being the main surface opposite to the first main surface, and an input selection processing unit that accepts, in the first operation area, a selection of an input means, either voice input or keyboard input, and a selection of a language to be used for the selected input means; an input receiving unit that receives an input of a first conversation content input by the first user through the voice input or the keyboard input and an input of a second conversation content input by the second user through the voice input; and a multilingual translation processing unit that obtains a first translation result in which the first conversation content is translated from a language selected in the first operation area into a language used by the second user, and a second translation result in which the second conversation content is translated from the language used by the second user into the language selected in the first operation area; and when the keyboard input is selected as the input means, the display processing unit displays a digital keyboard capable of inputting using the selected language in the first operation area that accepts the selection. death , the display processing unit displays first character information indicating the first translation result and second character information indicating the second translation result on the transparent display, and when the second user selects a content in the second operation area during a conversation with the first user, the display processing unit switches a display of the conversation content displayed on the transparent display to a display of the selected content. It is a conversation aid. A conversation assistance device according to one aspect of the present invention includes a display processing unit that displays a first operation area operated by the first user and a second operation area operated by the second user on a first operation terminal shared by a first user on a first main surface side where a first main surface of a transparent display is located and a second user on a second main surface side where a second main surface is located, the second main surface being a main surface opposite to the first main surface; an input selection processing unit that accepts, in the first operation area, a selection of an input means, either voice input or keyboard input, and a selection of a language to be used in the selected input means; an input acceptance unit that accepts an input of a first conversation content input by the first user by the voice input or the keyboard input, and an input of a second conversation content input by the second user by the voice input; a first translation result in which the first conversation content is translated from the language selected in the first operation area into the language used by the second user, and a translation result in which the second conversation content is translated from the language used by the second user into the first operation area. and a multilingual translation processing unit that acquires a second translation result translated into a selected language, when the keyboard input is selected as the input means, and the display processing unit displays a digital keyboard capable of input using the selected language in the first operation area that accepts the selection, and the display processing unit displays first character information indicating the first translation result and second character information indicating the second translation result on the transparent display, when the keyboard input is selected as the input means, the display processing unit displays code information in the first operation area including connection information for accepting input of the first conversation content from a second operation terminal owned by the first user, and the multilingual translation processing unit acquires a first translation result in which the first conversation content input to the second operation terminal connected using the connection information read from the code information is translated from the language selected in the first operation area into the language used by the second user.

[0008] A conversation assistance method according to one aspect of the present invention includes a display process for displaying a first operation area operated by the first user and a second operation area operated by the second user on a first operation terminal shared by a first user on a first main surface side where a first main surface of a transparent display is located and a second user on a second main surface side where a second main surface is located, the second main surface being the main surface opposite to the first main surface; and an input selection process for accepting, in the first operation area, a selection of an input means, either voice input or keyboard input, and a selection of a language to be used for the selected input means; an input receiving step of receiving an input of a first conversation content input by the first user through the voice input or the keyboard input and an input of a second conversation content input by the second user through the voice input; and a multilingual translation processing step of acquiring a first translation result in which the first conversation content is translated from a language selected in the first operation area into a language used by the second user, and a second translation result in which the second conversation content is translated from the language used by the second user into the language selected in the first operation area; and when the keyboard input is selected as the input means, the display processing step displays a digital keyboard capable of inputting using the selected language in the first operation area that accepts the selection. death , The display process displays first character information indicating the first translation result and second character information indicating the second translation result on the transparent display, and when the second user selects a content in the second operation area during a conversation with the first user, the display process switches a display of the conversation content displayed on the transparent display to a display of the selected content. A computer-implemented method for speech assistance. Furthermore, a conversation assistance method according to one aspect of the present invention includes a display process for displaying a first operation area operated by the first user and a second operation area operated by the second user on a first operation terminal shared by a first user on a first main surface side where a first main surface of a transparent display is located and a second user on a second main surface side where a second main surface is located, the second main surface being a main surface opposite to the first main surface; an input selection process for accepting, in the first operation area, a selection of an input means, either voice input or keyboard input, and a selection of a language to be used in the selected input means; an input acceptance process for accepting an input of a first conversation content input by the first user by the voice input or the keyboard input, and an input of a second conversation content input by the second user by the voice input; a first translation result in which the first conversation content is translated from the language selected in the first operation area into the language used by the second user, and a translation process for translating the second conversation content from the language used by the second user into the language selected in the first operation area. and a multilingual translation processing process for acquiring a second translation result obtained by translating the first conversation content input to the second operation terminal using the connection information read from the code information, wherein the display processing process, when the keyboard input is selected as the input means, displays a digital keyboard capable of input using the selected language in the first operation area that accepted the selection, and the display processing process displays first character information indicating the first translation result and second character information indicating the second translation result on the transparent display, and when the keyboard input is selected as the input means, the display processing process displays code information in the first operation area including connection information for accepting input of the first conversation content from a second operation terminal owned by the first user, and the multilingual translation processing process acquires a first translation result in which the first conversation content input to the second operation terminal connected using the connection information read from the code information is translated from the language selected in the first operation area to the language used by the second user.

[0009] A program according to one aspect of the present invention includes a computer, a display processing means for displaying a first operation area operated by the first user and a second operation area operated by the second user on a first operation terminal shared by a first user on a first main surface side where a first main surface of a transparent display is located and a second user on a second main surface side where a second main surface is located, the second main surface being the main surface opposite to the first main surface, and an input selection processing means for accepting, in the first operation area, a selection of an input means, either voice input or keyboard input, and a selection of a language to be used for the selected input means; an input receiving means for receiving an input of a first conversation content input by the first user through the voice input or the keyboard input and an input of a second conversation content input by the second user through the voice input; and a multilingual translation processing means for acquiring a first translation result in which the first conversation content is translated from the language selected in the first operation area into a language used by the second user, and a second translation result in which the second conversation content is translated from the language used by the second user into the language selected in the first operation area; When the keyboard input is selected as the input means, the display processing means displays a digital keyboard capable of inputting using the selected language in the first operation area that accepts the selection. death , the display processing means displays first character information indicating the first translation result and second character information indicating the second translation result on the transparent display, and when the second user selects a content in the second operation area during a conversation with the first user, the display processing means switches a display of the conversation content displayed on the transparent display to a display of the selected content. It is a program. Furthermore, a program according to one aspect of the present invention provides a computer program comprising: a display processing means for displaying, on a first operation terminal shared by a first user on a first principal surface side where a first principal surface of a transparent display is located, and a second user on a second principal surface side where a second principal surface is located, which is a principal surface opposite to the first principal surface, a first operation area operated by the first user and a second operation area operated by the second user; an input selection processing means for accepting, in the first operation area, a selection of an input means, either voice input or keyboard input, and a selection of a language to be used in the selected input means; an input accepting means for accepting an input of a first conversation content input by the first user by the voice input or the keyboard input, and an input of a second conversation content input by the second user by the voice input; and a second translation result in which the first conversation content is translated into a language selected in the first operation area and a second translation result in which the first conversation content is translated into a language selected in the first operation area, wherein, when the keyboard input is selected as the input means, the display processing means displays a digital keyboard capable of input using the selected language in the first operation area that accepted the selection, and the display processing means displays first character information indicating the first translation result and second character information indicating the second translation result on the transparent display, and when the keyboard input is selected as the input means, the display processing means displays code information in the first operation area including connection information for accepting input of the first conversation content from a second operation terminal owned by the first user, and the multilingual translation processing means acquires a first translation result in which the first conversation content input to the second operation terminal connected using the connection information read from the code information is translated from the language selected in the first operation area into the language used by the second user. Effect of the Invention

[0010] According to the present invention, two-way communication with a person who has difficulty speaking can be realized. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an overview of how a conversation display system according to a first embodiment is used. [Diagram 2] 1 is a diagram showing an example of a configuration of a conversation display system according to a first embodiment. [Diagram 3] 2 is a block diagram showing an example of a functional configuration of a tablet terminal according to the first embodiment. FIG. [Figure 4] 1 is a block diagram showing an example of a functional configuration of a multilingual translation device according to a first embodiment. [Diagram 5] FIG. 4 is a diagram showing an example of an input selection screen according to the first embodiment. [Figure 6] FIG. 4 is a diagram showing an example of a voice input screen according to the first embodiment. [Figure 7] FIG. 4 is a diagram showing an example of a keyboard input screen according to the first embodiment. [Figure 8] FIG. 2 is a diagram showing an example of a conversation screen according to the first embodiment. [Figure 9] 5A to 5C are diagrams illustrating an example of a display change on an input screen according to the first embodiment. [Figure 10] FIG. 2 is a sequence diagram showing an example of a processing flow in the conversation display system according to the first embodiment. [Figure 11] 11 is a flowchart showing an example of the flow of an input selection process according to the first embodiment. [Figure 12] FIG. 11 is a block diagram showing an example of the configuration of a conversation display system according to a second embodiment. [Figure 13] FIG. 11 is a diagram illustrating an example of a functional configuration of a user terminal according to the second embodiment. [Figure 14] FIG. 11 is a diagram showing an example of a user terminal connection screen according to the second embodiment. [Figure 15] FIG. 11 is a diagram showing an example of a keyboard input screen according to the second embodiment. [Figure 16] FIG. 11 is a sequence diagram showing an example of a processing flow in a conversation display system according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0013] <<1. First embodiment>> The first embodiment will be described with reference to FIGS.

[0014] <1-1. Overview of use of the conversation display system> An overview of how the conversation display system according to the first embodiment is used will be described with reference to Fig. 1. Fig. 1 is a diagram illustrating an overview of how the conversation display system according to the first embodiment is used. The conversation display system in the first embodiment has a function as a partition for preventing droplets and crime, and also has a function of visualizing the conversation as characters in real time by instantly converting input conversation contents into characters and displaying them on the partition. Since the partition is transparent, users in the conversation can also see the facial expressions of the conversation partner. Therefore, the conversation display system has a mechanism that allows users to communicate smartly.

[0015] The conversation display system uses a conversation assistance device 10 shown in Fig. 1. The conversation display system is used, for example, in counter services at local governments, stores, stations, and the like. In this case, the conversation assistance device 10 is provided between a first user (e.g., a resident U1) and a second user (e.g., an operator U2) as shown in Fig. 1. In the following, a first embodiment will be described taking as an example a case in which the first user is a resident U1 and the second user is an operator U2.

[0016] 1, in conversation assistance device 10, a sound collection device 20, a transparent display 50, and a tablet terminal 60 are mounted on a display base 11. Display base 11 has a mounting portion 12, legs 13, a storage portion 14, and a tray portion 15.

[0017] The placement unit 12 is capable of placing one side of the transparent display 50 along the horizontal direction of the main surface thereof, and is set to a length corresponding to the side. In the example shown in Fig. 1, the first main surface side of the transparent display 50 faces the resident U1, and the second main surface side of the transparent display 50, which is the main surface opposite to the first main surface thereof, faces the operator U2. The transparent display 50 may be placed on the placement unit 12 so as to be placed horizontally or vertically.

[0018] The legs 13 are provided at least on a first end, which is one end of the mounting portion 12, and support the mounting portion 12. Note that the legs 13 may be provided not only on the first end, but also on a second end, which is the end opposite to the first end.

[0019] The storage section 14 is provided at a second end, which is an end opposite to the end (first end) at which the legs 13 are provided in the mounting section 12, and can store an electronic device. The electronic device stored inside the storage section 14 is, for example, a sound collector 20, a device used for converting voice, and a hub or cable for connecting each device. As shown in FIG. 1, the sound collector 21 of the sound collector 20 is protruded outside the storage section 14 to collect the voice of a speaking user. When the legs 13 are provided only at either the first end or the second end of the mounting section 12, the storage section 14 is provided so as to support the mounting section 12 at the end side of the mounting section 12 where the legs 13 are not provided. On the other hand, when the legs 13 are provided at both the first end and the second end of the mounting section 12, the storage section 14 may be provided so as not to support the mounting section 12 by the legs 13 supporting both ends of the mounting section 12.

[0020] The tray section 15 is set to a length according to the longitudinal direction of the placement section 12, and the legs 13 are connected to a third end, which is one end of the placement section 15, and the storage section 14 is connected to a fourth end, which is the end opposite to the third end. When the legs 13 are also provided at the second end, the legs 13 at the second end are also connected to the fourth end together with the storage section 14. As shown in FIG. 1, in the longitudinal direction of the placement section 12, the legs 13 and the storage section 14 are spaced apart, and a spaced apart space is formed between the legs 13 and the storage section 14. The space is also a space that penetrates in the direction between the first main surface side (resident U1 side) and the second main surface side (operator U2 side) of the transparent display 50 (the lateral direction of the placement section 12).

[0021] Further, the tray section 15 is formed with a placement surface that is visible from the first main surface side and the second main surface side of the transparent display 50 and on which an object can be placed. The placement surface is formed in the tray section 15 at a portion corresponding to the space between the leg section 13 and the storage section 14. For example, a first operation terminal shared by the resident U1 and the operator U2 is placed on the placement surface. For this reason, the size of the placement surface may be set according to the size of the first operation terminal placed on the placement surface. For example, a tablet terminal 60 is placed on the placement surface as shown in FIG. 1. In this case, on the placement surface, the length of the tray section 15 along the thickness direction of the transparent display 50 is set to be longer than the length of the tablet terminal 60 in the short side direction, and the length of the tray section 15 along the horizontal direction of the main surface of the transparent display 50 is set to be longer than the length of the tablet terminal 60 in the long side direction.

[0022] Also, a recess may be formed in the center of the placement surface. The size of the recess may be set to be the same as the size of the placement surface. By providing the recess on the placement surface, it is possible to prevent the first operation terminal placed on the recess from shifting when the resident U1 or the operator U2 operates the first operation terminal placed on the recess. Also, since the recess on the placement surface prevents the first operation terminal from shifting, it is possible to prevent the first operation terminal from dropping when the conversation assistance device 10 is transported.

[0023] A gripping portion 151 may be formed at an end in the longitudinal direction of the tray portion 15 so as to be recessed so as to be grippable when transporting the conversation assistance device 10 (display base 11). The gripping portion 151 may be formed at the end of the tray portion 15 by processing the end of the tray portion 15 itself to make it recessed, or a gripping portion may be formed by connecting a component to the bottom surface of the tray portion 15 and making the end of the tray portion 15 including the component into a recessed shape. Furthermore, a material (for example, rubber) may be provided on the rear surface of the tray unit 15 (the surface that comes into contact with the installation location) to prevent it from falling over.

[0024] As described above, in the conversation assistance device 10, the transparent display 50 can be installed vertically on the placement part 12 of the display base 11. As a result, the transparent display 50 is installed so that the position of the user's face is located within the screen of the transparent display 50, and is provided to separate the resident U1 and the operator U2. Since the transparent display 50 is installed between the resident U1 and the operator U2 in this way, it is possible to prevent droplets and the like from directly scattering to the other party. The display base 11 may have an adjustment mechanism capable of changing the distance between the placement part 12 and the tray part 15. The adjustment mechanism may be capable of adjusting the height of the leg part 13 itself, may be capable of adjusting the height of the storage part 14 itself, or may be capable of adjusting the fixed position of the placement part 12. The adjustment mechanism allows the position of the transparent display 50 to be adjusted so that the position of the user's face is located within the screen of the transparent display 50 according to differences in the height, sitting height, and the like of the user.

[0025] In addition, in the conversation assistance device 10, a space formed between the leg 13 and the storage unit 14 passes between the resident U1 side and the operator U2 side. This allows the resident U1 and the operator U2 to pass and receive documents, slips, and other documents in the space. Furthermore, the height of the space is the same as the height of the storage unit 14, and can also be adjusted by an adjustment mechanism. This allows the resident U1 and the operator U2 to easily operate the tablet terminal 60 and pass and receive documents, etc., since the height of the space is sufficiently secured.

[0026] In addition, the display base 11 of the conversation assistance device 10 is a tray-integrated base by connecting the placement portion 12, the legs 13, the storage portion 14, and the tray portion 15. This allows the user to transport the conversation assistance device 10 together with the display base 11, the transparent display 50 placed on the placement portion 12, the electronic device stored in the storage portion 14, and the tablet terminal 60 placed on the tray portion 15, making transportation efficient. In the case of a base that is not a tray-integrated base, the electronic device and the tablet terminal 60 are placed directly on a table (e.g., a reception table) on which the display base 11 on which the transparent display 50 is placed is placed. In this case, the user needs to transport the display base 11 on which the transparent display 50 is placed, and the electronic device and the tablet terminal 60 separately, resulting in poor transportation efficiency.

[0027] <1-2. Structure of the conversation display system> The above describes an overview of the use of the conversation display system according to the first embodiment. Next, the configuration of the conversation display system according to the first embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the configuration of the conversation display system according to the first embodiment.

[0028] As shown in FIG. 2, the conversation display system 1 includes a conversation assistance device 10 and a multilingual translation device 80. The network NW may be configured to transmit and receive information using, for example, a Local Area Network (LAN), a Wide Area Network (WAN), a telephone network (mobile phone network, landline telephone network, etc.), a regional Internet Protocol (IP) network, the Internet, etc.

[0029] (1) Conversation assistance device 10 The conversation assistance device 10 is a device that assists a conversation between a resident U1 and an operator U2. As described with reference to Fig. 1, the conversation assistance device 10 has various electronic devices mounted on a display base 11 having a mounting section 12, legs 13, a storage section 14, and a tray section 15 connected thereto. As shown in Fig. 2, the electronic devices mounted on the display base 11 are, for example, a sound collection device 20, an audio interface 30, a USB (Universal Serial Bus) hub 40, a transparent display 50, and a tablet terminal 60.

[0030] The sound collection device 20, the audio interface 30, and the USB hub 40 are housed inside the housing portion 14 of the display base 11. The transparent display 50 is placed on the mounting portion 12 of the display base 11. The tablet terminal 60 is placed on the tray portion 15 of the display base 11.

[0031] (1-1) Sound collection device 20 The sound collector 20 is a directional microphone or an omnidirectional microphone. The sound collector 20 is installed on the display base 11 so as to be able to collect the user's voice. For example, the sound collector 20 is installed so that the main body is housed inside the housing 14 and the sound collector is located outside the housing 14. The sound collector 20 is connected by wire to the audio interface 30 inside the housing 14, and transmits the collected voice to the audio interface 30.

[0032] As shown in FIG. 2, conversation assistance device 10 according to the first embodiment includes sound collection devices 20-1 and 20-2. The sound collector 20-1 is installed so as to be able to collect the voice of the resident U1. For example, the sound collector 20-1 is placed outside the storage unit 14 so that the main body is housed inside the storage unit 14 and the sound collector is located on the first main surface side (the resident U1 side) of the transparent display 50. The sound collector 20-2 is installed so as to be able to collect the voice of the operator U2. For example, the sound collector 20-2 is placed so as to be able to collect the voice of the operator U2. For example, the main body is housed inside the storage unit 14 and the sound collector is located outside the storage unit 14 so that the main body is housed inside the storage unit 14 and the sound collector is located on the second main surface side (the operator U2 side) of the transparent display 50. The sound collection device 20-1 and the sound collection device 20-2 are each connected by wire to the audio interface 30. The sound collection device 20-1 and the sound collection device 20-2 transmit the collected sounds to the audio interface 30.

[0033] (1-2) Audio Interface 30 The audio interface 30 is a device for converting sound into data. The audio interface 30 is housed inside the housing 14, and is connected by wire to the sound collection device 20 and the USB hub 40 inside the housing 14. The audio interface 30 converts the sound received from the sound collection device 20 into data. The audio interface 30 transmits the data obtained by converting the sound (hereinafter also referred to as "audio data") to the tablet terminal 60 via the USB hub 40.

[0034] (1-3) USB Hub 40 The USB hub 40 is a device for connecting electronic devices mounted on the display base 11. As shown in Fig. 2, the audio interface 30, the transparent display 50, the tablet terminal 60, and the external power supply 70 are wired to the USB hub 40 using cables according to the standards of each device. The USB hub 40 is connected to an external power source 70 via a power cable, and supplies power from the external power source 70 to other electronic devices. The power is supplied to the sound collection device 20, the audio interface 30, and the tablet terminal 60 via the USB hub 40.

[0035] (1-4) Transparent display 50 The transparent display 50 is a device for displaying the contents of a conversation between users. The transparent display 50 is placed on the placement section 12, and is connected by wire to the USB hub 40 inside the accommodation section 14. The transparent display 50 is connected to the USB hub 40 via, for example, an HDMI (High-Definition Multimedia Interface; registered trademark) cable. As a result, the transparent display 50 receives data of a conversation screen showing the contents of the conversation from the tablet terminal 60 via the USB hub 40, and displays the conversation screen. The power cable of the transparent display 50 is connected to the external power source 70 through the inside of the housing portion 14. Thus, the transparent display 50 is supplied with power from the external power source 70.

[0036] The transparent display 50 is a display that has a degree of transparency that allows users to see each other's facial expressions. Therefore, since users can check the facial expressions of the other user, face-to-face communication is not obstructed. In addition, since the transparent display 50 has transparency, the background (the other user) can be seen through, and thus the display does not visually obstruct the view, blends into the surrounding environment, and enables a variety of expressions, such as displaying an image superimposed on the real object (the other user), thereby expanding the scope of communication.

[0037] The screen size of the transparent display 50 has an area large enough to function as a partition, and may be, for example, about 10 to 50 inches. The transparent display 50 can display various information other than the conversation screen. This allows various video expressions to be used to broaden the scope of communication.

[0038] (1-5) 60 tablet devices The tablet terminal 60 is a terminal shared by the resident U1 and the operator U2, and is an example of a first operation terminal. The tablet terminal 60 is placed on the placement surface of the tray unit 15. Therefore, the resident U1 can operate the tablet terminal 60 from the first main surface side of the transparent display 50, and the operator U2 can operate the tablet terminal 60 from the second main surface side of the transparent display 50. The tablet terminal 60 is connected by wire to the USB hub 40 inside the housing section 14. The tablet terminal 60 is connected to the USB hub 40 via, for example, a USB cable. The tablet terminal 60 is also connected to the multilingual translation device 80 via the network NW so as to be able to communicate with each other.

[0039] On the tablet terminal 60, various screens are displayed by an application (hereinafter also referred to as a "conversation assistance app") for functioning as a terminal that assists the conversation between the resident U1 and the operator U2. The function of the conversation assistance app may be provided by installing the conversation assistance app on the tablet terminal 60 (i.e., a native app), or may be provided by a Web system (i.e., a Web app). In the case of a Web system, the function of the conversation assistance app is provided via a Web browser.

[0040] A first operation area operated by the resident U1 and a second operation area operated by the operator U2 are displayed on the tablet terminal 60. For example, the first operation area and the second operation area are displayed on the same screen on the tablet terminal 60. This allows the operator U2 to check how the resident U1 is operating the screen on the terminal that he or she operates, and therefore allows the operator U2 to easily support the operation of the resident U1.

[0041] In the first operation area, an operation menu for the user U1 is displayed, such as for selecting an input means or a language. In the second operation area, an operation menu for the operator U2 is displayed, such as for setting the conversation assistance device 10, setting the display on the transparent display 50, or switching the display.

[0042] In selecting the input means, the resident U1 can select either voice input or keyboard input. Therefore, a resident U1 who has no trouble speaking can input the contents of the conversation by speaking by selecting voice input on the tablet terminal 60. On the other hand, a resident U1 who has difficulty speaking can input the contents of the conversation by typing by selecting keyboard input on the tablet terminal 60. This allows even a resident U1 who has difficulty speaking, such as a person with hearing loss or speech impairment, to communicate with the operator U2 through conversation.

[0043] When the conversation content is input by voice input, the tablet terminal 60 receives voice data from the audio interface 30 via the USB hub 40. The tablet terminal 60 transmits the voice data to the multilingual translation device 80 via the network NW, and receives the translation result of the conversation content indicated by the voice data from the multilingual translation device 80. The tablet terminal 60 transmits data of a conversation screen that displays the translation result to the transparent display 50, and causes it to be displayed.

[0044] In selecting the language, the resident U1 can select the language to be used for input. The selectable languages ​​include, but are not limited to, Japanese, English, Chinese (simplified characters), Chinese (traditional characters), Korean, Indonesian, Spanish, Thai, Filipino, French, Vietnamese, Portuguese, and Burmese. When keyboard input is selected as the input method, a digital keyboard of the language selected from these languages ​​is displayed on the tablet terminal 60. Therefore, the resident U1 who uses a language different from that of the operator U2 can input the conversation content in his / her own language by selecting the language he / she uses on the tablet terminal 60. As a result, even if the resident U1 uses a language different from that of the operator U2, he / she can communicate with the operator U2 through conversation.

[0045] When the conversation content is input by keyboard input, the tablet terminal 60 acquires data (hereinafter also referred to as "text data") indicating the conversation content input by typing from the digital keyboard in text form. The tablet terminal 60 transmits the text data to the multilingual translation device 80 via the network NW, and receives the translation result of the conversation content indicated by the text data from the multilingual translation device 80. The tablet terminal 60 transmits data of a conversation screen that displays the translation result to the transparent display 50, and causes it to be displayed.

[0046] Furthermore, by having the resident U1 and the operator U2 share the tablet terminal 60, there is no need to provide multiple tablet terminals 60, which makes it possible to secure space for other uses and prevent the wiring from becoming complicated.

[0047] (2) Multilingual translation device 80 The multilingual translation device 80 is a device that translates conversation content expressed in a language used by one user into conversation content expressed in a language used by another user. The multilingual translation device 80 is configured, for example, by one or more PCs (Personal Computers) and server devices (for example, cloud servers).

[0048] The multilingual translation device 80 is communicably connected to the tablet terminal 60 via the network NW. As a result, the multilingual translation device 80 receives voice data or text data from the tablet terminal 60 and transmits the translation results of each data to the tablet terminal 60. When the contents of the conversation are input by voice input, the multilingual translation device 80 receives voice data from the tablet terminal 60 and performs voice recognition processing on the voice data. The multilingual translation device 80 converts the voice represented by the voice data into characters through the voice recognition processing, and generates text data representing the characters. The multilingual translation device 80 performs translation processing on the text data obtained by the voice recognition processing. When the conversation content is input by keyboard input, the multilingual translation device 80 receives the text data from the tablet terminal 60 and executes a translation process on the text data.

[0049] <1-3. Functional configuration of tablet devices> The configuration of the conversation display system 1 according to the first embodiment has been described above. Next, the functional configuration of the tablet terminal 60 according to the first embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the functional configuration of the tablet terminal 60 according to the first embodiment. As shown in FIG. 3, the tablet terminal 60 includes a communication unit 610, an input unit 620, a storage unit 630, a control unit 640, and an output unit 650.

[0050] (1) Communications Department 610 The communication unit 610 has a function of transmitting and receiving various information. The communication unit 610 is communicatively connected to the USB hub 40 by a wired connection. This allows the communication unit 610 to transmit and receive various information to and from the audio interface 30 and the transparent display 50 via the USB hub 40. The communication unit 610 is also communicatively connected to the multilingual translation device 80 via the network NW, and transmits and receives various information.

[0051] (2) Input unit 620 The input unit 620 has a function of receiving input. The input unit 620 is configured by, for example, an input device that the tablet terminal 60 has as hardware, such as a button, a touch panel, a microphone, and the like.

[0052] (3) Storage section 630 The storage unit 630 has a function of storing various information. The storage unit 630 is configured by a storage medium provided in the tablet terminal 60 as hardware, such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, an electrically erasable programmable read only memory (EEPROM), a random access read / write memory (RAM), a read only memory (ROM), or any combination of these storage media.

[0053] (4) Control unit 640 The control unit 640 has a function of controlling the overall operation of the tablet terminal 60. The control unit 640 is realized, for example, by causing a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) that the tablet terminal 60 has as hardware to execute a program. As shown in FIG. 3, the control unit 640 includes an input selection processing unit 641, an input receiving unit 642, a multilingual translation processing unit 643, and a display processing unit 644.

[0054] (4-1) Input selection processing unit 641 The input selection processing unit 641 has a function of performing processing related to input selection. The input selection processing unit 641 accepts a selection of an input means, either voice input or keyboard input, and a selection of a language to be used for the selected input means in a first operation area displayed on the output unit 650.

[0055] (4-2) Input Reception Unit 642 The input reception unit 642 has a function of receiving input of conversation contents. The input reception unit 642 receives the input of the first conversation contents input by the resident U1 through voice input or keyboard input, and the input of the second conversation contents input by the operator U2 through voice input. When the resident U1 inputs the first conversation contents through voice input, the input reception unit 642 receives voice data received by the communication unit 610 from the audio interface 30 via the USB hub 40 as the input of the first conversation contents. When the resident U1 inputs the first conversation contents through keyboard input, the input reception unit 642 receives text data input by typing at the input unit 620 as the input of the first conversation contents. When the operator U2 inputs the second conversation contents through voice input, the input reception unit 642 receives voice data received by the communication unit 610 from the audio interface 30 via the USB hub 40 as the input of the second conversation contents.

[0056] (4-3) Multilingual translation processing unit 643 The multilingual translation processing unit 643 has a function of performing processing related to multilingual translation. The multilingual translation processing unit 643 causes the multilingual translation device 80 to translate the content of the conversation between the resident U1 and the operator U2, and acquires the translation result.

[0057] When the input receiving unit 642 receives the input of the first conversation content, the multilingual translation processing unit 643 transmits the first conversation content input by the resident U1 to the tablet terminal 60 to the multilingual translation device 80 via the communication unit 610. The first conversation content is translated by the multilingual translation device 80 from the language selected in the first operation area to the language used by the operator U2. The first translation result of the first conversation content is transmitted from the multilingual translation device 80 to the tablet terminal 60 via the network NW. The multilingual translation processing unit 643 acquires the first translation result received by the communication unit 610 from the multilingual translation device 80. As a result, the multilingual translation processing unit 643 acquires the first translation result in which the first conversation content is translated from the language selected in the first operation area to the language used by the operator U2.

[0058] When the input receiving unit 642 receives the input of the second conversation content, the multilingual translation processing unit 643 transmits the second conversation content input by the operator U2 to the tablet terminal 60 to the multilingual translation device 80 via the communication unit 610. The second conversation content is translated by the multilingual translation device 80 from the language used by the operator U2 to the language selected in the first operation area. The second translation result of the second conversation content is transmitted from the multilingual translation device 80 to the tablet terminal 60 via the network NW. The multilingual translation processing unit 643 acquires the second translation result received by the communication unit 610 from the multilingual translation device 80. As a result, the multilingual translation processing unit 643 acquires the second translation result in which the second conversation content is translated from the language used by the operator U2 to the language selected in the first operation area.

[0059] (4-4) Display processing unit 644 The display processing unit 644 has a function of performing processing related to display. The display processing unit 644 controls the display on the tablet terminal 60 and the display on the transparent display 50.

[0060] The display processing unit 644 displays a first operation area operated by a resident U1 and a second operation area operated by an operator U2 on a tablet terminal 60 shared by a resident U1 on the first main surface side of the transparent display 50 and an operator U2 on the second main surface side. When keyboard input is selected as an input means in the first operation area, the display processing unit 644 displays a digital keyboard capable of inputting in the selected language in the first operation area that accepted the selection.

[0061] The display processing unit 644 displays, on the transparent display 50, the first character information indicating the first translation result and the second character information indicating the second translation result. The display processing unit 644 displays the first character information of the first conversation content input by the resident U1 on the transparent display 50 so as to be visible from the second main surface side. Here, when the first character information is displayed on the second main surface side of the transparent display 50, it is sufficient that the operator U2 who is on the second main surface side can read the displayed characters. In this case, the display processing unit 644 displays characters corresponding to the first character information in a state where they are visible in an upside-down state when viewed from the first main surface side. That is, the resident U1 who is on the first main surface side visually recognizes the characters of the first character information displayed on the second main surface side of the transparent display 50 in a state where the left and right are reversed. Meanwhile, the display processing unit 644 displays the second character information of the second conversation content input by the operator U2 on the transparent display 50 so as to be visible from the first main surface side. Here, when the second character information is displayed on the first main surface side of the transparent display 50, it is sufficient that the resident U1 who is on the first main surface side can read the displayed characters. In this case, the display processing unit 644 displays characters corresponding to the second character information so as to be visible in an upside-down state as viewed from the second main surface side. That is, the operator U2 who is on the second main surface side visually recognizes the characters of the second character information displayed on the first main surface side of the transparent display 50 in a state in which the left and right are reversed.

[0062] The display processing unit 644 displays the first character information and the second character information on the transparent display 50 sequentially together with the input of the conversation content, distinguishing between the character information indicating the conversation content input by the resident U1 and the operator U2. For example, the display processing unit 644 displays the conversation content input by the resident U1 and the conversation content input by the operator U2 in different speech bubbles. The display processing unit 644 indicates whether the conversation content input by the resident U1 or the operator U2 is input by the direction of the tail (horn) of the speech bubble. The display processing unit 644 generates and displays a new speech bubble every time new conversation content is input. Further, the display processing unit 644 displays the first character information and the second character information on the transparent display 50 so as not to overlap each other when viewed from the first main surface side or the second main surface side. For example, the display processing unit 644 displays the first character information along one end of the transparent display 50 in the left-right direction, and displays the second character information along the end opposite to the end where the first character information is displayed, and displays each piece of character information arranged vertically in chronological order. As a result, the display processing unit 644 displays the first character information and the second character information in an interactive format. Note that, in the first embodiment, an example in which the first character information and the second character information are displayed in a horizontally written state will be described, but the present invention is not limited to such an example, and the first character information and the second character information may be displayed in a vertically written state.

[0063] The display processing unit 644 may switch the display on the transparent display 50 in response to an operation by the operator U2. For example, when the operator U2 performs an operation to turn on the transparent mode in the second operation area, the display processing unit 644 temporarily hides the content being displayed on the transparent display 50 and sets the display to a transparent state. When the operator U2 performs an operation to turn off the transparent mode in this state in the second operation area, the display processing unit 644 returns the display on the transparent display 50 to the original state. Furthermore, when the operator U2 selects a content in the second operation area while talking with the resident U1, the display processing unit 644 switches the display of the conversation content displayed on the transparent display 50 to the display of the selected content. That is, the display processing unit 644 can temporarily hide the conversation history displayed on the transparent display 50 and display a content instead. Note that the display processing unit 644 may display a content on the transparent display 50 when the resident U1 and the operator U2 are not talking and the conversation assistance device 10 is not in use. The content is, for example, signage content, advertisements, floor guides, operation guides, etc.

[0064] (5) Output section 650 The output unit 650 has a function of outputting various information. The output unit 650 is, for example, an output device that the tablet terminal 60 has as hardware, for example, a display device such as a display device or a touch screen (touch panel), or an audio output device such as a speaker.

[0065] <1-4. Functional configuration of the multilingual translation device> The functional configuration of the tablet terminal 60 according to the first embodiment has been described above. Next, the functional configuration of the multilingual translation device 80 according to the first embodiment will be described with reference to Fig. 4. Fig. 4 is a block diagram showing an example of the functional configuration of the multilingual translation device 80 according to the first embodiment. As shown in FIG. 4, the multilingual translation device 80 includes a communication unit 810, a storage unit 820, and a control unit 830.

[0066] (1) Communications Department 810 The communication unit 810 has a function of transmitting and receiving various information. The communication unit 810 is communicably connected to the tablet terminal 60 via the network NW, and transmits and receives various information.

[0067] (2) Storage section 820 The storage unit 820 has a function of storing various information. The storage unit 820 is configured by a storage medium provided as hardware in the multilingual translation device 80, such as a HDD, SSD, flash memory, EEPROM, RAM, ROM, or any combination of these storage media.

[0068] (3) Control unit 830 The control unit 830 has a function of controlling the overall operation of the multilingual translation device 80. The control unit 830 is realized, for example, by causing a CPU or a GPU that the multilingual translation device 80 has as hardware to execute a program. As shown in FIG. 4, the control unit 830 includes a voice recognition unit 831 and a multilingual translation unit 832.

[0069] (3-1) Voice recognition unit 831 The voice recognition unit 831 has a function of performing voice recognition processing. When the contents of the conversation are input by voice input, the voice recognition unit 831 acquires voice data that the communication unit 810 receives from the tablet terminal 60, and performs voice recognition processing on the voice data. The voice recognition unit 831 generates text data from the voice data by the voice recognition processing. When the contents of the conversation are input by keyboard input, the voice recognition unit 831 does not perform voice recognition processing.

[0070] (3-2) Multilingual Translation Department 832 The multilingual translation unit 832 has a function of performing translation processing. When the conversation content is input by voice input, the multilingual translation unit 832 performs translation processing on the text data generated by the voice recognition unit 831. When the conversation content is input by keyboard input, the multilingual translation unit 832 acquires the text data that the communication unit 810 receives from the tablet terminal 60, and performs translation processing on the text data. After the translation process, the multilingual translation unit 832 transmits the translation result obtained by the translation process to the tablet terminal 60 via the communication unit 810.

[0071] <1-5. Screen display example> The functional configuration of the multilingual translation device 80 according to the first embodiment has been described above. Next, examples of screen displays according to the first embodiment will be described with reference to Figs.

[0072] (1) Input selection screen An input selection screen according to the first embodiment will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of an input selection screen according to the first embodiment.

[0073] 5 is a screen displayed on the tablet terminal 60 for selecting an input means and a language. In the input selection screen G1, an area R1 is a first operation area operated by a resident U1, and an area R2 is a second operation area operated by an operator U2.

[0074] A message is displayed in area R3 in the first operation area, instructing the user to select either voice input or keyboard input as the input method. The user U1 can select the desired input method by pressing button B1 indicating voice input or button B2 indicating keyboard input. In addition, a message is displayed in area R4 instructing the user to select the language to be used for input. The user U1 can select the desired language by pressing one of the buttons B3 indicating each language.

[0075] The button B4 in the second operation area is a button for starting a translation mode in the selected language. The button B4 is automatically pressed when the resident U1 selects the language, so the operator U2 does not need to operate it. The button B5 is a button for displaying registered content on the transparent display 50. In the example shown in FIG. 5, the button B5 includes a button for displaying one of the registered contents and a button for displaying all the contents. The button B6 is a button for displaying a setting screen.

[0076] (2) Voice input screen The voice input screen according to the first embodiment will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of the voice input screen according to the first embodiment.

[0077] 6 is a screen displayed on the tablet terminal 60 when voice input is selected as the input means. In the voice input screen G2, an area R1 is a first operation area operated by a resident U1, and an area R2 is a second operation area operated by an operator U2.

[0078] The button B7 in the first operation area is a button for activating voice input on the part of the resident U1. The resident U1 can perform voice input by pressing the button B7 in the first operation area. A message instructing the resident U1 to press the button B7 to perform voice input is displayed in the area R5. The selected language is displayed in the area R6.

[0079] The button B7 in the second operation area is a button for activating voice input on the operator U2 side. The operator U2 can perform voice input by pressing the button B7 in the second operation area. The button B8 is a button for temporarily hiding the content displayed on the transparent display 50. When the button B8 is pressed, the display screen of the transparent display 50 becomes transparent (a state in which nothing is displayed), and when the button B8 is pressed again, the original display is restored. The button B9 is a button for deleting characters displayed on the transparent display 50. The button B9 is used when, for example, a mistranslation is displayed. The button B10 is a button for returning to the home screen. The slide bar SB is an operator for changing the size of the characters displayed on the transparent display 50 by sliding it left and right.

[0080] (3) Keyboard input screen A keyboard input screen according to the first embodiment will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of a keyboard input screen according to the first embodiment.

[0081] 7 is a screen displayed on the tablet terminal 60 when keyboard input is selected as the input means. In the keyboard input screen G3, an area R1 is a first operation area operated by a resident U1, and an area R2 is a second operation area operated by an operator U2.

[0082] The keyboard KB in the first operation area is a digital keyboard for user U1 to input data. In FIG. 7, as an example, a Burmese digital keyboard is displayed because keyboard input and Burmese language have been selected on the input selection screen G1. User U1 can input data by typing on the digital keyboard. Area R5 displays a message instructing user U1 to input requirements on the digital keyboard. Area R6 displays the selected language. Area R7 displays the content entered by user U1 on the digital keyboard. The second operation area has the same layout as the voice input screen G2.

[0083] (4) Conversation screen A conversation screen according to the first embodiment will be described with reference to Fig. 8. Fig. 8 is a diagram showing an example of a conversation screen according to the first embodiment. Fig. 8 shows, as an example, a conversation screen as viewed from the first main surface side (resident U1 side) of the transparent display 50. Note that, in the example shown in Fig. 8, a case will be described in which the resident U1 speaks English and the operator U2 speaks Japanese.

[0084] The conversation screen G4 shown in FIG. 8 is a screen displayed on the transparent display 50 when the resident U1 and the operator U2 are having a conversation. In the example shown in FIG. 8, as an example, in the left-right direction of the conversation screen G4, the conversation contents (first character information) of the resident U1 are displayed in chronological order in the up-down direction at the left end, and the conversation contents (second character information) of the operator U2 are displayed in chronological order in the up-down direction at the right end. In the conversation screen G4, the tail of the speech bubble is directed to the outside or center of the conversation screen G4, thereby distinguishing whether the conversation contents are those of the resident U1 or the operator U2. For example, when viewed from the first main surface side (the resident U1 side) of the transparent display 50, a speech bubble with its tail directed to the outside of the conversation screen G4 indicates the conversation contents of the resident U1, and a speech bubble with its tail directed to the center indicates the conversation contents of the operator U2. When viewed from the second main surface side (operator U2 side) of the transparent display 50, the speech bubble with its tail pointing outward on the conversation screen G4 indicates the conversation content of operator U2, and the speech bubble with its tail pointing toward the center indicates the conversation content of resident U1.

[0085] The speech bubbles F1 and F2 display the conversation contents input by the operator U2. When the operator U2 voice-inputs "Hello" in Japanese, a speech bubble F1 showing "Hello" in Japanese is generated and displayed on the second main surface side of the transparent display 50. In addition, "Hello" is translated from Japanese to English, and a speech bubble F2 showing "Hello." is generated and displayed on the first main surface side of the transparent display 50.

[0086] The speech bubbles F3 and F4 display the conversation contents input by the operator U2 in succession. When the operator U2 voice-inputs "How is it today?" in Japanese, a speech bubble F3 showing "How is it today?" in Japanese is generated and displayed on the second main surface side of the transparent display 50. In addition, "How is it today?" is translated from Japanese to English, and a speech bubble F4 showing "What brings you here today?" is generated and displayed on the first main surface side of the transparent display 50.

[0087] The speech bubbles F5 and F6 display the conversation content entered by the consumer U1. When the consumer U1 voice-inputs or keyboard-inputs "Please tell me a recommended Japanese restaurant around?" in English, a speech bubble F5 indicating "Please tell me a recommended Japanese restaurant around?" in English is generated and displayed on the first main surface side of the transparent display 50. In addition, "Please tell me a recommended Japanese restaurant around?" is translated from English to Japanese, and a speech bubble F6 indicating "Please tell me a recommended Japanese restaurant around here" is generated and displayed on the second main surface side of the transparent display 50.

[0088] (5) Changes in the input screen display A display change on an input screen according to the first embodiment will be described with reference to Fig. 9. Fig. 9 is a diagram showing an example of a display change on an input screen according to the first embodiment. Fig. 9 shows an example of a display change on a voice input screen.

[0089] The voice input screen G2-1 shown in FIG. 9 shows a state before the display change. In the voice input screen G2-1, the button B4 is in a selected state, and the button B7 is in an active state. In this state, it is assumed that the operator U2 presses a button corresponding to the content to be displayed on the transparent display 50 from among the buttons B5 in the region R2, which is the second operation region. In this case, the state of the voice input screen G2-1 changes to the state of the voice input screen G2-2. In the voice input screen G2-2, the button B4 is in an unselected state, and the button B7 is in an inactive state. In this way, in the voice input screen G2, pressing the button B5 makes it impossible for the resident U1 and the operator U2 to input voice. Similarly, in the keyboard input screen G3, pressing the button B5 makes it impossible for the resident U1 to input from the keyboard, and impossible for the operator U2 to input voice.

[0090] <1-6. Processing flow> Above, a description has been given of examples of screen displays according to the first embodiment. Next, a processing flow according to the first embodiment will be described with reference to Figs.

[0091] (1) Processing flow in the conversation display system 1 A process flow in the conversation display system 1 according to the first embodiment will be described with reference to Fig. 10. Fig. 10 is a sequence diagram showing an example of a process flow in the conversation display system 1 according to the first embodiment.

[0092] 10, first, the tablet terminal 60 performs an input selection process (step S101). The details of the input selection process will be described later. When the input means is selected by the resident U1 in the input selection process, the input acceptance unit 642 of the tablet terminal 60 accepts the input by the selected input means (step S102). If the conversation contents are input by voice input, the input acceptance unit 642 acquires voice data. If the conversation contents are input by keyboard input, the input acceptance unit 642 acquires text data. The multilingual translation processing unit 643 transmits the voice data or text data indicating the contents of the conversation to the multilingual translation device 80 via the communication unit 610 (step S103).

[0093] The multilingual translation device 80 checks the format of the data that the communication unit 810 receives from the tablet terminal 60 (step S104). If the data format is voice data (step S104 / YES), the process proceeds to step S105. On the other hand, if the data format is text data (step S104 / NO), the process proceeds to step S106.

[0094] When the process proceeds to step S105, the voice recognition unit 831 acquires voice data received by the communication unit 810, executes voice recognition processing on the voice data, and generates text data (step S105). After the text data is generated, the process proceeds to step S106. When the process proceeds to step S106, the multilingual translation unit 832 acquires the text data received by the communication unit 810 or the text data generated by the voice recognition unit 831, and executes a translation process on the text data (step S106). After the translation process, the multilingual translation unit 832 transmits the translation result to the tablet terminal 60 via the communication unit 810 (step S107).

[0095] The display processing unit 644 of the tablet terminal 60 transmits the translation result received by the communication unit 610 from the multilingual translation device 80 to the transparent display 50 via the communication unit 610 (step S108). The transparent display 50 displays the translation result received from the tablet terminal 60 (step S109).

[0096] (2) Input selection process flow The flow of the input selection process according to the first embodiment will be described with reference to Fig. 11. Fig. 11 is a flowchart showing an example of the flow of the input selection process according to the first embodiment.

[0097] As shown in FIG. 11, first, the display processing unit 644 of the tablet terminal 60 displays the input selection screen G1 on the output unit 650 (step S1011). Next, the input selection processing unit 641 receives a selection of an input means and an input language from the user U1 (step S1012).

[0098] If voice input is selected as the input means (step S1013 / voice input), the process proceeds to step S1014. If keyboard input is selected as the input means (step S1013 / keyboard input), the process proceeds to step S1015.

[0099] When the process proceeds to step S1014, the display processing unit 644 displays the voice input screen G2 on the output unit 650 (step S1014). When the process proceeds to step S1015, the display processing unit 644 displays a keyboard input screen G3 on which a digital keyboard of the selected language is displayed on the output unit 650 (step S1015).

[0100] As described above, the conversation assistance device 10 according to the first embodiment includes a display processing unit 644 that displays a first operation area operated by a resident U1 (first user) on the first main surface side of the transparent display 50, and a second operation area operated by an operator U2 on a tablet terminal 60 (first operation terminal) shared by the resident U1 and the operator U2 on the second main surface side of the transparent display 50, and an input selection processing unit 641 that accepts a selection of either voice input or keyboard input as an input means in the first operation area, and a selection of a language to be used in the selected input means. When keyboard input is selected as the input means, the display processing unit 644 displays a digital keyboard capable of input using the selected language in the first operation area that accepted the selection.

[0101] With this configuration, a consumer U1 who has difficulty speaking, such as a hearing-impaired or speech-impaired person, can communicate with an operator U2 by operating the tablet terminal 60 to select keyboard input and the language to be used, and then typing on the digital keyboard in the language that the consumer uses to input the content of the conversation. Therefore, the conversation assistance device 10 according to the first embodiment makes it possible to realize two-way communication with a person who has difficulty speaking.

[0102] <<2. Second embodiment>> The first embodiment has been described above. Next, the second embodiment will be described with reference to Figs. 12 to 16. In the above-mentioned first embodiment, an example has been described in which a resident U1 who wishes to perform keyboard input performs keyboard input using a tablet terminal 60 shared with an operator U2, but the present invention is not limited to such an example. For example, keyboard input may be performed using a user terminal owned by the resident U1. This allows the resident U1 to perform keyboard input using a keyboard and input operation (such as flick input) that he or she normally uses and is familiar with. In the following, as an example, a case where a user U1 performs keyboard input using his / her own smartphone will be described. Note that, in the following, descriptions that overlap with the description in the first embodiment will be omitted as appropriate.

[0103] <2-1. Structure of the conversation display system> The configuration of the conversation display system according to the second embodiment will be described with reference to Fig. 12. Fig. 12 is a block diagram showing an example of the configuration of the conversation display system according to the second embodiment. As shown in Fig. 12, the conversation display system 1a according to the second embodiment further includes a user terminal 90 in addition to the configuration of the conversation display system 1 according to the first embodiment.

[0104] The user terminal 90 is a terminal owned by the resident U1 and is an example of a second operation terminal. The user terminal 90 is, for example, a smartphone, a tablet terminal, or a PC. The user terminal 90 is communicably connected to the multilingual translation device 80 via the network NW.

[0105] <2-2. Functional configuration of tablet device> The configuration of the conversation display system 1a according to the second embodiment has been described above. Next, the functional configuration of the tablet terminal 60 according to the second embodiment will be described. The functional configuration of the tablet terminal 60 according to the second embodiment is similar to the functional configuration of the tablet terminal 60 according to the first embodiment. However, there are some differences in the functions of the functional configuration.

[0106] When keyboard input is selected as the input means, the display processing unit 644 displays code information including connection information for receiving input of the first conversation content from the user terminal 90 (second operation terminal) owned by the resident U1 in the first operation area. The code information is, for example, a two-dimensional code. The connection information is, for example, the address of the connection destination (multilingual translation device 80).

[0107] The multilingual translation processing unit 643 acquires a first translation result obtained by translating the first conversation content input to the user terminal 90 connected using the connection information read from the code information from the language selected in the first operation area into the language used by the operator U2. The first conversation content input to the user terminal 90 by the resident U1 is transmitted from the user terminal 90 to the multilingual translation device 80 via the network NW and translated by the multilingual translation device 80. The first translation result of the first conversation content is transmitted from the multilingual translation device 80 to the tablet terminal 60 via the network NW. The multilingual translation processing unit 643 acquires the first translation result received by the communication unit 610 from the multilingual translation device 80.

[0108] <2-3. Functional configuration of user terminal> The configuration of the conversation display system 1a according to the second embodiment has been described above. Next, the functional configuration of the user terminal 90 according to the second embodiment will be described with reference to Fig. 13. Fig. 13 is a diagram showing an example of the functional configuration of the user terminal 90 according to the second embodiment. As shown in FIG. 13, the user terminal 90 includes a communication unit 910, a reading unit 920, an input unit 930, a storage unit 940, a control unit 950, and an output unit 960.

[0109] (1) Communications Department 910 The communication unit 910 has a function of transmitting and receiving various information. The communication unit 910 is communicably connected to the multilingual translation device 80 via the network NW, and transmits and receives various information.

[0110] (2) Reading unit 920 The reading unit 920 has a function of reading code information. The reading unit 920 is configured, for example, with a camera that the user terminal 90 includes as hardware. When code information is detected within the imaging range of the camera, the reading unit 920 reads the code information and acquires connection information for connecting to the multilingual translation device 80.

[0111] (3) Input section 930 The input unit 930 has a function of receiving input. The input unit 930 is, for example, an input device that the user terminal 90 has as hardware, such as a button, a touch panel, a microphone, and the like.

[0112] (4) Storage section 940 The storage unit 940 has a function of storing various information. The storage unit 940 is configured by a storage medium provided in the user terminal 90 as hardware, such as a HDD, SSD, flash memory, EEPROM, RAM, ROM, or any combination of these storage media.

[0113] (5) Control unit 950 The control unit 950 has a function of controlling the overall operation of the user terminal 90. The control unit 950 is realized, for example, by causing a CPU or a GPU that the user terminal 90 has as hardware to execute a program. As shown in FIG. 13, the user terminal 90 includes a connection processing unit 951, an input processing unit 952, a transmission processing unit 953, and a display processing unit 954.

[0114] (5-1) Connection Processing Unit 951 The connection processing unit 951 has a function of performing processing for connecting the user terminal 90 to the conversation display system 1a. The connection processing unit 951 connects the user terminal 90 to the multilingual translation device 80 so that they can communicate with each other, based on the information acquired by the reading unit 920.

[0115] (5-2) Input processing unit 952 The input processing unit 952 has a function of performing processing related to input. The input processing unit 952 accepts the input of the first conversation content input by the person U1 through keyboard input. When the person U1 inputs the first conversation content through keyboard input into the user terminal 90, the input processing unit 952 accepts text data input by typing into the input unit 930 as the input of the first conversation content.

[0116] (5-3) Transmission Processing Unit 953 The transmission processing unit 953 has a function of performing processing related to transmission. When the input of the first conversation content is accepted by the input processing unit 952, the transmission processing unit 953 transmits text data of the first conversation content to the multilingual translation device 80 via the communication unit 910.

[0117] (5-4) Display processing unit 954 The display processing unit 954 has a function of performing processing related to display. When the connection processing unit 951 performs connection processing, the display processing unit 954 displays a keyboard input screen on the output unit 960.

[0118] (6) Output unit 960 The output unit 960 has a function of outputting various information. The output unit 960 is, for example, an output device provided in the user terminal 90 as hardware, for example, a display device such as a display device or a touch screen (touch panel), or an audio output device such as a speaker.

[0119] <2-4. Screen display example> The functional configuration of the user terminal 90 according to the second embodiment has been described above. Next, examples of screen displays according to the second embodiment will be described with reference to Figs.

[0120] (1) User device connection screen A user terminal connection screen according to the second embodiment will be described with reference to Fig. 14. Fig. 14 is a diagram showing an example of a user terminal connection screen according to the second embodiment.

[0121] 14 is a screen displayed on the tablet terminal 60 when keyboard input is selected as the input means and keyboard input is selected on the user terminal 90. In the user terminal connection screen G5, an area R1 is a first operation area operated by the resident U1, and an area R2 is a second operation area operated by the operator U2.

[0122] In the first operation area, a two-dimensional code CD (code information) is displayed as shown in Fig. 14. The consumer U1 can perform keyboard input using the user terminal 90 by reading the two-dimensional code CD using the user terminal 90. The second operation area has the same layout as the voice input screen G2.

[0123] (2) Keyboard input screen A keyboard input screen according to the second embodiment will be described with reference to Fig. 15. Fig. 15 is a diagram showing an example of a keyboard input screen according to the second embodiment.

[0124] The keyboard input screen G6 shown in Fig. 15 is a screen that is displayed on the user terminal 90 when the consumer U1 reads the code information displayed on the user terminal connection screen G5 with the user terminal 90. In the area R11, a message is displayed instructing the consumer U1 to input in the language selected by the consumer U1. In the area R12, the conversation content input by the consumer U1 is displayed in chronological order. The keyboard KB displays a digital keyboard in the language selected by the consumer U1 on the input selection screen G1 of the tablet terminal 60.

[0125] <2-5. Processing flow> The above describes the screen display example according to the second embodiment. Next, the process flow according to the second embodiment will be described with reference to Fig. 16. Fig. 16 is a sequence diagram showing an example of the process flow in the conversation display system 1a according to the second embodiment.

[0126] 16, first, the tablet terminal 60 accepts the selection of keyboard input as an input means (step S201). Specifically, the selection of keyboard input is accepted in the same manner as the input selection process described with reference to FIG. 11. Furthermore, the tablet terminal 60 also accepts the selection of keyboard input using the user terminal 90. When the tablet terminal 60 accepts the selection of keyboard input using the user terminal 90, it displays the user terminal connection screen G5 and displays the code information in the first operation area (step S202).

[0127] The reading unit 920 of the user terminal 90 activates the camera in response to an operation by the resident U1, and reads the code information displayed on the user terminal connection screen G5 of the tablet terminal 60 (step S203). The connection processing unit 951 of the user terminal 90 performs a connection process for establishing a communicable connection with the multilingual translation device 80 based on the connection information acquired from the code information read by the reading unit 920 (step S204). After the connection process, the display processing unit 954 of the user terminal 90 displays a keyboard input screen G6 on the output unit 960 (step S205). When the conversation content is input on the keyboard input screen G6, the transmission processing unit 953 of the user terminal 90 transmits the text data of the input conversation content to the multilingual translation device 80 via the communication unit 910 (step S206). After the text data is transmitted, the display processing unit 954 displays the transmission date and time of the conversation content for which the text data was transmitted on the keyboard input screen G6 (step S207).

[0128] The multilingual translation unit 832 of the multilingual translation device 80 acquires the text data received by the communication unit 810 from the user terminal 90, and executes a translation process on the text data (step S208). After the translation process, the multilingual translation unit 832 transmits the translation result to the tablet terminal 60 via the communication unit 810 (step S209).

[0129] The display processing unit 644 of the tablet terminal 60 transmits the translation result received by the communication unit 610 from the multilingual translation device 80 to the transparent display 50 via the communication unit 610 (step S210). The transparent display 50 displays the translation result received from the tablet terminal 60 (step S211).

[0130] <<3. Modifications>> The second embodiment has been described above. Next, modified examples of each embodiment will be described. The modified examples described below may be applied to each embodiment alone or in combination with each other. The modified examples may be applied in place of the configurations described in each embodiment, or may be applied in addition to the configurations described in each embodiment.

[0131] In the above-described embodiment, an anti-theft lock may be provided for the display base 11. The lock may be, for example, a chain lock, and may be provided at any position on the placement unit 12, the legs 13, the storage unit 14, and the tray unit 15. With the lock, for example, the electronic device stored inside the storage unit 14 and the tablet terminal 60 placed on the placement surface of the tray unit 15 can be locked (fixed) to the display base 11, thereby preventing theft.

[0132] The embodiment has been described above. In addition, some or all of the functions of the conversation display system 1, 1a, the conversation assistance device 10, the tablet terminal 60, the multilingual translation device 80, and the user terminal 90 in the above-mentioned embodiments may be realized by a computer. In that case, a program for realizing the functions may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the functions. Note that the "computer system" here includes hardware such as an OS and peripheral devices. In addition, the "computer-readable recording medium" refers to portable media such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, and a storage device such as a hard disk built into a computer system. Furthermore, the term "computer-readable recording medium" may also include something that dynamically holds a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, or something that holds a program for a certain period of time, such as volatile memory within a computer system that serves as the server or client in that case. Furthermore, the above program may be for realizing part of the functions described above, or may be capable of realizing the functions described above in combination with a program already recorded in a computer system, or may be realized using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0133] Although the embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0134] 1,1a...conversation display system, 10...conversation assistance device, 11...display base, 12...mounting section, 13...legs, 14...storage section, 15...tray section, 20 (20-1, 20-2)...sound collection device, 21...sound collection section, 30...audio interface, 40...USB hub, 50...transparent display, 60...tablet terminal, 70...external power supply, 80...multilingual translation device, 90...user terminal, 610...communication section, 620...input section, 630...storage section, 640...control unit, 641...input selection processing unit, 642...input reception unit, 643...multilingual translation processing unit, 644...display processing unit, 650...output unit, 810...communication unit, 820...storage unit, 830...control unit, 831...speech recognition unit, 832...multilingual translation unit, 910...communication unit, 920...reading unit, 930...input unit, 940...storage unit, 950...control unit, 951...connection processing unit, 952...input processing unit, 953...transmission processing unit, 954...display processing unit, 960...output unit, NW...network

Claims

1. a display processing unit that displays a first operation area operated by the first user and a second operation area operated by the second user on a first operation terminal shared by a first user on a first main surface side where a first main surface of a transparent display is located and a second user on a second main surface side where a second main surface is located, the second main surface being a main surface opposite to the first main surface; an input selection processing unit that accepts, in the first operation area, a selection of an input means, either a voice input or a keyboard input, and a selection of a language to be used in the selected input means; an input receiving unit that receives an input of a first conversation content input by the first user through the voice input or the keyboard input, and an input of a second conversation content input by the second user through the voice input; a multilingual translation processing unit that acquires a first translation result in which the first conversation content is translated from the language selected in the first operation area to a language used by the second user, and a second translation result in which the second conversation content is translated from the language used by the second user to the language selected in the first operation area; Equipped with when the keyboard input is selected as the input means, the display processing unit displays a digital keyboard capable of inputting using the selected language in the first operation area that has accepted the selection; the display processing unit displays, on the transparent display, first character information indicating the first translation result and second character information indicating the second translation result; the display processing unit, when the second user selects content in the second operation area during a conversation with the first user, switches a display of the conversation content displayed on the transparent display to a display of the selected content. Conversation aid.

2. The display processing unit distinguishes the first character information and the second character information so that it is possible to tell whether the first user or the second user has input the character information indicating the conversation content, and sequentially displays the first character information and the second character information on the transparent display together with the input of the conversation content. The conversation assistance device according to claim 1.

3. the display processing unit displays, on the transparent display, the first character information of the first conversation content input by the first user so as to be visible from the second main surface side, and displays, on the transparent display, the second character information of the second conversation content input by the second user so as to be visible from the first main surface side. The conversation assistance device according to claim 1.

4. the display processing unit displays the first character information and the second character information on the transparent display so as not to overlap each other when viewed from the first principal surface side or the second principal surface side. The conversation assistance device according to claim 1.

5. the display processing unit displays the first character information along one end of the transparent display in a left-right direction, and displays the second character information along an end opposite to the end where the first character information is displayed, and displays each piece of character information arranged vertically in chronological order. The conversation assistance device according to claim 1.

6. the display processing unit displays the first character information and the second character information in an interactive format. The conversation assistance device according to claim 5.

7. the display processing unit displays characters corresponding to the first character information in a visibly upside-down state as viewed from the first main surface side, and displays characters corresponding to the second character information in a visibly upside-down state as viewed from the second main surface side. The conversation assistance device according to claim 1.

8. the first character information and the second character information are displayed in a horizontally written state; The conversation assistance device according to any one of claims 1 to 7.

9. a display process for displaying a first operation area operated by the first user and a second operation area operated by the second user on a first operation terminal shared by a first user on a first main surface side where a first main surface of a transparent display is located and a second user on a second main surface side where a second main surface is located, the second main surface being a main surface opposite to the first main surface; an input selection process for accepting, in the first operation area, a selection of an input means, either voice input or keyboard input, and a selection of a language to be used in the selected input means; an input receiving step of receiving an input of a first conversation content input by the first user through the voice input or the keyboard input, and an input of a second conversation content input by the second user through the voice input; a multilingual translation process for obtaining a first translation result in which the first conversation content is translated from the language selected in the first operation area to the language used by the second user, and a second translation result in which the second conversation content is translated from the language used by the second user to the language selected in the first operation area; Including, When the keyboard input is selected as the input means, the display processing step displays a digital keyboard capable of inputting the selected language in the first operation area that accepts the selection; The display processing step includes displaying first character information indicating the first translation result and second character information indicating the second translation result on the transparent display; The display processing step includes, when the second user selects a content in the second operation area during a conversation with the first user, switching a display of the conversation content displayed on the transparent display to a display of the selected content. A computer-implemented method for speech assistance.

10. Computer, a display processing means for displaying a first operation area operated by the first user and a second operation area operated by the second user on a first operation terminal shared by a first user on a first main surface side where a first main surface of a transparent display is located and a second user on a second main surface side where a second main surface is located, the second main surface being a main surface opposite to the first main surface; an input selection processing means for receiving, in the first operation area, a selection of an input means, either a voice input or a keyboard input, and a selection of a language to be used in the selected input means; an input receiving means for receiving an input of a first conversation content input by the first user through the voice input or the keyboard input, and an input of a second conversation content input by the second user through the voice input; a multilingual translation processing means for acquiring a first translation result in which the first conversation content is translated from the language selected in the first operation area to a language used by the second user, and a second translation result in which the second conversation content is translated from the language used by the second user to the language selected in the first operation area; Function as a when the keyboard input is selected as the input means, the display processing means displays a digital keyboard capable of inputting using the selected language in the first operation area that has received the selection; The display processing means displays first character information indicating the first translation result and second character information indicating the second translation result on the transparent display; When the second user selects a content in the second operation area during a conversation with the first user, the display processing means switches a display of the conversation content displayed on the transparent display to a display of the selected content. program.

11. A display processing unit that displays a first operation area operated by the first user and a second operation area operated by the second user on a first operation terminal shared by a first user on a first main surface side where a first main surface of a transparent display is located, and a second user on a second main surface side where a second main surface is located, the second main surface being the main surface opposite to the first main surface; an input selection processing unit that accepts, in the first operation area, a selection of an input means, either a voice input or a keyboard input, and a selection of a language to be used in the selected input means; an input receiving unit that receives an input of a first conversation content input by the first user through the voice input or the keyboard input, and an input of a second conversation content input by the second user through the voice input; a multilingual translation processing unit that acquires a first translation result in which the first conversation content is translated from the language selected in the first operation area to a language used by the second user, and a second translation result in which the second conversation content is translated from the language used by the second user to the language selected in the first operation area; Equipped with when the keyboard input is selected as the input means, the display processing unit displays a digital keyboard capable of inputting using the selected language in the first operation area that has accepted the selection; the display processing unit displays, on the transparent display, first character information indicating the first translation result and second character information indicating the second translation result; the display processing unit, when the keyboard input is selected as the input means, displays, in the first operation area, code information including connection information for receiving input of the first conversation content from a second operation terminal owned by the first user; the multilingual translation processing unit obtains a first translation result in which the first conversation content input to the second operation terminal connected using the connection information read from the code information is translated from the language selected in the first operation area into the language used by the second user; Conversation aid.

12. A display processing process for displaying a first operation area operated by the first user and a second operation area operated by the second user on a first operation terminal shared by a first user on a first main surface side where a first main surface of a transparent display is located and a second user on a second main surface side where a second main surface is located, the second main surface being the main surface opposite to the first main surface; an input selection process for accepting, in the first operation area, a selection of an input means, either voice input or keyboard input, and a selection of a language to be used in the selected input means; an input receiving step of receiving an input of a first conversation content input by the first user through the voice input or the keyboard input, and an input of a second conversation content input by the second user through the voice input; a multilingual translation process for obtaining a first translation result in which the first conversation content is translated from the language selected in the first operation area to the language used by the second user, and a second translation result in which the second conversation content is translated from the language used by the second user to the language selected in the first operation area; Including, When the keyboard input is selected as the input means, the display processing step displays a digital keyboard capable of inputting the selected language in the first operation area that accepts the selection; The display processing step includes displaying first character information indicating the first translation result and second character information indicating the second translation result on the transparent display; the display processing step, when the keyboard input is selected as the input means, displays, in the first operation area, code information including connection information for receiving input of the first conversation content from a second operation terminal owned by the first user; The multilingual translation process includes obtaining a first translation result in which the first conversation content input to the second operation terminal connected using the connection information read from the code information is translated from the language selected in the first operation area into the language used by the second user. A computer-implemented method for speech assistance.

13. A computer comprising: a display processing means for displaying a first operation area operated by the first user and a second operation area operated by the second user on a first operation terminal shared by a first user on a first main surface side where a first main surface of a transparent display is located and a second user on a second main surface side where a second main surface is located, the second main surface being a main surface opposite to the first main surface; an input selection processing means for receiving, in the first operation area, a selection of an input means, either a voice input or a keyboard input, and a selection of a language to be used in the selected input means; an input receiving means for receiving an input of a first conversation content input by the first user through the voice input or the keyboard input, and an input of a second conversation content input by the second user through the voice input; a multilingual translation processing means for acquiring a first translation result in which the first conversation content is translated from the language selected in the first operation area into the language used by the second user, and a second translation result in which the second conversation content is translated from the language used by the second user into the language selected in the first operation area; Function as a when the keyboard input is selected as the input means, the display processing means displays a digital keyboard capable of inputting using the selected language in the first operation area that has received the selection; The display processing means displays first character information indicating the first translation result and second character information indicating the second translation result on the transparent display; the display processing means, when the keyboard input is selected as the input means, displays, in the first operation area, code information including connection information for receiving input of the first conversation content from a second operation terminal owned by the first user; the multilingual translation processing means obtains a first translation result in which the first conversation content input to the second operation terminal connected using the connection information read from the code information is translated from the language selected in the first operation area into the language used by the second user; program.

Images