Two-way communication system, server device, and communication program product

Abstract

In a system for speech recognition of spoken content, a user is enabled to understand a situation in which speech recognition is unsuitable. A system converts speech data into text data, wherein one or more control units, during acquisition of speech data, determine whether the acquired speech data can be converted into text data or whether the quality of the converted text data obtained by converting the acquired speech data is degraded, and output the determination result.

Description

Technical Field

[0001] The present invention relates to a system, a supporting method, a server device and a communication program. Background Art

[0002] A two-way communication system is known that enables remote communication by bidirectionally transmitting and receiving moving image data and voice data. In this two-way communication system, for example, by utilizing a voice-to-text function, the user's spoken content can be recognized in real time and converted into text data. As a result, the user's spoken content can be managed as text information using this two-way communication system.

[0003] <Prior Art Literature>

[0004] <Patent Document>

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2021-2747 Summary of the Invention

[0006] <Problems to be Solved by the Invention>

[0007] However, in the case of the above-mentioned two-way communication system, due to various reasons such as a malfunction of the microphone, deterioration of the communication status, and the influence of surrounding ambient sound, part of the utterance may not be properly recognized.

[0008] On the other hand, even if it is determined that a portion of the speech content has not been converted into appropriate text data after the remote communication ends, it is difficult to restore the text data.

[0009] The present invention enables a user to understand that a situation unsuitable for speech recognition has occurred in a system for performing speech recognition on spoken content.

[0010] <Methods used to solve the problem>

[0011] The first embodiment of the present invention is a system for converting voice data into text data. One or more control units, when acquiring voice data, determine whether the acquired voice data can be converted into text data, or whether the quality of the converted text data obtained by converting the acquired voice data is reduced, and output the determination result.

[0012] According to the first aspect of the present invention, in a system for performing speech recognition on utterances, a user can grasp the occurrence of a situation that is not suitable for speech recognition.

[0013] Furthermore, in a second aspect of the present invention, in the system according to the first aspect, the one or more control units output the determination result by clearly indicating a reason why the conversion to text data cannot be performed or a reason why the quality of the converted text data is degraded.

[0014] Furthermore, in a third aspect of the present invention, in the system according to the second aspect, the cause includes either an influence of ambient sound around a voice acquisition device that acquires the voice data or a failure of the voice acquisition device.

[0015] In addition, in the fourth mode of the present invention, in the system described in the third mode, when the size of the ambient sound data obtained in the acquisition of the voice data is greater than a given threshold, the one or more control units determine that the quality of the converted text data is reduced due to the influence of the ambient sound.

[0016] In addition, in the fifth embodiment of the present invention, in the system described in the third or fourth embodiment, when the size of the voice data being acquired and the ambient sound data acquired in the acquisition of the voice data are both smaller than a given threshold, the one or more control units determine that the voice acquisition device cannot be converted into text data due to a failure.

[0017] Furthermore, a sixth aspect of the present invention is the system according to any one of the second to fifth aspects, wherein the cause includes deterioration of a communication state of bidirectional communication.

[0018] Furthermore, in a seventh aspect of the present invention, in the system according to the sixth aspect, when bidirectional communication is disconnected during acquisition of the voice data, the one or more control units determine that conversion into text data is impossible due to deterioration of the communication state.

[0019] In addition, the eighth mode of the present invention is a system described in any one of the second to seventh modes, comprising: an operator terminal used by an operator; a supporter terminal used by a supporter who provides remote support to the operator; and a server device that realizes two-way communication between the operator terminal and the supporter terminal, and the control unit of the server device converts voice data into text data.

[0020] In addition, in the ninth aspect of the present invention, in the system described in the eighth aspect, the control unit of the server device determines whether the voice data being acquired can be converted into text data, or whether the quality of the converted text data obtained by converting the voice data being acquired is reduced, during the acquisition of the voice data, and the control unit of the supporter terminal outputs the determination result.

[0021] In addition, in the tenth embodiment of the present invention, in the system described in the ninth embodiment, the operator terminal has a voice acquisition device for acquiring the voice data, and the control unit of the server device determines that the quality of the converted text data is reduced due to the influence of the ambient sound around the voice acquisition device when the size of the ambient sound data acquired in the acquisition of the voice data is greater than a given threshold.

[0022] In addition, in the eleventh embodiment of the present invention, in the system described in the ninth embodiment, the operator terminal has a voice acquisition device for acquiring the voice data, and the control unit of the server device determines that the voice data cannot be converted into text data due to a failure of the voice acquisition device when the size of the voice data being acquired and the size of the ambient sound data acquired in the acquisition of the voice data are both smaller than a given threshold.

[0023] In addition, in the twelfth embodiment of the present invention, in the system described in the ninth embodiment, when the control unit of the server device determines that the two-way communication is disconnected because no voice data is sent from the operator terminal within a certain period of time, it is determined that the voice data obtained by the control unit of the operator terminal cannot be converted into text data due to the deterioration of the communication state of the two-way communication.

[0024] Furthermore, in a thirteenth aspect of the present invention, in the system according to any one of the tenth to twelfth aspects, the control unit of the supporter terminal displays the determination result using an icon indicating a reason why the conversion to text data cannot be performed or a reason why the quality of the converted text data is degraded.

[0025] In addition, in the fourteenth mode of the present invention, in the system described in the tenth or eleventh mode, the control unit possessed by the operator terminal determines whether the acquired voice data can be converted into text data, or whether the quality of the converted text data obtained by converting the acquired voice data is reduced, during the acquisition of the voice data, and reports the determination result through at least any one of voice, light, and vibration.

[0026] Furthermore, a fifteenth aspect of the present invention is the system according to any one of the ninth to fourteenth aspects, wherein the control unit of the server device stores the text data in association with the determination result.

[0027] Furthermore, a sixteenth aspect of the present invention is the system according to the fifteenth aspect, wherein the control unit included in the supporter terminal reads the stored text data and displays it in a display format corresponding to the corresponding determination result.

[0028] Furthermore, in a seventeenth aspect of the present invention, in the system according to the eighth to sixteenth aspects, the control unit of the server device uses the previous and next text data to correct portions that cannot be converted into text data or portions whose quality has deteriorated.

[0029] In addition, in the eighteenth mode of the present invention, in the system described in any one of the ninth to seventeenth modes, the control unit of the server device determines the work content of the operator based on the image captured by the operator terminal, and saves the determined work content in association with the judgment result.

[0030] In addition, in the nineteenth mode of the present invention, in the system described in any one of the ninth to eighteenth modes, the control unit possessed by the operator terminal saves the acquired voice data, and when there is a request from the server device, sends the voice data in the range corresponding to the request to the server device.

[0031] In addition, the twentieth embodiment of the present invention is a support method involving a system for converting voice data into text data, comprising: a judgment step, in which, during the acquisition of voice data, it is determined whether the voice data being acquired can be converted into text data, or whether the quality of the converted text data obtained by converting the voice data being acquired is reduced; and an output step, which outputs the judgment result.

[0032] In addition, the twenty-first embodiment of the present invention is a server device, which has a control unit that realizes two-way communication between an operator terminal used by an operator and a supporter terminal used by a supporter who provides remote support to the operator. The control unit converts voice data into text data, and in the acquisition of voice data performed by the operator terminal or the supporter terminal, when it is determined whether the voice data being acquired can be converted into text data or whether the quality of the converted text data obtained by converting the voice data being acquired is reduced, an instruction corresponding to the determination result is sent to the operator terminal or the supporter terminal.

[0033] In addition, the twenty-second embodiment of the present invention is a communication program, which is used to enable the control unit of a server device that realizes two-way communication between an operator terminal used by an operator and a supporter terminal used by a supporter who provides remote support to the operator to perform the following processes: a conversion process of converting voice data into text data; and a sending process of, in the acquisition of voice data by the operator terminal or the supporter terminal, when it is determined whether the acquired voice data can be converted into text data or whether the quality of the converted text data obtained by converting the acquired voice data is reduced, an instruction corresponding to the judgment result is sent to the operator terminal or the supporter terminal. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is the first diagram showing an example of a system configuration of a bidirectional communication system.

[0035] Figure 2 This is a diagram showing an example of the hardware configuration of a server device, and an example of the hardware configuration of an operator terminal and a supporter terminal.

[0036] Figure 3 This is the first diagram showing an example of the functional configuration of the operator terminal.

[0037] Figure 4 This is the first diagram showing an example of the details of the functional configuration of the server device.

[0038] Figure 5 This is the first diagram showing an example of the functional configuration of the supporter terminal.

[0039] Figure 6 This is a first sequence diagram showing the flow of communication processing when voice data is transmitted from the operator terminal to the supporter terminal in the two-way communication system.

[0040] Figure 7 This is a first sequence diagram showing the flow of communication processing when voice data is transmitted from a supporter terminal to a worker terminal in a two-way communication system.

[0041] Figure 8 It is a diagram showing an example of a display screen of a supporter terminal.

[0042] Figure 9 This is a second diagram showing an example of the functional configuration of the operator terminal.

[0043] Figure 10 This is a second diagram showing an example of the details of the functional configuration of the server device.

[0044] Figure 11 This is a second diagram showing an example of the functional configuration of the supporter terminal.

[0045] Figure 12 This is a second sequence diagram showing the flow of communication processing when voice data is transmitted from the operator terminal to the supporter terminal in the two-way communication system.

[0046] Figure 13 This is a second sequence diagram showing the flow of communication processing when voice data is transmitted from the supporter terminal to the operator terminal in the two-way communication system.

[0047] Figure 14 This is a second diagram showing an example of a system configuration of a bidirectional communication system.

[0048] Figure 15 This is a diagram showing an example of using text data.

[0049] Figure 16 The third diagram shows an example of the details of the functional configuration of the server device. DETAILED DESCRIPTION

[0050] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In addition, in this specification and the accompanying drawings, components having substantially the same functional structure are denoted by the same reference numerals, and repeated descriptions are omitted.

[0051] [First embodiment]

[0052] <System Structure of Bidirectional Communication System>

[0053] First, a system configuration of a bidirectional communication system as an example of a system according to the first embodiment will be described. Figure 1 FIG1 is a first diagram showing an example of a system configuration of a two-way communication system. Figure 1 As shown, the two-way communication system 100 includes a server device 110 and client terminals (an operator terminal 120 and a supporter terminal 130 ).

[0054] In addition, in this embodiment, the description

[0055] 150 workers working at the work site and

[0056] A supporter 160 who remotely supports the operator 150 by giving various work instructions to the operator 150 from a remote location

[0057] The situation of communicating using the two-way communication system 100. Therefore, as described above, in this embodiment, the client terminals constituting the two-way communication system 100 include the operator terminal 120 used by the operator 150 and the supporter terminal 130 used by the supporter 160. However, the client terminals constituting the two-way communication system 100 are not limited to the combination of the operator terminal 120 and the supporter terminal 130.

[0058] Server device 110 is a device that provides a cloud service (two-way communication service) via network 140. Specifically, a two-way communication program is installed in server device 110. By executing this program, server device 110 functions as a two-way communication unit 111 and a text data management unit 112. Thus, server device 110 provides two-way communication services to operators 150 and supporters 160.

[0059] Specifically, in the server device 110, the two-way communication unit 111 is connected to the operator terminal 120 and the supporter terminal 130 for communication.

[0060] Sending the moving image data and audio data received from the operator terminal 120 to the supporter terminal 130,

[0061] Sending a display instruction based on the detection result (described in detail later) detected in the operator terminal 120 to the supporter terminal 130,

[0062] Sending the audio data received from the supporter terminal 130 to the operator terminal 120,

[0063] A display instruction based on the detection result (described in detail later) detected in the supporter terminal 130 is transmitted to the supporter terminal 130 .

[0064] In addition, the two-way communication unit 111 determines based on the detection result detected in the operator terminal 120.

[0065] Whether the voice data based on the speech of the operator 150 obtained in the operator terminal 120 can be converted into text data; or

[0066] Whether the quality of text data obtained by converting the speech data based on the speech of the operator 150 obtained in the operator terminal 120 has deteriorated,

[0067] Then, a display instruction according to the determination result is transmitted to the supporter terminal 130 .

[0068] Similarly, the two-way communication unit 111 determines based on the detection result detected in the supporter terminal 130

[0069] Whether the voice data based on the speech of the supporter 160 obtained in the supporter terminal 130 can be converted into text data, or

[0070] Whether the quality of text data obtained by converting the voice data based on the speech of the supporter 160 obtained in the supporter terminal 130 has deteriorated,

[0071] Then, a display instruction according to the determination result is transmitted to the supporter terminal 130 .

[0072] In the server device 110 , the text data management unit 112 converts voice data included in audio data bidirectionally transmitted and received between the operator terminal 120 and the supporter terminal 130 into text data in real time and stores the converted data.

[0073] The worker terminal 120 is, for example, a wearable terminal worn at a work site by the worker 150. In the present embodiment, it is assumed that the work site is in an environment where the communication quality of bidirectional communication is likely to deteriorate.

[0074] The worker terminal 120 has the function of capturing a moving image of the work site and detecting sound including voice (e.g., voice based on the speech of the worker 150). Furthermore, the worker terminal 120 has the function of transmitting moving image data obtained by capturing the work site and sound data including voice data based on the speech of the worker 150 to the supporter terminal 130 via the network 140 and the server device 110.

[0075] In addition, the operator terminal 120 detects

[0076] ·Converting the speech data based on the speech of the operator 150 into text data, or

[0077] The quality of the transformed text data

[0078] The detection results of the “ambient sound state”, “microphone state” and “communication state” that cause the impact are reported to the operator 150 , and the detection results of the “ambient sound state” and “microphone state” are sent to the server device 110 .

[0079] Furthermore, the operator terminal 120 has a function of receiving and outputting audio data including voice data based on the speech of the supporter 160 .

[0080] The supporter terminal 130 is installed in, for example, an office where the supporter 160 resides, and is a terminal operated by the supporter 160 .

[0081] The supporter terminal 130 has a function of detecting sound including speech (e.g., speech from the supporter 160 ). Furthermore, the supporter terminal 130 has a function of transmitting sound data including speech data from the supporter 160 to the operator terminal 120 via the network 140 and the server device 110 .

[0082] In addition, the supporter terminal 130 detects

[0083] ·Converting voice data based on the speech of the supporter 160 into text data, or

[0084] The quality of the transformed text data

[0085] The “state of the ambient sound”, “state of the microphone” and “state of communication” that cause an impact, and the detection results of the “state of the ambient sound” and “state of the microphone” are transmitted to the server device 110 .

[0086] Furthermore, the supporter terminal 130 has a function of receiving and outputting moving image data obtained by capturing the state of the work site and audio data including voice data based on the speech of the worker 150 .

[0087] Furthermore, the supporter terminal 130 has a function of displaying a display screen in accordance with a display instruction transmitted from the server device 110 (a display instruction based on a detection result detected in the operator terminal 120 , a display instruction based on a detection result detected in the supporter terminal 130 ).

[0088] <Hardware Structure of Each Device>

[0089] Next, the hardware configuration of each device (the server device 110 , the operator terminal 120 , and the supporter terminal 130 ) constituting the bidirectional communication system 100 will be described. Figure 2 This is a diagram showing an example of the hardware configuration of a server device, and an example of the hardware configuration of an operator terminal and a supporter terminal.

[0090] like Figure 2 As shown in FIG. 2( a ), the server device 110 includes a processor 201 , a memory 202 , an auxiliary storage device 203 , an operating device 204 , a display device 205 , a communication device 206 , and a driver device 207 . The hardware components of the server device 110 are interconnected via a bus 208 .

[0091] The processor 201 includes various computing devices such as a CPU (Central Processing Unit) and reads various programs (such as a two-way communication program) onto the memory 202 and executes them.

[0092] The memory 202 includes main storage devices such as ROM (Read Only Memory) and RAM (Random Access Memory). The processor 201 and the memory 202 form a so-called computer (also called a "control unit"), which implements various functions by executing various programs read from the memory 202 by the processor 201.

[0093] The auxiliary storage device 203 stores various programs and various data used when the processor 201 executes the various programs. In addition, the auxiliary storage device 203 realizes a text data storage unit 450 described later.

[0094] The operation device 204 is an operation device used by the administrator of the server device 110 to perform various operations. The display device 205 is a display device that displays the processing results of various processes executed by the server device 110.

[0095] The communication device 206 is a communication device for communicating with an external device (for example, the operator terminal 120 and the supporter terminal 130 ) via the network 140 .

[0096] The drive device 207 is a device for setting the storage medium 210. The storage medium 210 mentioned here includes media that store information optically, electrically, or magnetically, such as CD-ROMs, floppy disks, and magneto-optical disks. Alternatively, the storage medium 210 may include semiconductor memory that stores information electrically, such as ROMs and flash memories.

[0097] The various programs installed in the auxiliary storage device 203 are installed by, for example, placing the distributed storage medium 210 in the drive device 207 and having the drive device 207 read the various programs stored in the storage medium 210. Alternatively, the various programs installed in the auxiliary storage device 203 may be downloaded from the network 140 via the communication device 206.

[0098] On the other hand, Figure 2 As shown in (b), the operator terminal 120 and the supporter terminal 130 include a processor 221, a memory 222, an auxiliary storage device 223, and a communication device 224. Furthermore, the operator terminal 120 includes a GPS (Global Positioning System) device 225, an acceleration sensor 226, and a camera 229. Furthermore, the operator terminal 120 and the supporter terminal 130 include a voice acquisition device 227, a voice output device 228, and an operating device 230. Furthermore, the supporter terminal 130 includes a display device 231 and a driver device 232. The hardware components of the operator terminal 120 and the supporter terminal 130 are interconnected via a bus 233.

[0099] The processor 221 includes various computing devices such as a CPU (Central Processing Unit) and reads various programs (for example, a client program described later) onto the memory 222 and executes them.

[0100] The memory 222 includes main storage devices such as ROM (Read Only Memory) and RAM (Random Access Memory). The processor 221 and the memory 222 form a so-called computer (also called a "control unit"), which realizes various functions by executing various programs read from the memory 222 by the processor 221.

[0101] The auxiliary storage device 223 stores various programs and various information used when the processor 221 executes the various programs.

[0102] The communication device 224 is a communication device used to transmit and receive various data (e.g., moving image data, audio data, etc.) to and from the server device 110. The GPS device 225 detects the location of the operator terminal 120 and outputs location data. The acceleration sensor 226 detects the movements of the operator 150 wearing the operator terminal 120 and outputs acceleration data.

[0103] The voice acquisition device 227 detects voices spoken by the worker 150 and the supporter 160 , ambient sounds around the work site, ambient sounds around the office, and the like, and outputs audio data.

[0104] The voice output device 228 is a device that outputs various data received from the server device 110 by voice to the worker 150 wearing the worker terminal 120 or the supporter 160 operating the supporter terminal 130 , for example.

[0105] The imaging device 229 captures the surroundings (work site) of the worker 150 wearing the worker terminal 120 and generates moving image data.

[0106] The operating device 230 accepts simple operations on the operator terminal 120 by the operator 150 , such as turning on / off the power switch of the operator terminal 120 . Alternatively, the operating device 230 accepts various operations on the supporter terminal 130 by the supporter 160 .

[0107] The display device 231 is a device that displays the moving image data received from the server device 110 and display contents according to the display instruction to the supporter 160 .

[0108] The drive device 232 is a device for setting the storage medium 240. The storage medium 240 mentioned here is the same as the storage medium 210 mentioned above.

[0109] In the case of the supporter terminal 130, various programs installed in the auxiliary storage device 223 are installed by, for example, placing the distributed storage medium 240 in the drive device 232 and reading the programs from the drive device 232. Alternatively, various programs installed in the auxiliary storage device 223 may be installed by downloading them from the network 140 via the communication device 224.

[0110] <Functional structure of the operator terminal>

[0111] Next, the functional configuration of the operator terminal 120 will be described. Figure 3 1 is a first diagram showing an example of the functional structure of the operator terminal. As described above, a client program is installed in the operator terminal 120. And, by executing the program, the operator terminal 120 can Figure 3As shown in FIG, the sound data acquisition unit 310, the speech separation unit 320, the sound pressure level calculation unit 330, the first detection unit 340, the second detection unit 350, the communication unit 360, the determination unit 370, and the speech output unit 380 function. Figure 3 In the example of , functions related to the processing of dynamic image data are omitted for simplicity of description, and only functions related to the processing of audio data are shown (the processing of audio data will be described below).

[0112] The audio data acquisition unit 310 acquires audio data detected by the audio acquisition device 227 and notifies the audio data to the audio separation unit 320 and the communication unit 360 .

[0113] The speech separation unit 320 separates the speech data notified by the speech data acquisition unit 310 into speech data based on the speech of the operator 150 and ambient sound data (sound data other than speech data) based on the ambient sound around the speech acquisition device 227. Furthermore, the speech separation unit 320 notifies the sound pressure level calculation unit 330 of the separated speech data and ambient sound data.

[0114] The sound pressure level calculation unit 330 calculates the magnitude (sound pressure level) of the speech data notified from the speech separation unit 320 and notifies the second detection unit 350. Furthermore, the sound pressure level calculation unit 330 calculates the magnitude (sound pressure level) of the ambient sound data notified from the speech separation unit 320 and notifies the first detection unit 340 and the second detection unit 350.

[0115] First detector 340 detects the state of ambient sound by determining whether the sound pressure level of the ambient sound data notified from sound pressure level calculator 330 is greater than or equal to a predetermined threshold. Specifically, first detector 340 detects the presence of the influence of ambient sound if it determines that the sound pressure level of the ambient sound data is greater than or equal to the predetermined threshold.

[0116] On the other hand, if the sound pressure level of the ambient sound data is determined to be less than a predetermined threshold, first detection unit 340 detects that there is no influence of the ambient sound. Furthermore, first detection unit 340 notifies communication unit 360 of the state of the ambient sound (influence of the ambient sound or no influence of the ambient sound) as a detection result.

[0117] The second detection unit 350 detects the status of the voice acquisition device 227 (microphone status) by determining whether the sound pressure level of the ambient sound data and the sound pressure level of the voice data notified from the sound pressure level calculation unit 330 are both less than a predetermined threshold. Specifically, if the second detection unit 350 determines that both are less than the predetermined threshold, it detects that the voice acquisition device 227 has malfunctioned (microphone abnormality). On the other hand, if the second detection unit 350 determines that either sound pressure level is greater than the predetermined threshold, it detects that the voice acquisition device 227 has not malfunctioned (microphone normality). Furthermore, the second detection unit 350 notifies the communication unit 360 of the microphone status (microphone abnormality, microphone normality) as the detection result.

[0118] The communication unit 360 transmits the sound data notified from the sound data acquisition unit 310 to the server device 110 , and transmits the state of the ambient sound (affected by the ambient sound, not affected by the ambient sound) and the state of the microphone (microphone abnormal, microphone normal) as detection results to the server device 110 .

[0119] Furthermore, the communication unit 360 detects the state of communication between the operator terminal 120 and the server device 110. Specifically, the communication unit 360 detects that communication has deteriorated if the communication connection between the operator terminal 120 and the server device 110 is disconnected (if no audio data is transmitted from the server device 110 for a certain period of time). Furthermore, the communication unit 360 detects that communication is normal if the communication connection between the operator terminal 120 and the server device 110 is maintained (if audio data is transmitted from the server device 110).

[0120] In addition, the communication unit 360 notifies the determination unit 370 of the state of ambient sound (affected by ambient sound, not affected by ambient sound), the state of the microphone (microphone abnormal, microphone normal), and the state of communication (communication deteriorated, communication normal) as detection results.

[0121] Furthermore, the communication unit 360 receives the audio data from the server device 110 and notifies the audio output unit 380 .

[0122] The determination unit 370 determines based on the detection result notified from the communication unit 360.

[0123] Whether the voice data based on the speech of the operator 150 obtained in the operator terminal 120 can be converted into text data; or

[0124] Whether the quality of text data obtained by converting the voice data based on the speech of the worker 150 , which is currently being acquired in the worker terminal 120 , has deteriorated.

[0125] Specifically, when notified of the influence of ambient sound, the determination unit 370 determines that the voice data based on the speech of the operator 150 cannot be converted into text data due to the influence of the ambient sound, or that the quality of the converted text data is degraded.

[0126] Furthermore, when notified of a microphone abnormality, the determination unit 370 determines that the voice data based on the speech of the operator 150 cannot be converted into text data due to a malfunction of the voice acquisition device 227 or that the quality of the converted text data is degraded.

[0127] Furthermore, when notified of communication deterioration, the determination unit 370 determines that the voice data based on the speech of the operator 150 cannot be converted into text data due to the communication deterioration, or that the quality of the converted text data has deteriorated.

[0128] Furthermore, when notified that there is no influence of ambient sound, the microphone is functioning properly, and communication is normal, the determination unit 370 determines that the voice data based on the speech of the operator 150 can be converted into text data or that the quality of the converted text data is not degraded.

[0129] The voice output unit 380 outputs the audio data notified by the communication unit 360 to the operator 150 via the voice output device 228. Furthermore, the voice output unit 380 reports the determination result notified by the determination unit 370 to the operator 150 via the voice output device 228. This allows the operator 150 to understand the occurrence of a situation where voice recognition of the operator's utterance is not suitable, and the cause thereof. As a result, the operator 150 can immediately take appropriate measures to address the cause, thereby avoiding situations where it is later discovered that a portion of the utterance was not properly converted into text data.

[0130] <Details of the functional configuration of the server device>

[0131] Next, the functional configuration of the server device 110 will be described in detail. As described above, the server device 110 has a two-way communication program installed therein, and by executing the program, the server device 110 functions as the two-way communication unit 111 and the text data management unit 112 .

[0132] Figure 4 1 is a first diagram showing an example of the details of the functional configuration of the server device. Figure 4 As shown, the bidirectional communication unit 111 further includes a communication unit 410 , a voice data acquisition unit 420 , and a determination unit 430 . Furthermore, the text data management unit 112 further includes a text data generation unit 440 .

[0133] The communication unit 410 receives audio data sent from the operator terminal 120, sends it to the supporter terminal 130, and notifies the audio data acquisition unit 420. In addition, the communication unit 410 receives audio data sent from the supporter terminal 130, sends it to the operator terminal 120, and notifies the audio data acquisition unit 420.

[0134] Furthermore, the communication unit 410 receives the detection results (ambient sound status, microphone status) transmitted from the operator terminal 120 and the detection results (ambient sound status, microphone status) transmitted from the supporter terminal 130 , and notifies the determination unit 430 of the results.

[0135] Furthermore, the communication unit 410 detects the communication status between the operator terminal 120 and the server device 110. Specifically, the communication unit 360 detects communication deterioration when the communication connection between the operator terminal 120 and the server device 110 is disconnected (when no audio data is transmitted from the operator terminal 120 for a certain period of time). Furthermore, the communication unit 360 detects normal communication when the communication connection between the operator terminal 120 and the server device 110 is maintained (when audio data is transmitted from the operator terminal 120). Furthermore, the communication unit 410 notifies the determination unit 430 of the communication status (communication deterioration, communication normal) as a detection result.

[0136] Similarly, the communication unit 410 detects the communication status between the supporter terminal 130 and the server device 110. Specifically, the communication unit 410 detects communication deterioration when the communication connection between the supporter terminal 130 and the server device 110 is disconnected (when no audio data is transmitted from the supporter terminal 130 for a certain period of time). Furthermore, the communication unit 410 detects normal communication when the communication connection between the supporter terminal 130 and the server device 110 is maintained. Furthermore, the communication unit 410 notifies the determination unit 430 of the communication status (communication deterioration, communication normal) as a detection result.

[0137] Furthermore, the communication unit 410 transmits a display instruction including the display content (the status of the ambient sound, the status of the microphone, and the status of the communication) notified by the determination unit 430 in response to the notification of the detection result (the status of the ambient sound, the status of the microphone, and the status of the communication) to the supporter terminal 130. Furthermore, the communication unit 410 transmits a display instruction including the display content (the status of the ambient sound, the status of the microphone, and the status of the communication) notified by the determination unit 430 in response to the notification of the detection result (the status of the ambient sound, the status of the microphone, and the status of the communication) to the supporter terminal 130.

[0138] The voice data acquisition unit 420 acquires voice data from the sound data notified by the communication unit 410 , and notifies the text data generation unit 440 of the text data management unit 112 .

[0139] The determination unit 430 determines based on the detection result (ambient sound state, microphone state, communication state) of the operator terminal 120 notified from the communication unit 410.

[0140] Whether the voice data based on the speech of the operator 150 obtained in the operator terminal 120 can be converted into text data; or

[0141] Whether the quality of text data obtained by converting the speech data based on the speech of the operator 150 obtained in the operator terminal 120 has deteriorated,

[0142] The display content (ambient sound status, microphone status, communication status) is determined according to the judgment result.

[0143] Specifically, upon receiving notification of the influence of ambient sound, determination unit 430 determines that the voice data based on the speech of operator 150 cannot be converted into text data due to the influence of the ambient sound, or that the quality of the converted text data has deteriorated. Determination unit 430 then determines display content (the state of the ambient sound) corresponding to the determination result.

[0144] Furthermore, upon receiving notification of a microphone abnormality, determination unit 430 determines that voice data based on the speech of operator 150 cannot be converted into text data, or that the quality of the converted text data has deteriorated, due to a malfunction of voice acquisition device 227. Determination unit 430 then determines display content (microphone status) corresponding to the determination result.

[0145] Furthermore, upon receiving notification of communication deterioration, determination unit 430 determines that the voice data based on operator 150's speech cannot be converted into text data due to the communication deterioration, or that the quality of the converted text data has deteriorated. Determination unit 430 then determines display content (communication status) corresponding to the determination result.

[0146] Furthermore, upon receiving notification that there is no ambient sound influence, the microphone is functioning properly, and the communication status is normal, determination unit 430 determines that the voice data based on the speech of operator 150 can be converted into text data or that the quality of the converted text data is not degraded. Determination unit 430 then determines display content (ambient sound status, microphone status, communication status) corresponding to the determination result.

[0147] Similarly, the determination unit 430 determines based on the detection result (ambient sound state, microphone state, communication state) of the supporter terminal 130 notified from the communication unit 410.

[0148] Whether the voice data based on the speech of the supporter 160 obtained in the supporter terminal 130 can be converted into text data; or

[0149] Whether the quality of text data obtained by converting the voice data based on the speech of the supporter 160 obtained in the supporter terminal 130 has deteriorated,

[0150] The display content (ambient sound status, microphone status) is determined according to the judgment result.

[0151] Specifically, upon receiving notification of the influence of ambient sound, determination unit 430 determines that the voice data based on the speech of supporter 160 cannot be converted into text data due to the influence of the ambient sound, or that the quality of the converted text data has deteriorated. Determination unit 430 then determines display content (the state of the ambient sound) corresponding to the determination result.

[0152] Furthermore, upon receiving notification of a microphone abnormality, determination unit 430 determines that voice data based on the voice of supporter 160 cannot be converted into text data, or that the quality of the converted text data has deteriorated, due to a malfunction in voice acquisition device 227. Determination unit 430 then determines display content (microphone status) corresponding to the determination result.

[0153] Furthermore, when notified of communication deterioration, the determination unit 430 determines that the voice data based on the speech of the operator 150 cannot be converted into text data due to the communication deterioration, or that the quality of the converted text data has deteriorated.

[0154] Furthermore, upon receiving notification that there is no ambient sound influence, the microphone is functioning properly, and communication is normal, determination unit 430 determines that the voice data based on the speech of supporter 160 can be converted into text data or that the quality of the converted text data is not degraded. Determination unit 430 then determines display content (ambient sound status, microphone status) corresponding to the determination result.

[0155] Furthermore, the determination unit 430 stores the determination result in the text data storage unit 450 in association with text data described later.

[0156] The text data generating unit 440 performs speech recognition on the speech data notified from the speech data acquiring unit 420 and converts the speech data into text data. The text data generating unit 440 stores the converted text data in the text data storage unit 450 in association with the determination result.

[0157] <Functional Structure of Supporter Terminal>

[0158] Next, the functional configuration of the supporter terminal 130 will be described. Figure 5 1 is a first diagram showing an example of the functional structure of the supporter terminal. As described above, the supporter terminal 130 has a client program installed. The supporter terminal 130 executes the program, thereby Figure 5 As shown, the supporter terminal 130 functions as a voice data acquisition unit 510, a voice separation unit 520, a sound pressure level calculation unit 530, a first detection unit 540, a second detection unit 550, and a communication unit 560. Furthermore, the supporter terminal 130 functions as a voice output unit 570, a display control unit 580, and a determination unit 590.

[0159] The audio data acquisition unit 510 acquires audio data detected by the audio acquisition device 227 and notifies the audio data to the audio separation unit 520 and the communication unit 560 .

[0160] The speech separation unit 520 separates the speech data notified from the speech data acquisition unit 510 into speech data based on the speech of the supporter 160 and ambient sound data (sound data other than speech data) based on the ambient sound around the speech acquisition device 227. Furthermore, the speech separation unit 520 notifies the sound pressure level calculation unit 530 of the separated speech data and ambient sound data.

[0161] The sound pressure level calculator 530 calculates the sound pressure level of the speech data notified from the speech separator 520 and notifies the second detector 550. Furthermore, the sound pressure level calculator 530 calculates the sound pressure level of the ambient sound data notified from the speech separator 520 and notifies the first detector 540 and the second detector 550.

[0162] First detector 540 detects the state of ambient sound by determining whether the sound pressure level of the ambient sound data notified from sound pressure level calculator 530 is greater than or equal to a predetermined threshold. Specifically, first detector 540 detects the presence of ambient sound influence if it determines that the sound pressure level of the ambient sound data is greater than or equal to the predetermined threshold.

[0163] On the other hand, if the sound pressure level of the ambient sound data is determined to be less than a predetermined threshold, first detection unit 540 detects that there is no influence of the ambient sound. Furthermore, first detection unit 540 notifies communication unit 560 of the state of the ambient sound (influence of the ambient sound or no influence of the ambient sound) as a detection result.

[0164] The second detection unit 550 detects the status of the voice acquisition device 227 (microphone status) by determining whether the sound pressure level of the ambient sound data and the sound pressure level of the voice data notified from the sound pressure level calculation unit 530 are both less than a predetermined threshold. Specifically, if the second detection unit 550 determines that both are less than the predetermined threshold, it detects that the voice acquisition device 227 has malfunctioned (microphone abnormality). On the other hand, if the second detection unit 550 determines that either sound pressure level is greater than the predetermined threshold, it detects that the voice acquisition device 227 has not malfunctioned (microphone normality). Furthermore, the second detection unit 550 notifies the communication unit 560 of the microphone status (microphone abnormality, microphone normality) as the detection result.

[0165] The communication unit 560 transmits the sound data notified from the sound data acquisition unit 510 to the server device 110 , and transmits the state of the ambient sound (affected by the ambient sound, not affected by the ambient sound) and the state of the microphone (microphone abnormal, microphone normal) as detection results to the server device 110 .

[0166] Furthermore, the communication unit 560 detects the state of communication between the supporter terminal 130 and the server device 110. Specifically, the communication unit 560 detects that communication has deteriorated if the communication connection between the supporter terminal 130 and the server device 110 is disconnected (if no audio data is transmitted from the server device 110 for a certain period of time). Furthermore, the communication unit 560 detects that communication is normal if the communication connection between the supporter terminal 130 and the server device 110 is maintained (if audio data is transmitted from the server device 110).

[0167] Furthermore, the communication unit 560 notifies the determination unit 590 of the communication status (communication deterioration, communication normal) as a detection result.

[0168] Furthermore, the communication unit 560 receives audio data from the server device 110 and notifies the audio output unit 570 of the received audio data.

[0169] Furthermore, the communication unit 560 receives a display instruction from the server device 110 and notifies the display control unit 580. The display instruction received by the communication unit 560 includes:

[0170] Display instructions corresponding to the detection results (ambient sound status, microphone status, communication status) of the operator terminal 120; and

[0171] Display instructions corresponding to the detection results (ambient sound status, microphone status) of the supporter terminal 130.

[0172] The audio output unit 570 outputs the audio data notified from the communication unit 560 to the supporter 160 via the audio output device 228 .

[0173] The determination unit 590 determines whether the communication state (communication deterioration, communication normal) is notified from the communication unit 410 as a detection result.

[0174] Whether the voice data based on the speech of the operator 150 obtained in the operator terminal 120 can be converted into text data; or

[0175] Whether the quality of text data obtained by converting the speech data based on the speech of the operator 150 obtained in the operator terminal 120 has deteriorated,

[0176] The display content (communication status) is determined according to the determination result. Furthermore, the determination unit 590 notifies the display control unit 580 of a display instruction including the determined display content (communication status).

[0177] The display control unit 580 displays the display content included in the display instruction notified from the communication unit 560 and the display instruction notified from the determination unit 590 on the display device 231 of the supporter terminal 130 .

[0178] <Communication processing flow (1)>

[0179] Next, the flow of the communication process of the two-way communication system 100 , that is, the communication process when audio data including voice data based on the speech of the worker 150 is transmitted from the worker terminal 120 to the supporter terminal 130 will be described. Figure 6 This is a first sequence diagram showing the flow of communication processing when voice data is transmitted from the operator terminal to the supporter terminal in the two-way communication system.

[0180] In step S601, the operator terminal 120 acquires audio data including voice data based on the speech of the operator 150. The operator terminal 120 transmits the acquired audio data to the server device 110.

[0181] In step S610, the server device 110 performs bidirectional communication processing. This bidirectional communication processing by the server device 110 includes receiving and transmitting audio data, voice recognition processing of voice data, detecting the communication status, obtaining and determining the detection results, determining display content and instructing display, and storing text data and determination results. Here, the server device 110 receives audio data and transmits it to the supporter terminal 130. Furthermore, the server device 110 performs voice recognition processing on the voice data and converts it into text data.

[0182] In step S620, the supporter terminal 130 performs a voice output process. The voice output process of the supporter terminal 130 includes acquisition and output of audio data, and display processing according to a display instruction. Here, the supporter terminal 130 acquires and outputs audio data.

[0183] In step S602 , the operator terminal 120 separates the audio data into voice data and ambient sound data.

[0184] In step S603 , the operator terminal 120 calculates the sound pressure level of the voice data and the sound pressure level of the ambient sound data.

[0185] In step S604 , the operator terminal 120 detects the state of the ambient sound based on the sound pressure level of the ambient sound data and transmits the detection result (the state of the ambient sound) to the server device 110 .

[0186] In step S610, the server device 110 performs a two-way communication process. Here, the server device 110 receives the detection result (ambient sound status), makes a determination based on the received detection result (ambient sound status), determines display content (ambient sound status), and transmits a display instruction corresponding to the determined display content to the supporter terminal 130.

[0187] Furthermore, if the server device 110 receives a detection result indicating no influence of ambient sound, it determines that the voice data based on the worker 150's speech can be converted into text data, or that the quality of the converted text data is not degraded. In this case, the server device 110 instructs the supporter terminal 130 to display an icon in blue indicating the state of ambient sound, for example.

[0188] Furthermore, upon receiving a detection result indicating the presence of ambient sound, the server device 110 determines that the audio data based on the worker's 150 speech cannot be converted into text data due to the ambient sound, or that the quality of the converted text data has deteriorated. In this case, the server device 110 instructs the supporter terminal 130 to display an icon indicating the ambient sound status in gray, for example.

[0189] In step S620, the supporter terminal 130 performs audio output processing. Here, the supporter terminal 130 performs display processing corresponding to the display instruction. For example, the supporter terminal 130 may display an icon in blue indicating the ambient sound status of the operator terminal 120. Alternatively, the supporter terminal 130 may display an icon in gray indicating the ambient sound status of the operator terminal 120.

[0190] In step S605, if the detection result in step S604 is "affected by ambient sound," the operator terminal 120 reports the determination result to the operator 150. Specifically, the operator terminal 120 reports the determination result to the operator 150 that the speech data based on the operator's 150 speech cannot be converted into text data due to the influence of ambient sound, or that the quality of the converted text data has deteriorated.

[0191] In step S606 , the operator terminal 120 detects the state of the microphone based on the sound pressure level of the ambient sound data and the sound pressure level of the voice data, and transmits the detection result (microphone state) to the server device 110 .

[0192] In step S610, the server device 110 performs a two-way communication process. Here, the server device 110 receives the detection result (microphone status), makes a judgment based on the received detection result (microphone status), determines the display content (microphone status), and transmits a display instruction corresponding to the determined display content to the supporter terminal 130.

[0193] Furthermore, upon receiving a normal microphone detection result, the server device 110 determines that the voice data based on the worker 150's speech can be converted into text data or that the quality of the converted text data is not degraded. In this case, the server device 110 instructs the supporter terminal 130 to display a black icon indicating the microphone status, for example.

[0194] Furthermore, upon receiving a microphone abnormality detection result, the server device 110 determines that the voice data based on the worker 150's speech cannot be converted into text data, or that the quality of the converted text data has deteriorated, due to a malfunction in the voice acquisition device 227. In this case, the server device 110 instructs the supporter terminal 130 to display an icon indicating the microphone status in gray, for example.

[0195] In step S620, the supporter terminal 130 performs voice output processing. Here, the supporter terminal 130 performs display processing corresponding to the display instruction. For example, the supporter terminal 130 displays an icon indicating the microphone status of the operator terminal 120 in black. Alternatively, the supporter terminal 130 displays an icon indicating the microphone status of the operator terminal 120 in gray.

[0196] In step S607, if the detection result in step S606 is "microphone abnormality," the operator terminal 120 reports the determination result to the operator 150. Specifically, the operator terminal 120 reports the determination result to the operator 150 that the speech data based on the operator's 150 speech cannot be converted into text data due to a malfunction of the speech acquisition device 227, or that the quality of the converted text data has deteriorated.

[0197] In step S608, operator terminal 120 detects the communication status. Furthermore, in step S609, if the detection result in step S608 is "communication deterioration," operator terminal 120 reports the determination result to operator 150. Specifically, operator terminal 120 reports the determination result to operator 150 that the speech data based on operator 150's speech cannot be converted into text data due to communication deterioration, or that the quality of the converted text data has deteriorated.

[0198] In step S610, the server device 110 performs a two-way communication process. Here, the server device 110 detects the communication status, makes a determination based on the detection result (communication status), determines display content (communication status), and transmits a display instruction corresponding to the determined display content to the supporter terminal 130.

[0199] Furthermore, if the server device 110 detects communication deterioration and determines that the voice data based on the operator 150's speech cannot be converted into text data due to the communication deterioration, or that the quality of the converted text data has deteriorated, the server device 110 instructs the supporter terminal 130 to display an icon indicating the communication status, for example.

[0200] In step S620 , the supporter terminal 130 performs a voice output process. Here, the supporter terminal 130 performs a display process corresponding to the display instruction. For example, an icon indicating the communication status between the operator terminal 120 and the server device 110 is displayed.

[0201] In step S610, server device 110 executes a two-way communication process. Here, server device 110 stores the converted text data in association with the determination result.

[0202] in addition, Figure 6 The communication process shown is repeatedly executed while the operator 150 and the supporter 160 continue to communicate using the two-way communication system 100 .

[0203] <Communication processing flow (2)>

[0204] Next, the flow of communication processing by the two-way communication system 100 , that is, when audio data including voice data based on the speech of the supporter 160 is transmitted from the supporter terminal 130 to the operator terminal 120 , will be described. Figure 7 This is a first sequence diagram showing the flow of communication processing when voice data is transmitted from a supporter terminal to a worker terminal in a two-way communication system.

[0205] In step S701 , the supporter terminal 130 acquires audio data including voice data based on the speech of the supporter 160 . The supporter terminal 130 transmits the acquired audio data to the server device 110 .

[0206] In step S710, the server device 110 performs bidirectional communication processing. This bidirectional communication processing by the server device 110 includes receiving and transmitting audio data, voice recognition processing of voice data, detecting the communication status, obtaining and determining the detection results, determining display content and instructing display, and storing text data and determination results. Here, the server device 110 receives audio data and transmits it to the operator terminal 120. Furthermore, the server device 110 performs voice recognition processing on the voice data, converting it into text data.

[0207] In step S720, the operator terminal 120 performs a voice output process. The voice output process of the operator terminal 120 includes acquisition and output of audio data. Here, the operator terminal 120 acquires and outputs audio data.

[0208] In step S702 , the supporter terminal 130 separates the audio data into voice data and ambient sound data.

[0209] In step S703 , the supporter terminal 130 calculates the sound pressure level of the voice data and the sound pressure level of the ambient sound data.

[0210] In step S704 , the supporter terminal 130 detects the state of the ambient sound based on the sound pressure level of the ambient sound data and transmits the detection result (the state of the ambient sound) to the server device 110 .

[0211] In step S710, the server device 110 performs a two-way communication process. Here, the server device 110 receives the detection result (ambient sound status), makes a determination based on the received detection result (ambient sound status), determines display content (ambient sound status), and transmits a display instruction corresponding to the determined display content to the supporter terminal 130.

[0212] Furthermore, if the server device 110 receives a detection result indicating no influence of ambient sound, it determines that the voice data based on the speech of the supporter 160 can be converted into text data, or that the quality of the converted text data is not degraded. In this case, the server device 110 instructs the supporter terminal 130 to display an icon in blue indicating the state of ambient sound, for example.

[0213] Furthermore, upon receiving a detection result indicating the presence of ambient sound, the server device 110 determines that the audio data based on the supporter's 160 speech cannot be converted into text data due to the ambient sound, or that the quality of the converted text data has deteriorated. In this case, the server device 110 instructs the supporter terminal 130 to display an icon indicating the ambient sound status in gray, for example.

[0214] In step S705, the supporter terminal 130 performs display processing in accordance with the display instruction from the server device 110. For example, the supporter terminal 130 displays an icon in blue that clearly indicates the state of the ambient sound of the supporter terminal 130. Alternatively, the supporter terminal 130 displays an icon in gray that clearly indicates the state of the ambient sound of the supporter terminal 130.

[0215] In step S706 , the supporter terminal 130 detects the status of the microphone based on the sound pressure level of the ambient sound data and the sound pressure level of the voice data, and transmits the detection result (microphone status) to the server device 110 .

[0216] In step S710, the server device 110 performs a two-way communication process. Here, the server device 110 receives the detection result (microphone status), makes a determination based on the received detection result (microphone status), determines display content (microphone status), and transmits a display instruction corresponding to the determined display content to the supporter terminal 130.

[0217] Furthermore, upon receiving a normal microphone detection result, the server device 110 determines that the voice data based on the voice of the supporter 160 can be converted into text data or that the quality of the converted text data is not degraded. In this case, the server device 110 instructs the supporter terminal 130 to display a black icon indicating the microphone status, for example.

[0218] Furthermore, upon receiving a microphone abnormality detection result, the server device 110 determines that the voice data based on the supporter's 160 speech cannot be converted into text data, or that the quality of the converted text data has deteriorated, due to a malfunction in the voice acquisition device 227. In this case, the server device 110 instructs the supporter terminal 130 to display an icon indicating the microphone status in gray, for example.

[0219] In step S707, the supporter terminal 130 performs display processing in accordance with the display instruction from the server device 110. For example, the supporter terminal 130 displays an icon in black that clearly indicates the status of the microphone of the supporter terminal 130. Alternatively, the supporter terminal 130 displays an icon in gray that clearly indicates the status of the microphone of the supporter terminal 130.

[0220] In step S708 , the supporter terminal 130 detects the status of communication, makes a determination based on the detection result (the status of communication), and determines display content (the status of communication).

[0221] Furthermore, if the supporter terminal 130 detects "communication deterioration," it determines that the voice data based on the supporter's 160 speech cannot be converted into text data due to communication deterioration, or that the quality of the converted text data has deteriorated. In this case, the supporter terminal 130 determines, for example, to display an icon indicating the communication status as the display content (communication status).

[0222] In step S709 , the supporter terminal 130 displays the determined display content. For example, the supporter terminal 130 displays an icon indicating the communication status between the supporter terminal 130 and the server device 110 .

[0223] In step S710, server device 110 performs bidirectional communication processing. Here, server device 110 detects the communication status and makes a determination based on the detection result (communication status). In addition, server device 110 stores the converted text data in association with the determination result.

[0224] in addition, Figure 7 The communication process shown is repeatedly executed while the operator 150 and the supporter 160 continue to communicate using the two-way communication system 100 .

[0225] <Screen example of the supporter terminal>

[0226] Next, an example of a display screen displayed on the display device 231 of the supporter terminal 130 will be described. Figure 8 FIG is a diagram showing an example of a display screen of a supporter terminal. Figure 8 As shown, the display screen 800 includes an area 810 for displaying moving image data transmitted from the operator terminal 120 , an area 820 for displaying the status of the operator terminal 120 , and an area 830 for displaying the status of the supporter terminal 130 .

[0227] Among them, in the area 810 , moving image data captured by the camera 229 of the operator terminal 120 and transmitted by the communication device 224 of the operator terminal 120 during the two-way communication process is displayed.

[0228] In area 820 , “microphone status”, “ambient sound status”, and “communication status” are displayed as statuses of the operator terminal 120 .

[0229] In the "Microphone Status" section, an icon indicating the status of the microphone of the operator terminal 120 is displayed. Figure 8 As shown, when the detection result (microphone status) detected by the operator terminal 120 is "microphone normal", the icon indicating the microphone status is displayed in black. In addition, when the detection result (microphone status) detected by the operator terminal 120 is "microphone abnormal", the icon indicating the microphone status is displayed in gray.

[0230] In the "Ambient Sound Status", an icon indicating the ambient sound status of the operator terminal 120 is displayed. Figure 8 As shown, when the detection result (ambient sound status) detected by the operator terminal 120 is "not affected by ambient sound", the icon indicating the ambient sound status is displayed in black. Furthermore, when the detection result (ambient sound status) detected by the operator terminal 120 is "affected by ambient sound", the icon indicating the ambient sound status is displayed in blue.

[0231] In the "communication status", an icon indicating the communication status between the operator terminal 120 and the server device 110 is displayed. Figure 8 As shown, when the detection result (communication status) detected by the server device 110 is "communication normal", the icon is not displayed. On the other hand, when the detection result (communication status) detected by the server device 110 is "communication deteriorated", the icon is displayed.

[0232] During the communication with the operator 150, the supporter 160 can understand the following by referring to the area 820:

[0233] Whether the operator terminal 120 is in a state suitable for performing speech recognition on speech data based on the speech of the operator 150 ;

[0234] Whether a situation unsuitable for voice recognition has occurred in the operator terminal 120 .

[0235] Furthermore, when the supporter 160 detects that the operator terminal 120 is not suitable for voice recognition of the voice data based on the speech of the operator 150, it can identify the cause (a failure of the voice acquisition device 227, the influence of ambient sound, or communication degradation). Therefore, the supporter 160 can urge the operator 150 to improve the situation, thereby avoiding a situation where it is later determined that part of the speech content was not converted into appropriate text data.

[0236] Similarly, in the area 830 , “microphone status”, “ambient sound status”, and “communication status” are displayed as statuses of the supporter terminal 130 .

[0237] In the "Microphone Status" field, an icon indicating the microphone status of the supporter terminal 130 is displayed. Figure 8 As shown, when the detection result (microphone status) detected by the supporter terminal 130 is "microphone normal", the icon indicating the microphone status is displayed in black. On the other hand, when the detection result (microphone status) detected by the supporter terminal 130 is "microphone abnormal", the icon indicating the microphone status is displayed in gray.

[0238] In the "Ambient Sound Status", an icon indicating the ambient sound status of the supporter terminal 130 is displayed. Figure 8 As shown, when the detection result (ambient sound status) detected by the supporter terminal 130 is "not affected by ambient sound", the icon indicating the ambient sound status is displayed in black. Alternatively, when the detection result (ambient sound status) detected by the supporter terminal 130 is "affected by ambient sound", the icon indicating the ambient sound status is displayed in blue.

[0239] In the "communication status", an icon indicating the communication status between the supporter terminal 130 and the server device 110 is displayed. Figure 8 As shown, when the detection result (communication status) detected by the supporter terminal 130 is "communication normal", the icon is not displayed. On the other hand, when the detection result (communication status) detected by the supporter terminal 130 is "communication deteriorated", the icon is displayed.

[0240] During the communication with the operator 150, the supporter 160 can understand the following by referring to the area 830:

[0241] Whether the supporter terminal 130 is in a suitable state for performing speech recognition on the voice data based on the supporter's speech;

[0242] Whether a situation has occurred in the supporter terminal 130 that is not suitable for voice recognition of the supporter's own speech.

[0243] Furthermore, when the supporter 160 detects that speech data based on their own speech is not suitable for speech recognition on the supporter terminal 130, they can identify the cause (a failure of the speech acquisition device 227, the influence of ambient sound, or communication degradation). Therefore, the supporter 160 can immediately correct the situation and avoid a situation where it is later determined that part of the speech content was not converted into appropriate text data.

[0244] Summary

[0245] As can be seen from the above description, the bidirectional communication system 100 as an example of the system according to the first embodiment

[0246] During the acquisition of voice data, the sound pressure data of the acquired voice data and the sound pressure data of ambient sound data other than the voice data are calculated. This determines whether the acquired voice data can be converted into text data or whether the quality of the converted text data is degraded.

[0247] The judgment result is reported to the operator, or the display content corresponding to the judgment result is displayed on the supporter terminal.

[0248] Thus, according to the two-way communication system 100 , an operator or a supporter can recognize that a situation unsuitable for voice recognition has occurred.

[0249] [Second embodiment]

[0250] In the first embodiment described above, the process of converting voice data into text data is simultaneously performed in the server device 110. However, the process of converting voice data into text data may also be performed in each client terminal, such as the operator terminal 120 and the supporter terminal 130. The following describes the second embodiment, focusing on the differences from the first embodiment.

[0251] <Functional structure of the operator terminal>

[0252] First, the functional configuration of the operator terminal 120 in the second embodiment will be described. Figure 9 The second figure is an example of the functional structure of the operator terminal. Figure 3 The difference between the functional structure described above is that the functions of the text data generating unit 910 and the communication unit 920 are different from those of the Figure 3 The function of the communication unit 360 is different.

[0253] The text data generating unit 910 performs speech recognition on the speech data notified from the speech separating unit 320 to convert the speech data into text data, and notifies the communication unit 920 of the text data.

[0254] Communication Department 920 Figure 3 In addition to the function of the communication unit 360 , the communication unit 360 also has a function of transmitting the text data notified from the text data generating unit 910 to the server device 110 .

[0255] <Details of the functional configuration of the server device>

[0256] Next, the functional configuration of the server device 110 in the second embodiment will be described in detail. Figure 10 FIG2 is a second diagram showing an example of the details of the functional configuration of the server device. Figure 4 The difference in the details of the functional configuration described is that the function of the communication unit 1010 is different from that of the Figure 4 The function of the communication unit 410 is different, and the voice data acquisition unit 420 and the text data generation unit 440 are not included.

[0257] Communication Department 1010 Figure 4 In addition to the functions of the communication unit 410, it also has the following functions:

[0258] A function of receiving text data transmitted from the operator terminal 120 and storing the received text data in the text data storage unit 450 in association with the determination result determined by the determination unit 430; and

[0259] A function of receiving text data transmitted from the supporter terminal 130 and storing the received text data in the text data storage unit 450 in association with the determination result determined by the determination unit 430 .

[0260] <Functional Structure of Supporter Terminal>

[0261] Next, the functional configuration of the supporter terminal 130 in the second embodiment will be described. Figure 11 FIG2 is a second diagram showing an example of the functional configuration of the supporter terminal. Figure 5 The difference between the functional structure described is that the text data generating unit 1110 and the communication unit 1120 are different from the Figure 5 The function of the communication unit 560 is different.

[0262] The text data generating unit 1110 performs speech recognition on the speech data notified from the speech separating unit 520 to convert the speech data into text data, and notifies the communication unit 1120 .

[0263] Communication Department 1120 Figure 3 In addition to the functions of the communication unit 560 , the communication unit 560 also has a function of transmitting the text data notified from the text data generating unit 1110 to the server device 110 .

[0264] <Communication processing flow (1)>

[0265] Next, the flow of the communication process of the two-way communication system 100 in the second embodiment, that is, the communication process when audio data including voice data based on the speech of the operator 150 is transmitted from the operator terminal 120 to the supporter terminal 130 will be described. Figure 12This is a second sequence diagram showing the flow of communication processing when voice data is transmitted from the operator terminal to the supporter terminal in the two-way communication system. Figure 6 The first sequence diagram described is different in step S1201 and step S1210.

[0266] In step S1201 , the operator terminal 120 performs speech recognition on the speech data and converts it into text data. The operator terminal 120 then transmits the converted text data to the server device 110 .

[0267] In step S1210, server device 110 performs bidirectional communication processing. This bidirectional communication processing by server device 110 includes receiving and sending audio data, receiving text data, detecting communication status, acquiring and determining detection results, determining display content and instructing display, and storing text data and determination results. Here, server device 110 stores the received text data in association with the determination results.

[0268] <Communication processing flow (2)>

[0269] Next, the flow of the communication process of the two-way communication system 100 in the second embodiment, that is, the communication process when audio data including voice data based on the speech of the supporter 160 is transmitted from the supporter terminal 130 to the operator terminal 120 will be described. Figure 13 This is a second sequence diagram showing the flow of communication processing when voice data is transmitted from the supporter terminal to the operator terminal in the two-way communication system. Figure 7 The first sequence diagram described is different in step S1301 and step S1310.

[0270] In step S1301 , the supporter terminal 130 performs speech recognition on the speech data and converts it into text data. The supporter terminal 130 then transmits the converted text data to the server device 110 .

[0271] In step S1310, server device 110 performs bidirectional communication processing. This bidirectional communication processing by server device 110 includes receiving and sending audio data, receiving text data, detecting communication status, obtaining and determining detection results, determining display content and instructing display, and storing text data and determination results. Here, server device 110 stores the received text data in association with the determination results.

[0272] Summary

[0273] As can be seen from the above description, the two-way communication system 100, which is an example of a system according to the second embodiment, has the same functions as those of the first embodiment. Furthermore, the two-way communication system 100, which is an example of a system according to the second embodiment, is configured so that instead of having the server device 110 collectively perform speech recognition processing on speech data, each client terminal performs speech recognition processing on the speech data.

[0274] Thus, according to the second embodiment, the same effects as those of the first embodiment can be achieved, and a situation in which the voice recognition accuracy is reduced due to noise accompanying the transmission of audio data between each client terminal and the server device can be avoided.

[0275] [Third embodiment]

[0276] In the first and second embodiments described above, the following situations are described:

[0277] 150 workers working at the work site and

[0278] The two-way communication system is applied in a scenario where a supporter 160 remotely supports the operator 150 by providing various operational instructions to the operator 150. However, the application of the two-way communication system is not limited to this application; for example, it can be applied to other scenarios as long as remote communication is carried out. In the third embodiment, the application of the two-way communication system in a scenario where remote users hold a meeting will be described.

[0279] <System Structure of Bidirectional Communication System>

[0280] First, a system configuration of a bidirectional communication system as an example of a system according to the third embodiment will be described. Figure 14 FIG2 is a second diagram showing an example of a system configuration of a two-way communication system. Figure 14 As shown, a two-way communication system 1400 includes a server device 110 and client terminals (Web terminals 1420 and 1430).

[0281] In addition, in this embodiment, the following situation is described:

[0282] User 1450 and

[0283] ·User 1460

[0284] A two-way communication system is applied in conference scenarios.

[0285] The server device 110 is the same as the server device 110 described in the first or second embodiment.

[0286] Furthermore, the Web terminals 1420 and 1430 are the same as the supporter terminal 130 described in the first or second embodiment.

[0287] In the third embodiment, the text data storage unit 450 (in Figure 14 The text data (not shown) and the determination results can also be downloaded by the user 1450 or 1460 after the meeting, for example, to be used as a meeting record.

[0288] <Examples of using text data>

[0289] Next, an example of using text data in a two-way communication system 1400 as an example of a system according to the third embodiment will be described. Figure 15 This is a diagram showing an example of using text data. Figure 15 In the illustrated usage example, the text data management unit 112 further includes a text data correction unit 1510 .

[0290] The text data correction unit 1510 corrects the text data stored in the text data storage unit 450 based on the corresponding determination result. The correction functions of the text data correction unit 1510 include:

[0291] A function to visualize a judgment result by adding a mark indicating the cause of a given judgment result (failure of the voice acquisition device 227, influence of ambient sound, communication deterioration) to text data; and

[0292] A function to correct a portion of text data that is related to a given judgment result (failure of the voice acquisition device 227, influence of ambient sound, communication deterioration) using previous and next text data.

[0293] exist Figure 15 In FIG. 1 , reference numeral 1520 represents an example of text data stored in the text data storage unit 450. Figure 15 As shown, the text data stored in the text data storage unit 450 is stored in a manner that allows identification of which of the Web terminal 1420 and the Web terminal 1430 the text data was input through and converted from the voice data.

[0294] exist Figure 9 In FIG. 1 , reference numeral 1530 indicates a state in which the text data correction unit 1510 adds a mark to the region surrounded by a dotted line in the text data indicated by reference numeral 1520 to visualize the determination result.

[0295] Among them, reference numeral 1531 indicates text data when it is determined that the voice data can be converted into text data or the quality of the converted text data is not reduced due to notification of "no influence of ambient sound", "microphone is normal", and "communication is normal".

[0296] On the other hand, reference numeral 1532 shows text data after being marked, in which case it is determined that the voice data cannot be converted into text data due to the influence of ambient sound, or that the quality of the converted text data has deteriorated. As shown by reference numeral 1532, text data correction unit 1510 adds a mark corresponding to the reason to the corresponding position in the text data.

[0297] Similarly, reference numeral 1533 shows text data with a mark added when, upon receiving a notification of "microphone abnormality," it is determined that the voice acquisition device 227 is malfunctioning and that the voice data cannot be converted into text data, or that the quality of the converted text data has deteriorated. As shown by reference numeral 1533, the text data correction unit 1510 adds a mark corresponding to the cause to the corresponding position in the text data.

[0298] Similarly, reference numeral 1534 shows text data that has been marked when, upon receiving notification of "communication deterioration," it is determined that the voice data cannot be converted into text data due to communication deterioration, or that the quality of the converted text data has deteriorated. As shown by reference numeral 1534, text data correction unit 1510 adds a mark corresponding to the cause to the corresponding position in the text data.

[0299] In this manner, in the third embodiment, server device 110 adds a tag corresponding to the reason for the determination result to the text data. Thus, according to the third embodiment, when displaying text data, client terminals (web terminals 1420, 1430) can visualize the determination result associated with the text data. Consequently, according to the third embodiment, even if a portion of the text data fails to undergo proper speech recognition, users 1450, 1450 can understand the reason.

[0300] In the above description, a specific example of the function of visualizing the cause of the judgment result among the correction functions of the text data correction unit 1510 is given. However, a function of correcting the text data before and after use, for example,

[0301] · The text data shown in reference numerals 1532 to 1534 are corrected to

[0302] ·Text data indicated by reference numeral 1531 .

[0303] This is because the determination result is displayed during communication, and thus the time range in which a situation unsuitable for speech recognition occurs can be shortened (it can be brought within the time range in which the text data before and after use can be corrected).

[0304] Summary

[0305] As can be seen from the above description, the bidirectional communication system 1400 as an example of the system according to the third embodiment has the same functions as those of the first or second embodiment described above, and further has the following functions:

[0306] A function to visualize the causes of judgment results associated with text data;

[0307] A function that uses the preceding and following text data to correct areas related to judgment results.

[0308] Thus, according to the third embodiment, the same effects as those of the first or second embodiment can be achieved, and even if part of the text data is not properly voice recognized, the cause can be understood and the text data can be corrected to be appropriate.

[0309] [Fourth embodiment]

[0310] In the first and second embodiments described above, the communication process for transmitting dynamic image data to the supporter terminal 130 is omitted. However, the dynamic image data may be stored in association with text data, for example. In this case, for example, the dynamic image data may be analyzed to determine the content of the operator's 150 work, convert it into text information, and store the determined content of the work in association with the text data.

[0311] Figure 16 FIG3 is a third diagram showing an example of the details of the functional configuration of the server device. Figure 4 The difference from the first figure is that a work content determination unit 1610 is included.

[0312] The work content identification unit 1610 analyzes the moving image data notified from the communication unit 410 to identify the work content performed by the operator 150, thereby converting the moving image data into text information. Furthermore, the work content identification unit 1610 stores the identified work content in the text data storage unit 450 in association with the text data and the determination result.

[0313] By storing the work content in association with the text data and the determination result in this manner, for example, the supporter 160 can understand the work content performed by the worker 150 and confirm the utterances of the worker 150 when browsing the text data.

[0314] [Fifth embodiment]

[0315] In the first and second embodiments described above, the operator terminal 120 reports the determination result to the operator 150 via the voice output device 228. However, the method of reporting the determination result to the operator 150 is not limited to this. For example, the operator terminal 120 may be configured such that an LED or the like is illuminated in a color corresponding to the determination result. Alternatively, the operator terminal 120 may be configured such that a vibrator or the like is vibrated at a frequency or amplitude corresponding to the determination result.

[0316] Furthermore, in the first and second embodiments described above, the server device 110 transmits a display instruction corresponding to the determination result for the operator terminal 120 and a display instruction corresponding to the determination result for the supporter terminal 130 to the supporter terminal 130. However, the instruction transmitted by the server device 110 is not limited to a display instruction corresponding to the determination result, and the transmission destination is not limited to the supporter terminal 130. For example, the server device 110 may also be configured to transmit a notification instruction corresponding to the determination result to the operator terminal 120.

[0317] Furthermore, in the above embodiments, the influence of ambient sound and microphone anomalies are cited as reasons for failure to convert to text data or for reduced quality of the converted text data. However, the reasons for failure to convert to text data or for reduced quality of the converted text data are not limited to the influence of ambient sound and microphone anomalies. For example, the influence of voice data preceding the currently acquired voice data on the currently acquired voice data can be cited. This is because voice data preceding the currently acquired voice data may sometimes also influence the currently acquired voice data due to echo.

[0318] Furthermore, the presence or absence of an influence of speech data preceding the currently acquired speech data on the currently acquired speech data can be determined using the same mechanism as the presence or absence of an influence of ambient sound. For example, the presence or absence of an influence of speech data preceding the currently acquired speech data on the currently acquired speech data can be determined by separating the speech data preceding the currently acquired speech data from the currently acquired speech data and calculating the magnitude (sound pressure level) of the speech data preceding the currently acquired speech data.

[0319] In addition, the display method for displaying the presence or absence of influence on the acquired voice data may be the same as the display method for displaying the presence or absence of influence of the ambient sound, or may be a different display method.

[0320] Furthermore, in each of the above-described embodiments, the state of ambient sound is detected by determining whether the sound pressure level of the ambient sound data notified from the sound pressure level calculation unit 330 is greater than a given threshold (i.e., an absolute value). However, the method for detecting the state of ambient sound is not limited to this. For example, the state of ambient sound can also be detected by determining whether the ratio of the magnitude (sound pressure level) of the voice data to the magnitude (sound pressure level) of the ambient sound data satisfies a given condition (i.e., a relative value). Specifically, if the sound pressure level of the voice data is less than that of the ambient sound data, it can be determined that the ambient sound is affecting the sound, whereas if the sound pressure level of the voice data is greater than that of the ambient sound data, it can be determined that the ambient sound is not affecting the sound.

[0321] Furthermore, while the above embodiments do not describe in detail the methods used by the sound data acquisition units 310 and 510 to acquire sound data, the sound data acquisition units 310 and 510 may also be configured to perform various pre-processing upon acquiring sound data. This pre-processing may include, for example, low-pass filtering to remove high-frequency components from the sound data, or removing hum noise by removing low-frequency components from the sound data. Alternatively, pre-processing may include, for example, analyzing the waveform of each given interval to remove howling or echo.

[0322] Furthermore, in the above-mentioned embodiments, the use of Figure 8 The icons shown are used to display the status of the operator terminal 120 and the status of the supporter terminal 130. However, the display method of the status of the operator terminal 120 and the status of the supporter terminal 130 is not limited to this, and they may be displayed in any other display format.

[0323] Furthermore, in the third embodiment described above, as a mark indicating the reason for a given determination result added to text data, Figure 15 However, the display method of the mark indicating the reason for a given determination result added to the text data is not limited to this, and the mark may be displayed in any other display format.

[0324] Furthermore, in the above embodiments, the client terminal transmits audio data in real time to the server device 110. However, the configuration of the client terminal is not limited to this. For example, the client terminal may be configured to transmit and sequentially store audio data in real time, and upon a request from the server device 110, transmit audio data in a range corresponding to the request to the server device 110.

[0325] Furthermore, in the first and second embodiments described above, the text data generation unit is implemented in the server device 110, while the text data generation unit is implemented in the client terminal (operator terminal 120, supporter terminal 130). However, the functional division between the server device 110 and the client terminal (operator terminal 120, supporter terminal 130) is not limited to this. For example, the determination unit 430 of the server device 110 may also be implemented in the client terminal (operator terminal 120, supporter terminal 130).

[0326] Furthermore, the text data correction unit 1510 described in the third embodiment can also be implemented in the two-way communication system 100 described in the first embodiment. In this case, the supporter terminal 130 can visualize the determination results associated with the text data when downloading and displaying the text data. As a result, according to the fifth embodiment, even if a portion of the text data fails to undergo proper speech recognition, the supporter 160 can understand the cause.

[0327] Furthermore, in the above-mentioned first embodiment, a case where a wearable terminal is used as the operator terminal 120 has been described. However, a portable terminal may be used instead of the wearable terminal.

[0328] In addition, in the above embodiments, the server device 110 is described as executing the two-way communication program as a single unit. However, if the server device 110 is composed of one or more computers, the two-way communication program can be installed in each of the one or more computers and executed in a distributed computing manner.

[0329] In addition, in each of the above embodiments, as an example of a method for installing the client program, a method of installing via network download is mentioned. At this time, although the download source is not specifically mentioned, when installing via this method, the download source can be, for example, a server device that stores the client program in an accessible manner. In addition, the server device can also be a cloud device that accepts access from client terminals (supporter terminal 130, web terminal 1420, 1430) via the network and downloads the client program on the condition of charging a fee. In other words, the server device can also be a cloud device that provides services for the client program.

[0330] Although the embodiments have been described above, it will be understood that various changes in form and details can be made without departing from the spirit and scope of the claims.

[0331] This application claims priority based on Japanese Patent Application No. 2022-140702 filed on September 5, 2022, the entire contents of which are incorporated herein by reference.

[0332] Label Description

[0333] 100: Two-way communication system

[0334] 110: Server device

[0335] 111: Two-way communication department

[0336] 112: Text Data Management Department

[0337] 120: Operator terminal

[0338] 130: Supporter Terminal

[0339] 310: Audio data acquisition unit

[0340] 320: Speech separation unit

[0341] 330: Sound pressure level calculation unit

[0342] 340: First Detection Department

[0343] 350: Second Detection Unit

[0344] 360: Ministry of Communications

[0345] 370: Judgment Department

[0346] 380: Voice output unit

[0347] 410: Ministry of Communications

[0348] 420: Voice data acquisition unit

[0349] 430: Judgment Department

[0350] 440: Text data generation unit

[0351] 510: Audio data acquisition unit

[0352] 520: Speech separation unit

[0353] 530: Sound pressure level calculation department

[0354] 540: First Detection Department

[0355] 550: Second Detection Unit

[0356] 560: Ministry of Communications

[0357] 570: Voice output unit

[0358] 580: Display control unit

[0359] 590: Judgment Department

[0360] 800: Display screen

[0361] 910: Text data generation unit

[0362] 920: Ministry of Communications

[0363] 1010: Ministry of Communications

[0364] 1110: Text data generation unit

[0365] 1120: Ministry of Communications

[0366] 1400: Two-way communication system

[0367] 1420: Web Terminal

[0368] 1430: Web terminal

[0369] 1510: Text Data Correction Department

[0370] 1610: Operation content determination unit.

Claims

1. A two-way communication system comprising: an operator terminal used by an operator; a supporter terminal used by a supporter who remotely supports the operator; and a server device for implementing two-way communication between the operator terminal and the supporter terminal; wherein a control unit of the server device converts voice data into text data. The control unit of the server device, Before the communication ends, it is determined whether the voice data acquired by the control unit of the operator terminal during the communication can be converted into text data. If the two-way communication is determined to be disconnected due to the absence of voice data transmitted from the operator terminal within a certain period of time, it is determined that the voice data obtained in the communication cannot be converted into text data due to the deterioration of the communication state of the two-way communication. The control unit included in the supporter terminal outputs the determination result.

2. The two-way communication system according to claim 1, wherein: The control unit of the server device, Before the communication ends, it is determined whether the quality of text data converted by the control unit of the operator terminal from the voice data acquired during the communication has deteriorated. When the size of the ambient sound data obtained in the communication is above a given threshold, or when the ratio of the size of the voice data obtained in the communication to the size of the ambient sound data satisfies a given condition, it is determined that the quality of the converted text data is reduced due to the influence of the ambient sound around the voice acquisition device possessed by the operator terminal.

3. The two-way communication system according to claim 1, wherein: The control unit of the server device, Before the communication ends, it is determined whether the voice data acquired by the control unit of the operator terminal during the communication can be converted into text data. If the size of the voice data and the size of the ambient sound data acquired during the communication are both smaller than a predetermined threshold, it is determined that the voice acquisition device included in the operator terminal is faulty and cannot be converted into text data.

4. The two-way communication system according to any one of claims 1 to 3, wherein: The control unit included in the supporter terminal displays the determination result using an icon that clearly indicates a reason why the conversion to text data cannot be performed or a reason why the quality of the converted text data is degraded.

5. The two-way communication system according to any one of claims 1 to 3, wherein: The control unit included in the operator terminal reports the determination result by at least one of voice, light, and vibration.

6. The two-way communication system according to claim 1, wherein: The control unit included in the server device stores the text data in association with the determination result.

7. The two-way communication system according to claim 6, wherein: The control unit included in the supporter terminal reads the stored text data and displays it in a display format corresponding to the corresponding determination result.

8. The two-way communication system according to claim 6, wherein: The control unit included in the server device uses the previous and next text data to correct portions that cannot be converted into text data or portions whose quality has deteriorated.

9. The two-way communication system according to claim 6, wherein: The control unit included in the server device specifies the content of work performed by the operator based on the image captured by the operator terminal, and stores the specified content of work in association with the determination result.

10. The two-way communication system according to claim 6, wherein: The control unit included in the operator terminal stores the voice data acquired during the communication, and when a request is received from the server device, transmits the voice data in a range corresponding to the request to the server device.

11. A server device, wherein a control unit converts voice data into text data, and realizes bidirectional communication between an operator terminal used by an operator and a supporter terminal used by a supporter who remotely supports the operator. The control unit, Before the communication ends, it is determined whether the voice data acquired by the control unit of the operator terminal during the communication can be converted into text data. If the two-way communication is determined to be disconnected due to the absence of voice data transmitted from the operator terminal within a certain period of time, it is determined that the voice data obtained in the communication cannot be converted into text data due to the deterioration of the communication state of the two-way communication. An instruction according to the determination result is transmitted to the operator terminal or the supporter terminal.

12. A communication program product comprising a communication program for causing a control unit of a server device that implements bidirectional communication between an operator terminal used by an operator and a supporter terminal used by a supporter who remotely supports the operator and converts voice data into text data to execute the following steps: Before the communication ends, a step of determining whether the voice data acquired by the control unit of the operator terminal during the communication can be converted into text data; When it is determined that bidirectional communication is disconnected due to the absence of voice data transmitted from the operator terminal within a predetermined period, determining that the voice data acquired during the communication cannot be converted into text data due to a deterioration of the communication state of the bidirectional communication; and transmitting an instruction corresponding to the determination result to the operator terminal or the supporter terminal.

Images