System, support method, server device, and communication program

Abstract

In a system for speech recognition of speaking content, a user can grasp that a situation that is not suitable for speech recognition has occurred. In a system for converting voice data into text data, one or more control units determine whether or not the acquired voice data can be converted into text data, or whether or not the quality of the converted text data obtained by converting the acquired voice data is reduced during acquisition of the voice data. And outputting a judgment result.

Description

Technical Field

[0001] The present invention relates to a system, a supporting method, a server device and a communication program. Background Art

[0002] There is a known two-way communication system that enables remote communication by bidirectionally sending and receiving dynamic image data and voice data. In the two-way communication system, for example, by using a voice text wake-up function, the user's speech content can be recognized in real time and converted into text data. As a result, according to the two-way communication system, the user's speech content can be managed as text information.

[0003] <Prior Art Literature>

[0004] <Patent Documents>

[0005] Patent Document 1: Japanese Patent Application Publication No. 2021-2747 Summary of the invention

[0006] <Problems to be Solved by the Invention>

[0007] However, in the case of the above-mentioned two-way communication system, due to various reasons such as a failure of the microphone, a deterioration of the communication state, and the influence of surrounding ambient sound, there are cases where a part of the utterance cannot be properly voice recognized.

[0008] On the other hand, even if it is found that a part of the speech content has not been converted into appropriate text data after the remote communication ends, it is difficult to restore the text data.

[0009] The present invention enables a user to understand that a situation unsuitable for speech recognition has occurred in a system for performing speech recognition on spoken content.

[0010] <Methods used to solve the problem>

[0011] The first mode of the present invention is a system for converting voice data into text data. One or more control units determine whether the voice data being acquired can be converted into text data, or whether the quality of the converted text data obtained by converting the voice data being acquired is reduced, and output the determination result.

[0012] According to the first aspect of the present invention, in a system for performing speech recognition on utterances, a user can understand that a situation unsuitable for speech recognition has occurred.

[0013] Furthermore, in a second aspect of the present invention, in the system according to the first aspect, the one or more control units output the determination result by clearly indicating a reason why the conversion into text data cannot be performed or a reason why the quality of the converted text data is reduced.

[0014] Furthermore, in a third aspect of the present invention, in the system according to the second aspect, the cause includes any one of an influence of ambient sound around a voice acquisition device that acquires the voice data, or a failure of the voice acquisition device.

[0015] In addition, in the fourth mode of the present invention, in the system described in the third mode, when the size of the ambient sound data obtained in the acquisition of the voice data is greater than a given threshold, the one or more control units determine that the quality of the converted text data is reduced due to the influence of the ambient sound.

[0016] In addition, in the fifth mode of the present invention, in the system described in the third or fourth mode, when the size of the voice data being acquired and the ambient sound data acquired in the acquisition of the voice data are both smaller than a given threshold, the one or more control units determine that the voice acquisition device cannot be converted into text data due to a failure of the voice acquisition device.

[0017] Furthermore, in a sixth aspect of the present invention, in the system according to any one of the second to fifth aspects, the cause includes deterioration of a communication state of bidirectional communication.

[0018] Furthermore, in a seventh aspect of the present invention, in the system described in the sixth aspect, when bidirectional communication is disconnected during acquisition of the voice data, the one or more control units determine that conversion into text data is impossible due to deterioration of the communication state.

[0019] In addition, the eighth mode of the present invention is a system recorded in any one of the second to seventh modes, comprising: an operator terminal used by an operator; a supporter terminal used by a supporter who provides remote support to the operator; and a server device that realizes two-way communication between the operator terminal and the supporter terminal, and the control unit of the server device converts voice data into text data.

[0020] In addition, in the ninth mode of the present invention, in the system described in the eighth mode, the control unit of the server device determines whether the voice data being acquired can be converted into text data, or whether the quality of the converted text data obtained by converting the voice data being acquired is reduced, and the control unit of the supporter terminal outputs the determination result.

[0021] In addition, in the tenth mode of the present invention, in the system described in the ninth mode, the operator terminal has a voice acquisition device for acquiring the voice data, and the control unit of the server device determines that the quality of the converted text data is reduced due to the influence of the ambient sound around the voice acquisition device when the size of the ambient sound data acquired during the acquisition of the voice data is greater than a given threshold.

[0022] In addition, in the eleventh mode of the present invention, in the system described in the ninth mode, the operator terminal has a voice acquisition device for acquiring the voice data, and the control unit of the server device determines that the voice data cannot be converted into text data due to a malfunction of the voice acquisition device when the size of the voice data being acquired and the size of the ambient sound data acquired in the acquisition of the voice data are both smaller than a given threshold.

[0023] Furthermore, in the twelfth mode of the present invention, in the system described in the ninth mode, when the control unit of the server device determines that the two-way communication is disconnected because no voice data is sent from the operator terminal within a certain period of time, it is determined that the voice data obtained by the control unit of the operator terminal cannot be converted into text data due to the deterioration of the communication state of the two-way communication.

[0024] In addition, in a thirteenth aspect of the present invention, in the system according to any one of the tenth to twelfth aspects, the control unit of the supporter terminal displays the determination result using an icon that clearly indicates the reason why the conversion to text data cannot be performed or the reason why the quality of the converted text data is reduced.

[0025] In addition, in the fourteenth mode of the present invention, in the system described in the tenth or eleventh mode, the control unit possessed by the operator terminal determines whether the voice data being acquired can be converted into text data, or whether the quality of the converted text data obtained by converting the voice data being acquired is reduced, and reports the determination result by at least any one of voice, light, and vibration.

[0026] Furthermore, a fifteenth aspect of the present invention is the system according to any one of the ninth to fourteenth aspects, wherein the control unit of the server device stores the text data in association with the determination result.

[0027] Furthermore, in a sixteenth aspect of the present invention, in the system according to the fifteenth aspect, the control unit included in the supporter terminal reads out the stored text data and displays it in a display mode corresponding to the corresponding determination result.

[0028] Furthermore, in a seventeenth aspect of the present invention, in the system described in any one of the eighth to sixteenth aspects, the control unit included in the server device uses the previous and next text data to correct a portion that cannot be converted into text data or a portion whose quality is reduced.

[0029] In addition, in the eighteenth mode of the present invention, in the system described in any one of the ninth to seventeenth modes, the control unit of the server device determines the work content of the operator based on the image taken by the operator terminal, and saves the determined work content in association with the judgment result.

[0030] In addition, in the nineteenth mode of the present invention, in the system described in any one of the ninth to eighteenth modes, the control unit possessed by the operator terminal saves the acquired voice data, and when there is a request from the server device, sends the voice data in the range corresponding to the request to the server device.

[0031] In addition, the twentieth embodiment of the present invention is a support method involving a system for converting voice data into text data, comprising: a judgment step, in the acquisition of voice data, judging whether the voice data being acquired can be converted into text data, or whether the quality of the converted text data obtained by converting the voice data being acquired is reduced; and an output step, outputting the judgment result.

[0032] In addition, the twenty-first embodiment of the present invention is a server device, which has a control unit, which realizes two-way communication between an operator terminal used by an operator and a supporter terminal used by a supporter who remotely supports the operator, and the control unit converts voice data into text data, and in the acquisition of voice data by the operator terminal or the supporter terminal, when it is determined whether the voice data being acquired can be converted into text data or whether the quality of the converted text data obtained by converting the voice data being acquired is reduced, an instruction corresponding to the determination result is sent to the operator terminal or the supporter terminal.

[0033] In addition, the twenty-second mode of the present invention is a communication program, which is used to enable the control unit of a server device that realizes two-way communication between an operator terminal used by an operator and a supporter terminal used by a supporter who remotely supports the operator to execute the following processes: a conversion process of converting voice data into text data; and a sending process of, in the acquisition of voice data by the operator terminal or the supporter terminal, when it is determined whether the voice data being acquired can be converted into text data or whether the quality of the converted text data obtained by converting the voice data being acquired is reduced, an instruction corresponding to the determination result is sent to the operator terminal or the supporter terminal. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is the first diagram showing an example of a system configuration of a bidirectional communication system.

[0035] Figure 2 It is a diagram showing an example of the hardware configuration of a server device, and an example of the hardware configuration of an operator terminal and a supporter terminal.

[0036] Figure 3 This is the first diagram showing an example of the functional configuration of the operator terminal.

[0037] Figure 4 This is the first diagram showing an example of the details of the functional configuration of the server device.

[0038] Figure 5 This is the first diagram showing an example of the functional configuration of the supporter terminal.

[0039] Figure 6 This is a first sequence diagram showing the flow of communication processing when voice data is transmitted from the operator terminal to the supporter terminal in the two-way communication system.

[0040] Figure 7 This is a first sequence diagram showing the flow of communication processing when voice data is transmitted from a supporter terminal to an operator terminal in a two-way communication system.

[0041] Figure 8 It is a diagram showing an example of a display screen of a supporter terminal.

[0042] Fig. 9 The second diagram shows an example of the functional configuration of the operator terminal.

[0043] Fig.10 The second diagram shows an example of the details of the functional configuration of the server device.

[0044] Fig.11 It is a second diagram showing an example of the functional configuration of the supporter terminal.

[0045] Fig.12 This is a second sequence diagram showing the flow of communication processing when voice data is transmitted from the operator terminal to the supporter terminal in the two-way communication system.

[0046] Fig.13 This is a second sequence diagram showing the flow of communication processing when voice data is transmitted from the supporter terminal to the operator terminal in the two-way communication system.

[0047] Fig.14 The second diagram shows an example of a system configuration of a bidirectional communication system.

[0048] Fig.15 This is a diagram showing an example of using text data.

[0049] Fig.16 The third diagram shows an example of the details of the functional configuration of the server device. DETAILED DESCRIPTION

[0050] Hereinafter, each embodiment will be described with reference to the drawings. In addition, in this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and repeated descriptions are omitted.

[0051] [First embodiment]

[0052] <System structure of two-way communication system>

[0053] First, a system configuration of a bidirectional communication system as an example of a system according to the first embodiment will be described. Figure 1 1 is a first diagram showing an example of a system configuration of a two-way communication system. Figure 1 As shown, the two-way communication system 100 includes a server device 110 and client terminals (an operator terminal 120 and a supporter terminal 130 ).

[0054] In addition, in this embodiment,

[0055] 150 workers working at the work site and

[0056] A supporter 160 remotely supports the operator 150 by remotely instructing the operator 150 on various operations.

[0057] The situation of communicating using the two-way communication system 100. Therefore, as described above, in this embodiment, the client terminals constituting the two-way communication system 100 include the operator terminal 120 used by the operator 150 and the supporter terminal 130 used by the supporter 160. However, the client terminals constituting the two-way communication system 100 are not limited to the combination of the operator terminal 120 and the supporter terminal 130.

[0058] The server device 110 is a device that provides a cloud service (two-way communication service) via the network 140. Specifically, a two-way communication program is installed in the server device 110, and by executing the program, the server device 110 functions as a two-way communication unit 111 and a text data management unit 112. Thus, the server device 110 provides a two-way communication service to the operator 150 and the supporter 160.

[0059] Specifically, in the server device 110, the two-way communication unit 111 is connected to the operator terminal 120 and the supporter terminal 130 for communication.

[0060] · Sending the moving image data and audio data received from the operator terminal 120 to the supporter terminal 130,

[0061] · Sending a display instruction based on the detection result (described in detail later) detected in the operator terminal 120 to the supporter terminal 130,

[0062] · Sending the audio data received from the supporter terminal 130 to the operator terminal 120,

[0063] A display instruction based on a detection result (described in detail later) detected in the supporter terminal 130 is transmitted to the supporter terminal 130 .

[0064] In addition, the two-way communication unit 111 determines based on the detection result detected in the operator terminal 120.

[0065] Whether the voice data based on the speech of the operator 150 obtained in the operator terminal 120 can be converted into text data; or

[0066] whether the quality of text data obtained by converting the voice data based on the speech of the operator 150 obtained in the operator terminal 120 is deteriorated,

[0067] Then, a display instruction according to the determination result is transmitted to the supporter terminal 130 .

[0068] Similarly, the two-way communication unit 111 determines based on the detection result detected in the supporter terminal 130.

[0069] Whether the voice data based on the speech of the supporter 160 obtained in the supporter terminal 130 can be converted into text data, or

[0070] whether the quality of text data obtained by converting the voice data based on the speech of the supporter 160 obtained in the supporter terminal 130 is deteriorated,

[0071] Then, a display instruction according to the determination result is transmitted to the supporter terminal 130 .

[0072] In the server device 110 , the text data management unit 112 converts voice data included in audio data bidirectionally transmitted and received between the operator terminal 120 and the supporter terminal 130 into text data in real time and stores the text data.

[0073] The worker terminal 120 is, for example, a wearable terminal worn at a work site by the worker 150. In addition, in the present embodiment, it is assumed that the work site is in an environment where the communication quality of two-way communication is likely to deteriorate.

[0074] The operator terminal 120 has a function of capturing a moving image of the work site and a function of detecting sound including voice (for example, voice based on the speech of the operator 150). In addition, the operator terminal 120 has a function of transmitting moving image data obtained by capturing the state of the work site and sound data including voice data based on the speech of the operator 150 to the supporter terminal 130 via the network 140 and the server device 110.

[0075] In addition, the operator terminal 120 detects

[0076] ·Converting the speech data based on the speech of the operator 150 into text data, or

[0077] The quality of the transformed text data

[0078] The detection results of the “ambient sound state”, “microphone state” and “communication state” that cause influence are reported to the operator 150 , and the detection results of the “ambient sound state” and “microphone state” are sent to the server device 110 .

[0079] Furthermore, the operator terminal 120 has a function of receiving and outputting audio data including voice data based on the speech of the supporter 160 .

[0080] The supporter terminal 130 is installed in, for example, an office where the supporter 160 resides, and is a terminal operated by the supporter 160 .

[0081] The supporter terminal 130 has a function of detecting sound including voice (for example, voice based on the speech of the supporter 160 ). In addition, the supporter terminal 130 has a function of transmitting sound data including voice data based on the speech of the supporter 160 to the operator terminal 120 via the network 140 and the server device 110 .

[0082] In addition, the supporter terminal 130 detects

[0083] ·Converting voice data based on the speech of the supporter 160 into text data, or

[0084] The quality of the transformed text data

[0085] The “state of the ambient sound”, “state of the microphone” and “state of communication” that cause influence, and the detection results of the “state of the ambient sound” and “state of the microphone” are sent to the server device 110.

[0086] Furthermore, the supporter terminal 130 has a function of receiving and outputting moving image data obtained by photographing the state of the work site and audio data including voice data based on the speech of the worker 150 .

[0087] Furthermore, the supporter terminal 130 has a function of displaying a display screen in accordance with a display instruction transmitted from the server device 110 (a display instruction based on a detection result detected in the operator terminal 120 , a display instruction based on a detection result detected in the supporter terminal 130 ) or the like.

[0088] <Hardware structure of each device>

[0089] Next, the hardware configuration of each device (server device 110 , operator terminal 120 , supporter terminal 130 ) constituting the bidirectional communication system 100 will be described. Figure 2 It is a diagram showing an example of the hardware configuration of a server device, and an example of the hardware configuration of an operator terminal and a supporter terminal.

[0090] like Figure 2 As shown in FIG. 2( a ), the server device 110 includes a processor 201 , a memory 202 , an auxiliary storage device 203 , an operation device 204 , a display device 205 , a communication device 206 , and a driver device 207 . The hardware components of the server device 110 are connected to each other via a bus 208 .

[0091] The processor 201 includes various computing devices such as a CPU (Central Processing Unit) etc. The processor 201 reads various programs (for example, a two-way communication program etc.) to the memory 202 and executes them.

[0092] The memory 202 has a main storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The processor 201 and the memory 202 form a so-called computer (also referred to as a "control unit"), and the processor 201 executes various programs read out to the memory 202, and the computer realizes various functions.

[0093] The auxiliary storage device 203 stores various programs and various data used when the processor 201 executes the various programs. In addition, the auxiliary storage device 203 realizes a text data storage unit 450 described later.

[0094] The operation device 204 is an operation device for the administrator of the server device 110 to perform various operations. The display device 205 is a display device for displaying the processing results of various processes executed by the server device 110.

[0095] The communication device 206 is a communication device for communicating with an external device (for example, the operator terminal 120 or the supporter terminal 130 ) via the network 140 .

[0096] The drive device 207 is a device for setting the storage medium 210. The storage medium 210 mentioned here includes a medium that stores information optically, electrically or magnetically, such as a CD-ROM, a floppy disk, an optical magnetic disk, etc. In addition, the storage medium 210 may also include a semiconductor memory that stores information electrically, such as a ROM, a flash memory, etc.

[0097] In addition, various programs installed in the auxiliary storage device 203 are installed by, for example, placing the distributed storage medium 210 in the drive device 207 and having the drive device 207 read out various programs stored in the storage medium 210. Alternatively, various programs installed in the auxiliary storage device 203 may be installed by downloading from the network 140 via the communication device 206.

[0098] On the other hand, Figure 2 As shown in (b), the operator terminal 120 and the supporter terminal 130 include a processor 221, a memory 222, an auxiliary storage device 223, and a communication device 224. In addition, the operator terminal 120 includes a GPS (Global Positioning System) device 225, an acceleration sensor 226, and a camera device 229. In addition, the operator terminal 120 and the supporter terminal 130 include a voice acquisition device 227, a voice output device 228, and an operation device 230. Furthermore, the supporter terminal 130 includes a display device 231 and a driver device 232. In addition, the hardware components constituting the operator terminal 120 and the supporter terminal 130 are connected to each other via a bus 233.

[0099] The processor 221 includes various computing devices such as a CPU (Central Processing Unit) etc. The processor 221 reads various programs (such as a client program described later) to the memory 222 and executes them.

[0100] The memory 222 has a main storage device such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The processor 221 and the memory 222 form a so-called computer (also referred to as a "control unit"), and the processor 221 executes various programs read out to the memory 222, and the computer realizes various functions.

[0101] The auxiliary storage device 223 stores various programs and various information used when the processor 221 executes the various programs.

[0102] The communication device 224 is a communication device for transmitting and receiving various data (such as dynamic image data, sound data, etc.) between the server device 110. The GPS device 225 detects the position of the operator terminal 120 and outputs the position data. The acceleration sensor 226 detects the movement of the operator 150 wearing the operator terminal 120 and outputs the acceleration data.

[0103] The speech acquisition device 227 detects speech based on the speech of the worker 150 and the supporter 160 , ambient sound around the work site, ambient sound around the office, and the like, and outputs sound data.

[0104] The voice output device 228 is a device that outputs various data received from the server device 110 by voice to the operator 150 wearing the operator terminal 120 or the supporter 160 operating the supporter terminal 130 , for example.

[0105] The imaging device 229 captures the surroundings (working site) of the worker 150 wearing the worker terminal 120 , and generates moving image data.

[0106] The operating device 230 accepts simple operations of the operator terminal 120 by the operator 150 , such as turning on / off the power switch of the operator terminal 120 . Alternatively, the operating device 230 accepts various operations of the supporter terminal 130 by the supporter 160 .

[0107] The display device 231 is a device that displays the moving image data received from the server device 110 and display contents according to the display instruction to the supporter 160 .

[0108] The drive device 232 is a device for setting the storage medium 240. The storage medium 240 mentioned here is the same as the storage medium 210 mentioned above.

[0109] In the case of the supporter terminal 130, various programs installed in the auxiliary storage device 223 are installed by, for example, placing the distributed storage medium 240 in the drive device 232 and reading it out by the drive device 232. Alternatively, various programs installed in the auxiliary storage device 223 may be installed by downloading from the network 140 via the communication device 224.

[0110] <Functional structure of operator terminal>

[0111] Next, the functional configuration of the operator terminal 120 will be described. Figure 3 1 is a first diagram showing an example of the functional structure of the operator terminal. As described above, the operator terminal 120 has a client program installed therein. The operator terminal 120 executes the program, thereby Figure 3As shown in FIG. 1 , the sound data acquisition unit 310, the speech separation unit 320, the sound pressure level calculation unit 330, the first detection unit 340, the second detection unit 350, the communication unit 360, the determination unit 370, and the speech output unit 380 function. Figure 3 In the example of , in order to simplify the description, the functions related to the processing of dynamic image data are omitted, and only the functions related to the processing of sound data are extracted and shown (the processing of sound data is described below).

[0112] The audio data acquisition unit 310 acquires audio data detected by the audio acquisition device 227 , and notifies the audio separation unit 320 and the communication unit 360 .

[0113] The speech separation unit 320 separates the speech data notified by the speech data acquisition unit 310 into speech data based on the speech of the operator 150 and ambient sound data (sound data other than speech data) based on the ambient sound of the surroundings of the speech acquisition device 227. In addition, the speech separation unit 320 notifies the separated speech data and ambient sound data to the sound pressure level calculation unit 330.

[0114] The sound pressure level calculation unit 330 calculates the size (sound pressure level) of the voice data notified from the voice separation unit 320, and notifies it to the second detection unit 350. In addition, the sound pressure level calculation unit 330 calculates the size (sound pressure level) of the ambient sound data notified from the voice separation unit 320, and notifies it to the first detection unit 340 and the second detection unit 350.

[0115] The first detector 340 detects the state of the ambient sound by determining whether the sound pressure level of the ambient sound data notified from the sound pressure level calculator 330 is greater than a given threshold. Specifically, the first detector 340 detects that there is an influence of the ambient sound when determining that the sound pressure level of the ambient sound data is greater than a given threshold.

[0116] On the other hand, when the first detection unit 340 determines that the sound pressure level of the ambient sound data is less than a given threshold, it detects that there is no influence of the ambient sound. Furthermore, the first detection unit 340 notifies the communication unit 360 of the state of the ambient sound (influence of the ambient sound, no influence of the ambient sound) as a detection result.

[0117] The second detection unit 350 detects the state of the voice acquisition device 227 (the state of the microphone) by determining whether the sound pressure level of the ambient sound data and the sound pressure level of the voice data notified from the sound pressure level calculation unit 330 are both less than a given threshold. Specifically, when the second detection unit 350 determines that both are less than a given threshold, it detects that the voice acquisition device 227 has a fault (microphone abnormality). On the other hand, when the second detection unit 350 determines that the sound pressure level of either is greater than a given threshold, it detects that the voice acquisition device 227 has not a fault (microphone normal). Furthermore, the second detection unit 350 notifies the communication unit 360 of the state of the microphone (microphone abnormality, microphone normal) as a detection result.

[0118] The communication unit 360 transmits the sound data notified from the sound data acquisition unit 310 to the server device 110 , and transmits the state of the ambient sound (affected by the ambient sound, not affected by the ambient sound) and the state of the microphone (microphone abnormal, microphone normal) as detection results to the server device 110 .

[0119] In addition, the communication unit 360 detects the state of communication between the operator terminal 120 and the server device 110. Specifically, when the communication connection between the operator terminal 120 and the server device 110 is disconnected (when no sound data is sent from the server device 110 for a certain period of time), the communication unit 360 detects that the communication is deteriorated. In addition, when the communication connection between the operator terminal 120 and the server device 110 is continued (when sound data is sent from the server device 110), the communication unit 360 detects that the communication is normal.

[0120] In addition, the communication unit 360 notifies the determination unit 370 of the state of ambient sound (affected by ambient sound, not affected by ambient sound), the state of the microphone (microphone abnormality, microphone normal), and the state of communication (communication deterioration, communication normal) as detection results.

[0121] Furthermore, the communication unit 360 receives the audio data from the server device 110 and notifies the audio output unit 380 .

[0122] The determination unit 370 determines based on the detection result notified from the communication unit 360.

[0123] Whether the voice data based on the speech of the operator 150 obtained in the operator terminal 120 can be converted into text data; or

[0124] Whether the quality of text data obtained by converting the voice data based on the speech of the operator 150 being acquired in the operator terminal 120 has deteriorated.

[0125] Specifically, when notified of the influence of the ambient sound, the determination unit 370 determines that the voice data based on the speech of the operator 150 cannot be converted into text data due to the influence of the ambient sound, or that the quality of the converted text data is reduced.

[0126] Furthermore, when the determination unit 370 is notified of the microphone abnormality, it determines that the voice data based on the speech of the operator 150 cannot be converted into text data due to the failure of the voice acquisition device 227 or the quality of the converted text data is reduced.

[0127] Furthermore, when notified of the communication deterioration, the determination unit 370 determines that the voice data based on the speech of the operator 150 cannot be converted into text data due to the communication deterioration, or that the quality of the converted text data is degraded.

[0128] Furthermore, when notified that there is no influence of ambient sound, the microphone is normal, and communication is normal, the determination unit 370 determines that the voice data based on the speech of the operator 150 can be converted into text data or that the quality of the converted text data is not reduced.

[0129] The voice output unit 380 outputs the voice data notified from the communication unit 360 to the operator 150 via the voice output device 228. In addition, the voice output unit 380 reports the determination result notified from the determination unit 370 to the operator 150 via the voice output device 228. Thus, the operator 150 can understand the occurrence of a situation in which it is not suitable to perform voice recognition on the speech content of the operator 150 and the cause thereof. As a result, the operator 150 can immediately take measures according to the cause, thereby avoiding a situation in which it is later determined that a part of the speech content is not properly converted into text data.

[0130] <Details of functional configuration of server device>

[0131] Next, the functional configuration of the server device 110 will be described in detail. As described above, the two-way communication program is installed in the server device 110, and the two-way communication unit 111 and the text data management unit 112 function as the program are executed.

[0132] Figure 4 1 is a first diagram showing an example of the details of the functional configuration of the server device. Figure 4 As shown, the two-way communication unit 111 further includes a communication unit 410 , a voice data acquisition unit 420 , and a determination unit 430 . In addition, the text data management unit 112 further includes a text data generation unit 440 .

[0133] The communication unit 410 receives voice data transmitted from the operator terminal 120, transmits it to the supporter terminal 130, and notifies the voice data acquisition unit 420. The communication unit 410 receives voice data transmitted from the supporter terminal 130, transmits it to the operator terminal 120, and notifies the voice data acquisition unit 420.

[0134] Furthermore, the communication unit 410 receives the detection results (ambient sound state, microphone state) transmitted from the operator terminal 120 and the detection results (ambient sound state, microphone state) transmitted from the supporter terminal 130 , and notifies the determination unit 430 of the same.

[0135] In addition, the communication unit 410 detects the state of communication between the operator terminal 120 and the server device 110. Specifically, when the communication connection between the operator terminal 120 and the server device 110 is disconnected (when no voice data is sent from the operator terminal 120 for a certain period of time), the communication unit 360 detects that the communication is deteriorated. In addition, when the communication connection between the operator terminal 120 and the server device 110 is continued (when voice data is sent from the operator terminal 120), the communication unit 360 detects that the communication is normal. In addition, the communication unit 410 notifies the determination unit 430 of the state of communication (communication deterioration, communication normal) as a detection result.

[0136] Similarly, the communication unit 410 detects the state of communication between the supporter terminal 130 and the server device 110. Specifically, when the communication connection between the supporter terminal 130 and the server device 110 is disconnected (when the voice data is not transmitted from the supporter terminal 130 for a certain period of time), the communication unit 410 detects that the communication is deteriorated. In addition, when the communication connection between the supporter terminal 130 and the server device 110 is continued, the communication unit 410 detects that the communication is normal. In addition, the communication unit 410 notifies the determination unit 430 of the state of communication (communication deterioration, communication normal) as a detection result.

[0137] In addition, the communication unit 410 transmits a display instruction including display contents (state of ambient sound, state of microphone, state of communication) notified from the determination unit 430 in response to notification of the detection result (state of ambient sound, state of microphone, state of communication) about the operator terminal 120 to the supporter terminal 130. In addition, the communication unit 410 transmits a display instruction including display contents (state of ambient sound, state of microphone) notified from the determination unit 430 in response to notification of the detection result (state of ambient sound, state of microphone, state of communication) about the supporter terminal 130 to the supporter terminal 130.

[0138] The voice data acquisition unit 420 acquires the voice data from the sound data notified by the communication unit 410 , and notifies the text data generation unit 440 of the text data management unit 112 .

[0139] The determination unit 430 determines based on the detection result (ambient sound state, microphone state, communication state) of the operator terminal 120 notified from the communication unit 410.

[0140] Whether the voice data based on the speech of the operator 150 obtained in the operator terminal 120 can be converted into text data; or

[0141] whether the quality of text data obtained by converting the voice data based on the speech of the operator 150 obtained in the operator terminal 120 is deteriorated,

[0142] And determine the display content (ambient sound status, microphone status, communication status) according to the determination result.

[0143] Specifically, when the determination unit 430 is notified of the influence of the ambient sound, it determines that the voice data based on the speech of the operator 150 cannot be converted into text data due to the influence of the ambient sound, or the quality of the converted text data is reduced. Then, the determination unit 430 determines the display content (the state of the ambient sound) according to the determination result.

[0144] Furthermore, when the determination unit 430 is notified of the abnormality of the microphone, it determines that the voice data based on the speech of the operator 150 cannot be converted into text data or the quality of the converted text data is reduced due to the failure of the voice acquisition device 227. Then, the determination unit 430 determines the display content (microphone status) according to the determination result.

[0145] Furthermore, when the determination unit 430 is notified of the communication deterioration, it determines that the voice data based on the speech of the operator 150 cannot be converted into text data due to the communication deterioration, or the quality of the converted text data is reduced. Then, the determination unit 430 determines the display content (communication status) according to the determination result.

[0146] Furthermore, when notified that there is no influence of ambient sound, the microphone is normal, and the communication status is normal, the determination unit 430 determines that the voice data based on the speech of the operator 150 can be converted into text data, or that the quality of the converted text data is not reduced. Then, the determination unit 430 determines the display content (ambient sound status, microphone status, communication status) according to the determination result.

[0147] Similarly, the determination unit 430 determines based on the detection result (ambient sound state, microphone state, communication state) of the supporter terminal 130 notified from the communication unit 410.

[0148] Whether the voice data based on the speech of the supporter 160 acquired in the supporter terminal 130 can be converted into text data; or

[0149] whether the quality of text data obtained by converting the voice data based on the speech of the supporter 160 obtained in the supporter terminal 130 is deteriorated,

[0150] And determine the display content (ambient sound status, microphone status) according to the determination result.

[0151] Specifically, when notified of the influence of the ambient sound, the determination unit 430 determines that the voice data based on the speech of the supporter 160 cannot be converted into text data due to the influence of the ambient sound, or the quality of the converted text data is reduced. Then, the determination unit 430 determines the display content (the state of the ambient sound) according to the determination result.

[0152] Furthermore, when the determination unit 430 is notified of the abnormality of the microphone, it determines that the voice data based on the speech of the supporter 160 cannot be converted into text data or the quality of the converted text data is reduced due to the failure of the voice acquisition device 227. Then, the determination unit 430 determines the display content (microphone status) according to the determination result.

[0153] Furthermore, when notified of the communication deterioration, the determination unit 430 determines that the voice data based on the speech of the operator 150 cannot be converted into text data due to the communication deterioration, or that the quality of the converted text data is degraded.

[0154] Furthermore, when notified that there is no influence of ambient sound, the microphone is normal, and the communication is normal, the determination unit 430 determines that the voice data based on the speech of the supporter 160 can be converted into text data or the quality of the converted text data is not reduced. Then, the determination unit 430 determines the display content (the state of ambient sound, the state of microphone) according to the determination result.

[0155] Furthermore, the determination unit 430 stores the determination result in the text data storage unit 450 in association with text data described later.

[0156] The text data generating unit 440 performs speech recognition on the speech data notified from the speech data acquiring unit 420 and converts the speech data into text data. The text data generating unit 440 stores the converted text data in the text data storage unit 450 in association with the determination result.

[0157] <Functional structure of the supporter terminal>

[0158] Next, the functional configuration of the supporter terminal 130 will be described. Figure 5 1 is a first diagram showing an example of the functional structure of the supporter terminal. As described above, the supporter terminal 130 has a client program installed therein. The supporter terminal 130 executes the program, thereby Figure 5 As shown, the supporter terminal 130 functions as a voice data acquisition unit 510, a voice separation unit 520, a sound pressure level calculation unit 530, a first detection unit 540, a second detection unit 550, and a communication unit 560. Furthermore, the supporter terminal 130 functions as a voice output unit 570, a display control unit 580, and a determination unit 590.

[0159] The audio data acquisition unit 510 acquires the audio data detected by the audio acquisition device 227 , and notifies the audio separation unit 520 and the communication unit 560 .

[0160] The speech separation unit 520 separates the speech data notified from the speech data acquisition unit 510 into speech data based on the speech of the supporter 160 and ambient sound data (sound data other than speech data) based on ambient sound around the speech acquisition device 227. The speech separation unit 520 also notifies the sound pressure level calculation unit 530 of the separated speech data and ambient sound data.

[0161] The sound pressure level calculation unit 530 calculates the sound pressure level of the speech data notified from the speech separation unit 520, and notifies the second detection unit 550. The sound pressure level calculation unit 530 also calculates the sound pressure level of the ambient sound data notified from the speech separation unit 520, and notifies the first detection unit 540 and the second detection unit 550.

[0162] The first detector 540 detects the state of the ambient sound by determining whether the sound pressure level of the ambient sound data notified from the sound pressure level calculator 530 is greater than a given threshold. Specifically, the first detector 540 detects that there is an influence of the ambient sound when determining that the sound pressure level of the ambient sound data is greater than a given threshold.

[0163] On the other hand, when the first detection unit 540 determines that the sound pressure level of the ambient sound data is less than a given threshold, it detects that there is no influence of the ambient sound. Furthermore, the first detection unit 540 notifies the communication unit 560 of the state of the ambient sound (with influence of the ambient sound, without influence of the ambient sound) as a detection result.

[0164] The second detection unit 550 detects the state of the voice acquisition device 227 (the state of the microphone) by determining whether the sound pressure level of the ambient sound data and the sound pressure level of the voice data notified from the sound pressure level calculation unit 530 are both less than a given threshold. Specifically, when the second detection unit 550 determines that both are less than a given threshold, it detects that the voice acquisition device 227 has a fault (microphone abnormality). On the other hand, when the second detection unit 550 determines that the sound pressure level of either is above a given threshold, it detects that the voice acquisition device 227 has not a fault (microphone normal). Furthermore, the second detection unit 550 notifies the communication unit 560 of the state of the microphone (microphone abnormality, microphone normal) as a detection result.

[0165] The communication unit 560 transmits the sound data notified from the sound data acquisition unit 510 to the server device 110 , and transmits the state of the ambient sound (affected by the ambient sound, not affected by the ambient sound) and the state of the microphone (microphone abnormal, microphone normal) as detection results to the server device 110 .

[0166] In addition, the communication unit 560 detects the state of communication between the supporter terminal 130 and the server device 110. Specifically, when the communication connection between the supporter terminal 130 and the server device 110 is disconnected (when no voice data is sent from the server device 110 for a certain period of time), the communication unit 560 detects that the communication is deteriorated. In addition, when the communication connection between the supporter terminal 130 and the server device 110 is continued (when voice data is sent from the server device 110), the communication unit 560 detects that the communication is normal.

[0167] Furthermore, the communication unit 560 notifies the determination unit 590 of the state of communication (communication deterioration, communication normality) as a detection result.

[0168] Furthermore, the communication unit 560 receives audio data from the server device 110 and notifies the audio output unit 570 of the received audio data.

[0169] Furthermore, the communication unit 560 receives a display instruction from the server device 110 and notifies the display control unit 580. In addition, the display instruction received by the communication unit 560 includes:

[0170] · Display instructions corresponding to the detection results (ambient sound status, microphone status, communication status) of the operator terminal 120; and

[0171] Display instructions corresponding to the detection results (ambient sound status, microphone status) of the supporter terminal 130.

[0172] The audio output unit 570 outputs the audio data notified from the communication unit 560 to the supporter 160 via the audio output device 228 .

[0173] The determination unit 590 determines based on the communication status (communication deterioration, communication normality) notified from the communication unit 410 as the detection result.

[0174] Whether the voice data based on the speech of the operator 150 obtained in the operator terminal 120 can be converted into text data; or

[0175] whether the quality of text data obtained by converting the voice data based on the speech of the operator 150 obtained in the operator terminal 120 is deteriorated,

[0176] The display content (the status of the call) is determined according to the determination result. Furthermore, the determination unit 590 notifies the display control unit 580 of a display instruction including the determined display content (the status of the communication).

[0177] The display control unit 580 displays the display content included in the display instruction notified from the communication unit 560 and the display instruction notified from the determination unit 590 on the display device 231 of the supporter terminal 130 .

[0178] <Communication process flow (1)>

[0179] Next, the flow of the communication process of the two-way communication system 100 , that is, the communication process when audio data including voice data based on the speech of the worker 150 is transmitted from the worker terminal 120 to the supporter terminal 130 will be described. Figure 6 This is a first sequence diagram showing the flow of communication processing when voice data is transmitted from the operator terminal to the supporter terminal in the two-way communication system.

[0180] In step S601, the operator terminal 120 acquires audio data including voice data based on the speech of the operator 150. The operator terminal 120 transmits the acquired audio data to the server device 110.

[0181] In step S610, the server device 110 performs a two-way communication process. The two-way communication process of the server device 110 includes receiving and sending processing of voice data, voice recognition processing of voice data, communication status detection processing, acquisition and determination processing of detection results, determination of display content and display instruction processing, and storage processing of text data and determination results. Here, the server device 110 receives voice data and sends it to the supporter terminal 130. In addition, the server device 110 performs voice recognition processing on the voice data and converts it into text data.

[0182] In step S620, the supporter terminal 130 performs a voice output process. The voice output process of the supporter terminal 130 includes acquisition and output of audio data and display processing according to a display instruction. Here, the supporter terminal 130 acquires and outputs audio data.

[0183] In step S602 , the operator terminal 120 separates the audio data into voice data and ambient sound data.

[0184] In step S603 , the operator terminal 120 calculates the sound pressure level of the speech data and the sound pressure level of the ambient sound data.

[0185] In step S604 , the operator terminal 120 detects the state of the ambient sound based on the sound pressure level of the ambient sound data, and transmits the detection result (the state of the ambient sound) to the server device 110 .

[0186] In step S610, the server device 110 performs a two-way communication process. Here, the server device 110 receives the detection result (the state of the ambient sound), makes a determination based on the received detection result (the state of the ambient sound), determines the display content (the state of the ambient sound), and sends a display instruction corresponding to the determined display content to the supporter terminal 130.

[0187] In addition, when receiving the detection result that there is no influence of the ambient sound, the server device 110 determines that the voice data based on the speech of the operator 150 can be converted into text data or the quality of the converted text data is not reduced. In this case, the server device 110 instructs the supporter terminal 130 to display an icon indicating the state of the ambient sound in blue, for example.

[0188] Furthermore, when receiving the detection result indicating the influence of the ambient sound, the server device 110 determines that the voice data based on the speech of the operator 150 cannot be converted into text data or the quality of the converted text data is reduced due to the influence of the ambient sound. In this case, the server device 110 instructs the supporter terminal 130 to display an icon indicating the state of the ambient sound in gray, for example.

[0189] In step S620, the supporter terminal 130 performs a voice output process. Here, the supporter terminal 130 performs a display process corresponding to the display instruction. For example, the supporter terminal 130 displays an icon indicating the state of the ambient sound of the operator terminal 120 in blue. Alternatively, the supporter terminal 130 displays an icon indicating the state of the ambient sound of the operator terminal 120 in gray.

[0190] In step S605, when the detection result in step S604 is "affected by ambient sound", the operator terminal 120 reports the determination result to the operator 150. Specifically, the operator terminal 120 reports the determination result that the voice data based on the speech of the operator 150 cannot be converted into text data due to the influence of ambient sound, or the quality of the converted text data is reduced to the operator 150.

[0191] In step S606 , the operator terminal 120 detects the state of the microphone based on the sound pressure level of the ambient sound data and the sound pressure level of the voice data, and transmits the detection result (the state of the microphone) to the server device 110 .

[0192] In step S610, the server device 110 performs a two-way communication process. Here, the server device 110 receives the detection result (microphone status), makes a judgment based on the received detection result (microphone status), determines the display content (microphone status), and sends a display instruction corresponding to the determined display content to the supporter terminal 130.

[0193] In addition, when receiving the detection result that the microphone is normal, the server device 110 determines that the voice data based on the speech of the operator 150 can be converted into text data or the quality of the converted text data is not reduced. In this case, the server device 110 instructs the supporter terminal 130 to display an icon indicating the status of the microphone in black, for example.

[0194] Furthermore, when receiving the detection result of microphone abnormality, the server device 110 determines that the voice data based on the speech of the operator 150 cannot be converted into text data or the quality of the converted text data is reduced due to a failure of the voice acquisition device 227. In this case, the server device 110 instructs the supporter terminal 130 to display an icon indicating the state of the microphone in gray, for example.

[0195] In step S620, the supporter terminal 130 performs a voice output process. Here, the supporter terminal 130 performs a display process corresponding to the display instruction. For example, the supporter terminal 130 displays an icon indicating the state of the microphone of the operator terminal 120 in black. Alternatively, the supporter terminal 130 displays an icon indicating the state of the microphone of the operator terminal 120 in gray.

[0196] In step S607, when the detection result in step S606 is "microphone abnormality", the operator terminal 120 reports the determination result to the operator 150. Specifically, the operator terminal 120 reports the determination result that the voice data based on the speech of the operator 150 cannot be converted into text data due to a failure of the voice acquisition device 227, or the quality of the converted text data is reduced to the operator 150.

[0197] In step S608, the operator terminal 120 detects the state of communication. In addition, in step S609, when the detection result in step S608 is "communication deterioration", the operator terminal 120 reports the determination result to the operator 150. Specifically, the operator terminal 120 reports the determination result that the voice data based on the speech of the operator 150 cannot be converted into text data due to the deterioration of communication, or the quality of the converted text data is reduced to the operator 150.

[0198] In step S610, the server device 110 performs a two-way communication process. Here, the server device 110 detects the state of communication, makes a determination based on the detection result (the state of communication), determines display content (the state of communication), and sends a display instruction corresponding to the determined display content to the supporter terminal 130.

[0199] In addition, when the server device 110 obtains the detection result of communication deterioration, it is determined that the voice data based on the speech of the operator 150 cannot be converted into text data due to the deterioration of communication, or the quality of the converted text data is reduced. In this case, the server device 110 instructs the supporter terminal 130 to display an icon indicating the state of communication, for example.

[0200] In step S620, the supporter terminal 130 performs a voice output process. Here, the supporter terminal 130 performs a display process corresponding to the display instruction. For example, an icon indicating the communication status between the operator terminal 120 and the server device 110 is displayed.

[0201] In step S610, server device 110 executes a two-way communication process. Here, server device 110 stores the converted text data in association with the determination result.

[0202] in addition, Figure 6 The communication process shown is repeatedly executed while the operator 150 and the supporter 160 continue to communicate using the two-way communication system 100 .

[0203] <Flow of communication processing (2)>

[0204] Next, the flow of the communication process by the two-way communication system 100 , that is, the communication process when the audio data including the voice data based on the speech of the supporter 160 is transmitted from the supporter terminal 130 to the operator terminal 120 will be described. Figure 7 This is a first sequence diagram showing the flow of communication processing when voice data is transmitted from a supporter terminal to an operator terminal in a two-way communication system.

[0205] In step S701, the supporter terminal 130 acquires audio data including voice data based on the speech of the supporter 160. Furthermore, the supporter terminal 130 transmits the acquired audio data to the server device 110.

[0206] In step S710, the server device 110 performs a two-way communication process. The two-way communication process of the server device 110 includes receiving and sending processing of voice data, voice recognition processing of voice data, communication status detection processing, acquisition and determination processing of detection results, determination of display content and display instruction processing, and storage processing of text data and determination results. Here, the server device 110 receives voice data and sends it to the operator terminal 120. In addition, the server device 110 performs voice recognition processing on the voice data and converts it into text data.

[0207] In step S720, the operator terminal 120 performs a voice output process. The voice output process of the operator terminal 120 includes acquisition and output processes of voice data. Here, the operator terminal 120 acquires and outputs voice data.

[0208] In step S702 , the supporter terminal 130 separates the audio data into voice data and ambient sound data.

[0209] In step S703 , the supporter terminal 130 calculates the sound pressure level of the speech data and the sound pressure level of the ambient sound data.

[0210] In step S704 , the supporter terminal 130 detects the state of the ambient sound based on the sound pressure level of the ambient sound data, and transmits the detection result (the state of the ambient sound) to the server device 110 .

[0211] In step S710, the server device 110 performs a two-way communication process. Here, the server device 110 receives the detection result (the state of the ambient sound), makes a determination based on the received detection result (the state of the ambient sound), determines the display content (the state of the ambient sound), and sends a display instruction corresponding to the determined display content to the supporter terminal 130.

[0212] In addition, when receiving the detection result that there is no influence of the ambient sound, the server device 110 determines that the voice data based on the speech of the supporter 160 can be converted into text data or the quality of the converted text data is not reduced. In this case, the server device 110 instructs the supporter terminal 130 to display an icon indicating the state of the ambient sound in blue, for example.

[0213] Furthermore, when receiving the detection result indicating the influence of the ambient sound, the server device 110 determines that the voice data based on the speech of the supporter 160 cannot be converted into text data or the quality of the converted text data is reduced due to the influence of the ambient sound. In this case, the server device 110 instructs the supporter terminal 130 to display an icon indicating the state of the ambient sound in gray, for example.

[0214] In step S705, the supporter terminal 130 performs display processing according to the display instruction from the server device 110. For example, the supporter terminal 130 displays an icon indicating the state of the ambient sound of the supporter terminal 130 in blue. Alternatively, the supporter terminal 130 displays an icon indicating the state of the ambient sound of the supporter terminal 130 in gray.

[0215] In step S706 , the supporter terminal 130 detects the state of the microphone based on the sound pressure level of the ambient sound data and the sound pressure level of the voice data, and transmits the detection result (the state of the microphone) to the server device 110 .

[0216] In step S710, the server device 110 performs a two-way communication process. Here, the server device 110 receives the detection result (microphone status), makes a determination based on the received detection result (microphone status), determines display content (microphone status), and sends a display instruction corresponding to the determined display content to the supporter terminal 130.

[0217] In addition, when receiving the detection result that the microphone is normal, the server device 110 determines that the voice data based on the speech of the supporter 160 can be converted into text data or the quality of the converted text data is not reduced. In this case, the server device 110 instructs the supporter terminal 130 to display an icon indicating the state of the microphone in black, for example.

[0218] Furthermore, when receiving the detection result of microphone abnormality, the server device 110 determines that the voice data based on the speech of the supporter 160 cannot be converted into text data or the quality of the converted text data is reduced due to the failure of the voice acquisition device 227. In this case, the server device 110 instructs the supporter terminal 130 to display an icon indicating the state of the microphone in gray, for example.

[0219] In step S707, the supporter terminal 130 performs display processing according to the display instruction from the server device 110. For example, the supporter terminal 130 displays an icon indicating the state of the microphone of the supporter terminal 130 in black. Alternatively, the supporter terminal 130 displays an icon indicating the state of the microphone of the supporter terminal 130 in gray.

[0220] In step S708 , the supporter terminal 130 detects the state of communication, makes a determination based on the detection result (the state of communication), and determines display content (the state of communication).

[0221] In addition, when the detection result is "communication deterioration", the supporter terminal 130 determines that the voice data based on the speech of the supporter 160 cannot be converted into text data due to the deterioration of communication, or the quality of the converted text data is reduced. In this case, the supporter terminal 130 determines, for example, an icon that clearly indicates the status of communication as the display content (communication status).

[0222] In step S709 , the supporter terminal 130 displays the determined display content. For example, the supporter terminal 130 displays an icon that clearly indicates the state of communication between the supporter terminal 130 and the server device 110 .

[0223] In step S710, server device 110 performs a two-way communication process. Here, server device 110 detects the state of communication and makes a determination based on the detection result (the state of communication). In addition, server device 110 stores the converted text data in association with the determination result.

[0224] in addition, Figure 7 The communication process shown is repeatedly executed while the operator 150 and the supporter 160 continue to communicate using the two-way communication system 100 .

[0225] <Screen example of the supporter terminal>

[0226] Next, an example of a display screen displayed on the display device 231 of the supporter terminal 130 will be described. Figure 8 FIG. 2 is a diagram showing an example of a display screen of a supporter terminal. Figure 8 As shown, the display screen 800 includes an area 810 for displaying the moving image data transmitted from the operator terminal 120 , an area 820 for displaying the status of the operator terminal 120 , and an area 830 for displaying the status of the supporter terminal 130 .

[0227] Among them, in the area 810 , the moving image data captured by the camera device 229 of the operator terminal 120 and transmitted by the communication device 224 of the operator terminal 120 during the two-way communication process is displayed.

[0228] In area 820 , “microphone status”, “ambient sound status”, and “communication status” are displayed as statuses of the operator terminal 120 .

[0229] In the "microphone status" field, an icon indicating the status of the microphone of the operator terminal 120 is displayed. Figure 8 As shown, when the detection result (microphone status) detected in the operator terminal 120 is "microphone normal", the icon indicating the microphone status is displayed in black. In addition, when the detection result (microphone status) detected in the operator terminal 120 is "microphone abnormal", the icon indicating the microphone status is displayed in gray.

[0230] In the "Environmental Sound Status", an icon indicating the status of the ambient sound of the operator terminal 120 is displayed. Figure 8 As shown, when the detection result (state of ambient sound) detected in the operator terminal 120 is "without the influence of ambient sound", the icon indicating the state of ambient sound is displayed in black. In addition, when the detection result (state of ambient sound) detected in the operator terminal 120 is "with the influence of ambient sound", the icon indicating the state of ambient sound is displayed in blue.

[0231] In the "communication status", an icon indicating the communication status between the operator terminal 120 and the server device 110 is displayed. Figure 8 As shown, when the detection result (communication status) detected by the server device 110 is "communication normal", the icon is not displayed. On the other hand, when the detection result (communication status) detected by the server device 110 is "communication deteriorated", the icon is displayed.

[0232] The supporter 160 can understand the following by referring to the area 820 during the communication with the operator 150:

[0233] Whether the operator terminal 120 is in a state suitable for performing speech recognition on speech data based on the speech of the operator 150 ;

[0234] Whether a situation unsuitable for speech recognition has occurred in the operator terminal 120 .

[0235] Furthermore, when the supporter 160 understands that a situation has occurred in the operator terminal 120 that is not suitable for voice recognition of voice data based on the speech of the operator 150, the supporter 160 can understand the cause (failure of the voice acquisition device 227, influence of ambient sound, deterioration of communication). Therefore, the supporter 160 can urge the operator 150 to improve the situation, and can avoid a situation in which it is later determined that part of the speech content has not been converted into appropriate text data.

[0236] Similarly, in the area 830 , “microphone status”, “ambient sound status”, and “communication status” are displayed as the status of the supporter terminal 130 .

[0237] In the “microphone status”, an icon indicating the status of the microphone of the supporter terminal 130 is displayed. Figure 8 As shown, when the detection result (microphone status) detected in the supporter terminal 130 is "microphone normal", the icon indicating the microphone status is displayed in black. In addition, when the detection result (microphone status) detected in the supporter terminal 130 is "microphone abnormal", the icon indicating the microphone status is displayed in gray.

[0238] In the "ambient sound status", an icon indicating the ambient sound status of the supporter terminal 130 is displayed. Figure 8 As shown, when the detection result (the state of the ambient sound) detected by the supporter terminal 130 is "without the influence of the ambient sound", the icon indicating the state of the ambient sound is displayed in black. In addition, when the detection result (the state of the ambient sound) detected by the supporter terminal 130 is "with the influence of the ambient sound", the icon indicating the state of the ambient sound is displayed in blue.

[0239] In the "communication status", an icon indicating the communication status between the supporter terminal 130 and the server device 110 is displayed. Figure 8 As shown, when the detection result (communication status) detected by the supporter terminal 130 is "communication normal", the icon is not displayed. On the other hand, when the detection result (communication status) detected by the supporter terminal 130 is "communication deteriorated", the icon is displayed.

[0240] The supporter 160 can understand the following by referring to the area 830 during the communication with the operator 150:

[0241] Whether the supporter terminal 130 is in a state suitable for performing speech recognition on the speech data based on the supporter's speech;

[0242] Whether a situation in which speech recognition of the supporter's own speech is not suitable has occurred in the supporter terminal 130 .

[0243] Furthermore, when the supporter 160 understands that a situation in which speech recognition is not suitable for speech data based on the supporter's speech has occurred in the supporter terminal 130, the supporter 160 can understand the cause (failure of the speech acquisition device 227, influence of ambient sound, deterioration of communication). Therefore, the supporter 160 can immediately improve the situation and avoid a situation in which it is later determined that part of the speech content has not been converted into appropriate text data.

[0244] Summary

[0245] As can be seen from the above description, the bidirectional communication system 100 as an example of the system according to the first embodiment

[0246] When acquiring voice data, the sound pressure data of the acquired voice data and the sound pressure data of the ambient sound data other than the voice data are calculated. Thus, it is determined whether the acquired voice data can be converted into text data or whether the quality of the converted text data obtained by converting the acquired voice data is reduced.

[0247] The determination result is reported to the operator, or the display content corresponding to the determination result is displayed on the supporter terminal.

[0248] Thus, according to the two-way communication system 100, the operator or the supporter can understand that a situation that is not suitable for voice recognition has occurred.

[0249] [Second embodiment]

[0250] In the first embodiment described above, the case where the process of converting voice data into text data is performed in the server device 110 is described. However, the process of converting voice data into text data may be performed in each client terminal of the operator terminal 120 and the supporter terminal 130. The second embodiment will be described below, centering on the differences from the first embodiment described above.

[0251] <Functional structure of operator terminal>

[0252] First, the functional configuration of the operator terminal 120 in the second embodiment will be described. Fig. 9 is a second diagram showing an example of a functional configuration of an operator terminal. Figure 3 The difference between the functional configuration described above is that the functions of the text data generating unit 910 and the communication unit 920 are different from those of the Figure 3 The function of the communication unit 360 is different.

[0253] The text data generating unit 910 performs speech recognition on the speech data notified from the speech separating unit 320 to convert the speech data into text data, and notifies the communication unit 920 .

[0254] Communication Department 920 Figure 3 In addition to the function of the communication unit 360, it also has a function of transmitting the text data notified from the text data generating unit 910 to the server device 110.

[0255] <Details of functional configuration of server device>

[0256] Next, the functional configuration of the server device 110 in the second embodiment will be described in detail. Fig.10 FIG. 2 is a second diagram showing an example of the details of the functional configuration of the server device. Figure 4 The difference in the details of the functional configuration described is that the function of the communication unit 1010 is different from that of the Figure 4 The function of the communication unit 410 is different, and the voice data acquisition unit 420 and the text data generation unit 440 are not included.

[0257] Communication Department 1010 Figure 4 In addition to the functions of the communication unit 410, it also has the following functions:

[0258] A function of receiving text data transmitted from the operator terminal 120 and storing the received text data in the text data storage unit 450 in association with the determination result determined by the determination unit 430; and

[0259] A function of receiving text data transmitted from the supporter terminal 130 and storing the received text data in the text data storage unit 450 in association with the determination result determined by the determination unit 430 .

[0260] <Functional structure of the supporter terminal>

[0261] Next, the functional configuration of the supporter terminal 130 in the second embodiment will be described. Fig.11 FIG. 2 is a second diagram showing an example of a functional configuration of a supporter terminal. Figure 5 The difference between the functional configuration described above is that the functions of the text data generating unit 1110 and the communication unit 1120 are different from those of the Figure 5 The function of the communication unit 560 is different.

[0262] The text data generating unit 1110 performs speech recognition on the speech data notified from the speech separating unit 520 to convert the speech data into text data, and notifies the communication unit 1120 .

[0263] Communication unit 1120 Figure 3 In addition to the function of the communication unit 560, it also has a function of transmitting the text data notified from the text data generating unit 1110 to the server device 110.

[0264] <Communication process flow (1)>

[0265] Next, the flow of the communication process of the two-way communication system 100 in the second embodiment, that is, the communication process when audio data including voice data based on the speech of the operator 150 is transmitted from the operator terminal 120 to the supporter terminal 130 will be described. Fig.12This is a second sequence diagram showing the flow of communication processing when voice data is transmitted from the operator terminal to the supporter terminal in the two-way communication system. Figure 6 The first sequence diagram described is different in step S1201 and step S1210.

[0266] In step S1201 , the operator terminal 120 performs speech recognition on the speech data and converts the speech data into text data. The operator terminal 120 transmits the converted text data to the server device 110 .

[0267] In step S1210, the server device 110 performs a two-way communication process. The two-way communication process of the server device 110 includes receiving and sending audio data, receiving text data, detecting the state of communication, acquiring and determining the detection result, determining the display content and indicating the display, and storing the text data and the determination result. Here, the server device 110 stores the received text data in association with the determination result.

[0268] <Flow of communication processing (2)>

[0269] Next, the flow of the communication process of the two-way communication system 100 in the second embodiment, that is, the communication process when audio data including voice data based on the speech of the supporter 160 is transmitted from the supporter terminal 130 to the operator terminal 120 will be described. Fig.13 The second sequence diagram shows the flow of communication processing when voice data is transmitted from the supporter terminal to the operator terminal in the two-way communication system. Figure 7 The first sequence diagram described is different in step S1301 and step S1310.

[0270] In step S1301 , the supporter terminal 130 performs speech recognition on the speech data and converts it into text data. Furthermore, the supporter terminal 130 transmits the converted text data to the server device 110 .

[0271] In step S1310, the server device 110 performs a two-way communication process. The two-way communication process of the server device 110 includes receiving and sending audio data, receiving text data, detecting the state of communication, obtaining and determining the detection result, determining the display content and indicating the display, and storing the text data and the determination result. Here, the server device 110 stores the received text data in association with the determination result.

[0272] Summary

[0273] According to the above description, the two-way communication system 100 as an example of the system involved in the second embodiment has the same functions as the first embodiment described above. Furthermore, the two-way communication system 100 as an example of the second embodiment system is configured so that the voice recognition processing for the voice data is performed by each client terminal instead of the server device 110 performing the voice recognition processing for the voice data.

[0274] Thus, according to the second embodiment, the same effects as those of the first embodiment can be achieved, and a situation in which the accuracy of voice recognition is reduced due to noise generated when voice data is transmitted between each client terminal and the server device can be avoided.

[0275] [Third Embodiment]

[0276] In the first and second embodiments described above, the following situations are described:

[0277] 150 workers working at the work site and

[0278] The two-way communication system is applied in a scenario where the supporter 160 who remotely supports the operator 150 communicates with the operator 150 by remotely giving various work instructions to the operator 150. However, the application destination of the two-way communication system is not limited to this, and for example, it can also be applied to other scenarios as long as it is a scenario of remote communication. In the third embodiment, the case where the two-way communication system is applied in a scenario where remote users hold a meeting is described.

[0279] <System structure of two-way communication system>

[0280] First, a system configuration of a bidirectional communication system as an example of a system according to the third embodiment will be described. Fig.14 FIG. 2 is a second diagram showing an example of a system configuration of a two-way communication system. Fig.14 As shown, the two-way communication system 1400 includes a server device 110 and client terminals (Web terminals 1420 and 1430).

[0281] In addition, in this embodiment, the following situation is described:

[0282] ·User 1450 and

[0283] User 1460

[0284] A two-way communication system is applied in conference scenarios.

[0285] The server device 110 is the same as the server device 110 described in the first or second embodiment.

[0286] In addition, the Web terminals 1420 and 1430 are the same as the supporter terminal 130 described in the first or second embodiment.

[0287] In the third embodiment, the text data storage unit 450 (in Fig.14 The text data (not shown) and the determination results may also be downloaded by the user 1450 or 1460 after the meeting, for example, to be used as a meeting record.

[0288] <Example of using text data>

[0289] Next, an example of using text data in a two-way communication system 1400 as an example of a system according to the third embodiment will be described. Fig.15 This is a diagram showing an example of using text data. Fig.15 In the illustrated usage example, the text data management unit 112 further includes a text data correction unit 1510 .

[0290] The text data correction unit 1510 corrects the text data stored in the text data storage unit 450 based on the corresponding determination result. The correction functions of the text data correction unit 1510 include:

[0291] A function to visualize a determination result by adding a mark indicating the cause of a given determination result (failure of the voice acquisition device 227, influence of ambient sound, communication deterioration) to text data; and

[0292] A function to correct a portion of the text data that is related to a given determination result (failure of the voice acquisition device 227, influence of ambient sound, communication deterioration) using the previous and next text data.

[0293] exist Fig.15 In FIG. 1 , reference numeral 1520 indicates an example of text data stored in the text data storage unit 450. Fig.15 As shown, the text data stored in the text data storage unit 450 is stored in a manner that allows identification of which of the Web terminal 1420 and the Web terminal 1430 the text data is converted from the voice data input.

[0294] exist Fig. 9 In the figure, reference numeral 1530 shows a state in which the text data correction unit 1510 adds a mark to the area surrounded by the dotted line in the text data shown by reference numeral 1520 to visualize the determination result.

[0295] Among them, reference numeral 1531 indicates text data when it is determined that the voice data can be converted into text data or the quality of the converted text data is not reduced due to notification of "no influence of ambient sound", "microphone is normal" and "communication is normal".

[0296] On the other hand, reference numeral 1532 shows text data after being marked when it is determined that the voice data cannot be converted into text data due to the influence of the ambient sound or the quality of the converted text data is reduced by being notified of "the influence of the ambient sound". As shown by reference numeral 1532, in the text data correction unit 1510, a mark corresponding to the reason is added to the corresponding position of the text data.

[0297] Similarly, reference numeral 1533 shows text data after being marked when it is determined that the voice data cannot be converted into text data or the quality of the converted text data is reduced due to a failure of the voice acquisition device 227 by being notified of "microphone abnormality". As shown by reference numeral 1533, in the text data correction unit 1510, a mark corresponding to the cause is added to the corresponding position of the text data.

[0298] Similarly, reference numeral 1534 shows text data after being marked when it is determined that the voice data cannot be converted into text data due to the deterioration of communication or the quality of the converted text data is reduced by being notified of "communication deterioration". As shown by reference numeral 1534, in the text data correction unit 1510, a mark corresponding to the cause is added to the corresponding position of the text data.

[0299] Thus, in the third embodiment, the server device 110 adds a mark corresponding to the reason for the determination result to the text data. Thus, according to the third embodiment, the client terminal (Web terminal 1420, 1430) can visualize the determination result associated with the text data when displaying the text data. As a result, according to the third embodiment, even if a part of the text data is not properly voice recognized, the user 1450, 1450 can understand the reason.

[0300] In the above description, a specific example of the function of visualizing the cause of the determination result among the correction functions of the text data correction unit 1510 is cited. However, the function of correcting based on the text data before and after use, for example, can visualize the cause of the determination result.

[0301] The text data indicated by reference numerals 1532 to 1534 are corrected to

[0302] ·Text data indicated by reference numeral 1531 .

[0303] This is because the determination result is displayed during communication, so that the time range in which a situation unsuitable for speech recognition occurs can be shortened (it can be within the time range in which the text data before and after use can be corrected).

[0304] Summary

[0305] As can be seen from the above description, the two-way communication system 1400 as an example of the system according to the third embodiment has the same functions as those of the first or second embodiment described above, and further has the following functions:

[0306] A function to visualize the cause of a decision result associated with text data;

[0307] A function that uses the text data before and after to correct the places related to the judgment results.

[0308] Thus, according to the third embodiment, the same effects as those of the first or second embodiment can be achieved, and even when a part of the text data is not properly voice recognized, the cause can be understood and the text data can be corrected to be proper.

[0309] [Fourth Embodiment]

[0310] In the first and second embodiments described above, the description of the communication process when the dynamic image data is sent to the supporter terminal 130 is omitted, but the dynamic image data may be stored in association with the text data, for example. In this case, for example, the work content of the operator 150 may be determined by analyzing the dynamic image data, and then converted into text information, and the determined work content may be stored in association with the text data.

[0311] Fig.16 FIG3 is a third diagram showing an example of the details of the functional configuration of the server device. Figure 4 The difference from the first figure shown is that a work content specifying unit 1610 is included.

[0312] The work content determination unit 1610 analyzes the dynamic image data notified from the communication unit 410, determines the work content performed by the operator 150, and converts the dynamic image data into text information. In addition, the work content determination unit 1610 stores the determined work content in the text data storage unit 450 in association with the text data and the determination result.

[0313] By storing the work content, text data, and determination results in association with each other in this manner, for example, the supporter 160 can understand the work content performed by the worker 150 and confirm the utterance content of the worker 150 when browsing the text data.

[0314] [Fifth Embodiment]

[0315] In the first and second embodiments described above, it is described that the operator terminal 120 reports the determination result to the operator 150 via the voice output device 228. However, the method of reporting the determination result to the operator 150 is not limited to this. For example, it is also possible to configure an LED or the like at the operator terminal 120 so that it emits light in a color corresponding to the determination result. Alternatively, it is also possible to configure a vibrator or the like at the operator terminal 120 so that it vibrates at a frequency or amplitude corresponding to the determination result.

[0316] In addition, in the first and second embodiments described above, the server device 110 transmits a display instruction corresponding to the determination result for the operator terminal 120 and a display instruction corresponding to the determination result for the supporter terminal 130 to the supporter terminal 130. However, the instruction transmitted by the server device 110 is not limited to the display instruction corresponding to the determination result, and the transmission destination is not limited to the supporter terminal 130. For example, the server device 110 may be configured to transmit a notification instruction corresponding to the determination result to the operator terminal 120.

[0317] In addition, in each of the above embodiments, the influence of ambient sound and microphone abnormality are cited as reasons for failure to convert to text data or reasons for reduced quality of converted text data. However, the reasons for failure to convert to text data or reasons for reduced quality of converted text data are not limited to the influence of ambient sound and microphone abnormality. For example, the influence of voice data preceding the voice data being acquired on the voice data being acquired can be cited. This is because voice data preceding the voice data being acquired may sometimes affect the voice data being acquired due to echo.

[0318] In addition, whether the voice data preceding the voice data being acquired has an influence on the voice data being acquired can be determined by the same mechanism as the influence of the ambient sound. For example, the voice data preceding the voice data being acquired is separated from the voice data being acquired, and the size (sound pressure level) of the voice data preceding the voice data being acquired is calculated, thereby determining whether there is an influence on the voice data being acquired.

[0319] In addition, the display method for displaying the presence or absence of influence on the voice data being acquired may be the same display method as the display method for displaying the presence or absence of influence of the ambient sound, or may be a different display method.

[0320] In addition, in each of the above-mentioned embodiments, the state of the ambient sound is detected by determining whether the sound pressure level of the ambient sound data notified from the sound pressure level calculation unit 330 is above a given threshold value (i.e., an absolute value). However, the method for detecting the state of the ambient sound is not limited thereto. For example, the state of the ambient sound may be detected by determining whether the ratio of the size (sound pressure level) of the voice data to the size (sound pressure level) of the ambient sound data satisfies a given condition (i.e., a relative value). Specifically, when the sound pressure level of the voice data is smaller than the sound pressure level of the ambient sound data, it may be determined that there is an influence of the ambient sound, and when the sound pressure level of the voice data is greater than the sound pressure level of the ambient sound data, it may be determined that there is no influence of the ambient sound.

[0321] In addition, in the above-mentioned embodiments, the details of the method for acquiring sound data by the sound data acquisition unit 310, 510 are not mentioned, but the sound data acquisition unit 310, 510 can also be configured to perform various pre-processing when acquiring sound data. The pre-processing mentioned here can include processing such as removing high-frequency components by performing low-pass filtering on the sound data, and processing such as removing low-frequency components from the sound data to remove AC noise. Alternatively, in the pre-processing, for example, it can also include processing such as removing howling and echo by analyzing the waveform of each given interval.

[0322] In addition, in the above-mentioned embodiments, the use of Figure 8 The status of the operator terminal 120 and the status of the supporter terminal 130 are displayed using icons shown in FIG. However, the display method of the status of the operator terminal 120 and the status of the supporter terminal 130 is not limited to this, and they may be displayed in any other display format.

[0323] In the third embodiment, as a mark indicating the reason for a given determination result added to text data, Fig.15 However, the display method of the mark indicating the reason of a given determination result added to the text data is not limited to this, and it may be displayed in any other display method.

[0324] In addition, in the above-mentioned embodiments, it is described that the client terminal sends the sound data in real time to the server device 110. However, the structure of the client terminal is not limited to this. For example, the client terminal may also be configured to send and sequentially store the sound data in real time, and when there is a request from the server device 110, send the sound data in the range corresponding to the request to the server device 110.

[0325] In addition, in the first and second embodiments described above, the case where the text data generating unit is implemented in the server device 110 and the case where the text data generating unit is implemented in the client terminal (operator terminal 120, supporter terminal 130) are shown. However, the change in the function sharing between the server device 110 and the client terminal (operator terminal 120, supporter terminal 130) is not limited to this. For example, the determination unit 430 of the server device 110 can also be implemented in the client terminal (operator terminal 120, supporter terminal 130).

[0326] In addition, the text data correction unit 1510 described in the third embodiment can also be implemented in the two-way communication system 100 described in the first embodiment. In this case, the supporter terminal 130 can visualize the determination result associated with the text data when downloading and displaying the text data. As a result, according to the fifth embodiment, even if a part of the text data is not properly voice recognized, the supporter 160 can understand the reason.

[0327] Furthermore, in the above-mentioned first embodiment, a case where a wearable terminal is used as the operator terminal 120 is described, but a portable terminal may be used instead of the wearable terminal.

[0328] In the above embodiments, the server device 110 executes the two-way communication program as a single unit. However, when the server device 110 is composed of one or more computers, the two-way communication program may be installed in each of the one or more computers and executed in a distributed computing manner.

[0329] In addition, in each of the above-mentioned embodiments, as an example of a method for installing a client program, a method for installing via network download is mentioned. At this time, although the download source is not specifically mentioned, when installing by this method, the download source can be, for example, a server device that stores the client program in an accessible manner. In addition, the server device can also be a device on the cloud that accepts access from a client terminal (supporter terminal 130, Web terminal 1420, 1430) via the network and downloads the client program on the condition of charging. That is, the server device can also be a device on the cloud that provides services for the client program.

[0330] Although the embodiments have been described above, it should be understood that various changes in form or details may be made without departing from the spirit and scope of the claims.

[0331] This application claims priority based on Japanese Patent Application No. 2022-140702 filed on September 5, 2022, the entire contents of which are incorporated herein by reference.

[0332] Description of symbols

[0333] 100: Two-way communication system

[0334] 110: Server device

[0335] 111: Two-way communication department

[0336] 112: Text Data Management Department

[0337] 120: Operator terminal

[0338] 130: Supporter Terminal

[0339] 310: Audio data acquisition unit

[0340] 320: Voice separation unit

[0341] 330: Sound pressure level calculation unit

[0342] 340: First Detection Department

[0343] 350: Second detection unit

[0344] 360: Ministry of Communications

[0345] 370: Judgment Department

[0346] 380: Voice output unit

[0347] 410: Ministry of Communications

[0348] 420: Voice data acquisition unit

[0349] 430: Judgment Department

[0350] 440: Text data generation unit

[0351] 510: Audio data acquisition unit

[0352] 520: Voice separation unit

[0353] 530: Sound pressure level calculation department

[0354] 540: First Detection Department

[0355] 550: Second detection unit

[0356] 560: Ministry of Communications

[0357] 570: Voice output unit

[0358] 580: Display control unit

[0359] 590: Judgment Department

[0360] 800: Display screen

[0361] 910: Text data generation unit

[0362] 920: Ministry of Communications

[0363] 1010: Ministry of Communications

[0364] 1110: Text data generation unit

[0365] 1120: Ministry of Communications

[0366] 1400: Two-way communication system

[0367] 1420: Web Terminal

[0368] 1430: Web Terminal

[0369] 1510: Text Data Correction Department

[0370] 1610: Operation content determination unit.

Claims

1. A system is a system for converting speech data into text data, wherein: During acquisition of the voice data, one or more control units determine whether the voice data being acquired can be converted into text data or whether the quality of the converted text data obtained by converting the voice data being acquired is degraded, and output the determination result.

2. The system according to claim 1, wherein: The one or more control units output the determination result by clearly indicating the reason why the conversion into text data cannot be performed or the reason why the quality of the converted text data is reduced.

3. The system according to claim 2, wherein: The cause includes any one of the influence of ambient sound around a voice acquisition device that acquires the voice data, the influence of voice data preceding the voice data on the voice data, or a failure of the voice acquisition device.

4. The system according to claim 3, wherein: The one or more control units determine that the quality of the converted text data is reduced due to the influence of the ambient sound when the size of the ambient sound data obtained when the voice data is obtained is greater than a given threshold, or when the ratio of the size of the voice data to the size of the ambient sound data obtained when the voice data is obtained satisfies a given condition.

5. The system according to claim 3 or 4, wherein: When the size of the voice data being acquired and the size of the ambient sound data acquired during the acquisition of the voice data are both smaller than a given threshold, the one or more control units determine that the voice acquisition device cannot convert the voice data into text data due to a failure.

6. A system according to any one of claims 2 to 5, wherein: The causes include deterioration of the communication state of the bidirectional communication.

7. The system according to claim 6, wherein: When the bidirectional communication is disconnected during acquisition of the voice data, the one or more control units determine that the voice data cannot be converted into text data due to deterioration of the communication state.

8. A system according to any one of claims 2 to 7, wherein: The system includes an operator terminal used by an operator, a supporter terminal used by a supporter who remotely supports the operator, and a server device that realizes two-way communication between the operator terminal and the supporter terminal. The control unit included in the server device converts the voice data into text data.

9. The system according to claim 8, wherein: The control unit of the server device determines whether the acquired voice data can be converted into text data or whether the quality of the converted text data obtained by converting the acquired voice data is reduced during the acquisition of the voice data. The control unit included in the supporter terminal outputs the determination result.

10. The system according to claim 9, wherein: The operator terminal has a voice acquisition device for acquiring the voice data. The control unit of the server device determines that the quality of the converted text data is reduced due to the influence of the ambient sound around the voice acquisition device when the size of the ambient sound data acquired during the acquisition of the voice data is greater than a given threshold.

11. The system according to claim 9, wherein: The operator terminal has a voice acquisition device for acquiring the voice data. When the size of the voice data being acquired and the size of the ambient sound data acquired during the acquisition of the voice data are both smaller than a given threshold, the control unit of the server device determines that the voice acquisition device cannot be converted into text data due to a failure.

12. The system according to claim 9, wherein: When the control unit of the server device determines that the two-way communication is disconnected because no voice data is sent from the operator terminal within a certain period of time, it is determined that the voice data obtained by the control unit of the operator terminal cannot be converted into text data due to the deterioration of the communication state of the two-way communication.

13. A system according to any one of claims 10 to 12, wherein: The control unit included in the supporter terminal displays the determination result using an icon that clearly indicates a reason why the conversion into text data cannot be performed or a reason why the quality of the converted text data is degraded.

14. The system according to claim 10 or 11, wherein: The control unit of the operator terminal determines whether the acquired voice data can be converted into text data or whether the quality of the converted text data obtained by converting the acquired voice data is reduced during the acquisition of the voice data, and reports the determination result through at least one of voice, light, and vibration.

15. A system according to any one of claims 9 to 14, wherein: The control unit of the server device stores the text data in association with the determination result.

16. The system of claim 15, wherein: The control unit included in the supporter terminal reads the stored text data and displays it in a display format according to the corresponding determination result.

17. A system according to any one of claims 8 to 16, wherein: The control unit included in the server device uses the previous and next text data to correct the parts that cannot be converted into text data or the parts with reduced quality.

18. A system according to any one of claims 9 to 17, wherein: The control unit included in the server device specifies the work content of the operator based on the image captured by the operator terminal, and stores the specified work content in association with the determination result.

19. A system according to any one of claims 9 to 18, wherein: The control unit included in the operator terminal stores the acquired voice data, and when there is a request from the server device, transmits the voice data in a range corresponding to the request to the server device.

20. A supporting method for a system for converting speech data into text data, comprising the following steps: a determination step of determining, during acquisition of the voice data, whether the voice data being acquired can be converted into text data, or whether the quality of the converted text data obtained by converting the voice data being acquired is reduced; and Output process, output judgment result.

21. A server device comprising a control unit that realizes bidirectional communication between an operator terminal used by an operator and a supporter terminal used by a supporter who remotely supports the operator, wherein: The control unit, Convert voice data into text data, And in the acquisition of voice data by the operator terminal or the supporter terminal, when it is determined whether the voice data being acquired can be converted into text data, or whether the quality of the converted text data obtained by converting the voice data being acquired is reduced, an instruction corresponding to the determination result is sent to the operator terminal or the supporter terminal.

22. A communication program for causing a control unit of a server device that realizes two-way communication between an operator terminal used by an operator and a supporter terminal used by a supporter who remotely supports the operator to execute the following steps: A conversion step, converting the voice data into text data; A sending process, in which, during the acquisition of voice data by the operator terminal or the supporter terminal, when it is determined whether the voice data being acquired can be converted into text data, or whether the quality of the converted text data obtained by converting the voice data being acquired is reduced, an instruction corresponding to the determination result is sent to the operator terminal or the supporter terminal.

Images