Program, storage medium, control method for information processing device, and information processing device.

Abstract

To allow checking of the details of voice messages in a chat room even when they cannot be played back.SOLUTION: An information processing apparatus comprises display means that can display voice messages transmitted and received in a chat room, and comprises: communication means that transmits and receives voice messages in the chat room where a plurality of users has a chat with each other; transcription means that transcribes a content of the voice messages into text; instruction means for giving an instruction to transcribe the content of the voice messages into text; and control means that, when the instruction means is operated to give the instruction, causes the display means to display character strings obtained by the transcription means transcribing the content of the voice messages into text in the chat room in a mode different from normal text messages in the chat room.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] The present invention relates to Program, storage medium, control method for information processing device, and information processing device. .

Background Art

[0002] In recent years, communication between users using chat systems has become widespread. In such chat systems, services are provided that enable communication by transmitting not only text but also voice messages on the transmitting side and playing them back on the receiving side. Further, the technique disclosed in Patent Document 1 is an apparatus for handling voice data, in which a recognition processing unit converts input voice data into text data, and a display unit displays the text data as recognizable characters (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of Invention

Problems to be Solved by the Invention

[0004] However, the voice message displayed in the chat window of the chat system is only an icon for playback, and there is a problem that the content cannot be confirmed until the voice is played back. For example, in places where sound output is prohibited, the message cannot be confirmed immediately. On the other hand, as in the apparatus disclosed in Patent Document 1 within the chat system, there is also a problem that if all voice data is converted into characters and displayed, the visibility is degraded.

[0005] Accordingly, an object of the present invention is to make it possible to confirm the content of a voice message in a chat room even when the voice message cannot be played back.

Means for Solving the Problems

[0006] To achieve the above objective, one aspect of the present invention is: affection A program that causes a computer to execute a control method for an information processing device, wherein the control method is: The information processing device Received Ta sound The process of converting the content of a voice message into text, In association with the aforementioned voice message A display step, which displays the string converted into text by the above text conversion step, The aforementioned string, The aforementioned voice message It includes an object display step that displays an object that performs an action to post as a text message that is not associated with anything, Based on the selection of the object, the string is added as the text message, which is displayed separately from the voice message. This program is characterized by the following features. [Effects of the Invention]

[0007] According to the present invention, even if it is not possible to play back audio messages in a chat room, the content of those messages can be confirmed. [Brief explanation of the drawing]

[0008] [Figure 1] This is a system configuration diagram of this embodiment. [Figure 2] This is a block diagram of the chat application server in this embodiment. [Figure 3] This is a block diagram of the mobile terminal in this embodiment. [Figure 4] This is an explanatory diagram of the operation screen of the chat application in this embodiment. [Figure 5] This is a flowchart illustrating the operation of the chat server in this embodiment. [Figure 6] This is a flowchart illustrating the receiving operation of the mobile terminal in this embodiment. [Figure 7] This is a flowchart illustrating the message display operation of the mobile terminal in this embodiment. [Figure 8] This is a flowchart illustrating the read receipt information reception operation of the mobile terminal in this embodiment. [Figure 9]This is an explanatory diagram of the chat room management table in the present embodiment. [Figure 10] This is an explanatory diagram of the sequence of the entire chat system in the present embodiment. [Figure 11] This is a flowchart for explaining the operation of the mobile terminal in the first embodiment. [Figure 12] This is an explanatory diagram of the display of the chat room in the first embodiment. [Figure 13] This is an explanatory diagram of the display of the chat room in the first embodiment. [Figure 14] This is an explanatory diagram of the display of the chat room in the first embodiment. [Figure 15] This is a flowchart for explaining the operation of the mobile terminal in the second embodiment. [Figure 16] This is a flowchart for explaining the operation of the mobile terminal in the second embodiment. [Figure 17] This is an explanatory diagram of the display of the chat room in the second embodiment. [Figure 18] This is a flowchart for explaining the operation of the mobile terminal in the second embodiment. [Figure 19] This is a flowchart for explaining the operation of the mobile terminal in the third embodiment. [Figure 20] This is an explanatory diagram of the display of the chat room in the third embodiment. [Figure 21] This is a flowchart for explaining the operation of the mobile terminal in the fourth embodiment. [Figure 22] This is an explanatory diagram of the display of the chat room in the fourth embodiment. [Modes for Carrying Out the Invention]

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the configurations described in the following embodiments are merely illustrative, and the scope of the present invention is not limited by the configurations described in the embodiments. First, the first embodiment of the present invention will be described.

[0010] <First Embodiment> <System Configuration> Figure 1 is a configuration diagram showing an example of a system configuration according to an embodiment of the present invention. This system includes a communication base station 105 and a chat application server 107, which are connected to each other via the Internet 106 so that they can communicate with one another. The communication base station 105 communicates the required information with a mobile terminal 102 owned by a sender 100 and a mobile terminal 104 owned by a receiver 103.

[0011] Sender 100 sends a "voice message" using a chat application installed on mobile terminal 102. Next, the "voice message" sent by sender 100 is transmitted to the chat application server 107 via the internet 106 through the communication base station 105. Next, the chat application server 107 processes the received "voice message" and sends it to the destination. Finally, the "voice message" sent by the chat application server 107 is transmitted to the mobile terminal 104 owned by recipient 103 via the internet 106 and the communication base station 105.

[0012] <Hardware configuration of chat application server 107> Figure 2 is a configuration diagram showing an example of the configuration of the chat application server 107 according to this embodiment. The chat application server 107 comprises a control unit 201, a storage unit 202, and a network interface 211, which are connected to each other by a system bus 210 so that they can send and receive the necessary information.

[0013] <Control Unit 201> The control unit 201 reads and executes the control program 203, which is non-volatilely stored in the memory unit 202. This enables the various functions required for the chat application server 107. The control unit 201 is composed of one or more processors (CPU, DSP, etc.). The control unit 201 also includes a chat processing unit 209, which comprehensively controls each device connected to the system bus 210. The chat processing unit 209 interprets and responds to messages received from the chat application of the mobile terminal 102. In this way, the chat application server 107 has an automated dialogue function and also functions as a "chatbot".

[0014] <Storage section 202> The memory unit 202 (storage means) is used as internal storage. The memory unit 202 stores control programs 203, text data 204, image data 205, audio data 206, video data 207, registered user data 208, and system software, etc. The memory unit 202 is composed of storage devices such as HDD (hard disk drive), SSD (solid state drive), RAM, and ROM.

[0015] Text data 204 is the text data of messages posted by users or chatbots on the chat. Image data 205 is image data posted by users on the chat. Audio data 206 is the audio data of voice messages posted by users on the chat. Video data 207 is the video data of video messages posted by users on the chat. Registered user data 208 is a list of user ID and password combinations required for each user to log in to the chat application.

[0016] <Network Interface> Network I / F211 is an interface for network communication, connected to the Internet 106 via a LAN cable or the like. For example, a well-known network card can be used.

[0017] <Configuration of mobile terminal 102 and mobile terminal 104> Figure 3 shows the block diagram of mobile terminals 102 and 104. Note that mobile terminals 102 and 104 are examples of information processing terminals with communication capabilities, and are terminals that can be used in any location thanks to features such as wireless communication capabilities. Examples include smartphones, tablet terminals, and notebook PCs.

[0018] Both mobile terminals 102 and 104 have the same configuration, and the configuration of one of them, mobile terminal 102, will be described. Mobile terminal 102 is equipped with a system bus 308. A control unit 301, a storage unit 302, an input / output unit 303, a display unit 304 (display means), a microphone 305, a speaker 306, and a network interface 307 are connected to the system bus 308. Each device connected to the system bus 308 is capable of sending and receiving the necessary information from each other.

[0019] The control unit 301 reads and executes the control program stored non-volatilely in the memory unit 302. This enables various functions required by the mobile terminal 102. The control unit 201 is configured to have one or more processors (CPU, DSP, etc.). The control unit 301 also comprehensively controls each device connected to the system bus 308.

[0020] The memory unit 302 is used as internal storage. The memory unit 302 stores text data, audio data, image data, system software, etc. The memory unit 302 is composed of storage devices such as HDD (hard disk drive), SSD (solid state drive), RAM, and ROM.

[0021] The input / output unit 303 consists of an LCD touch panel or the like, and acquires user-operated information and sends the acquired information to the control unit 301. The input / output unit 303 also outputs the processing results of the control unit 301. User input may be implemented using hardware input devices such as switches or keyboards. The method for detecting touch panel input can be any common detection method, such as resistive touch, infrared touch, electromagnetic induction touch, or capacitive touch.

[0022] The display unit 304 displays information according to the image data. The display unit 304 can also display an operation screen, providing a user interface (UI) to the user. The microphone 305 is for inputting audio data. The speaker 306 is for outputting audio data. In this embodiment, audio output is performed using the speaker built into the mobile terminal 102. Alternatively, the control unit 301 may send audio to an external audio output device such as an external earphone or speaker connected to the mobile terminal 102 and output it. The network interface 307 is an interface for connecting to the internet 106 and performing network communication.

[0023] <Screenshot of the operation within a chat application on a mobile device> Figure 4(a) is an explanatory diagram of the operation screen of a chat application running on mobile terminals 102 and 104. At this time, the chat application is communicating with the chat application server 107 as needed. In the operation screen of Figure 4(a), messages 401 and 402 from user A and user B are displayed chronologically in the vertical direction of the screen. In this embodiment, the message 402 from the mobile terminal operator is placed on the right side of the display screen, but the display position of the user's message may be placed on the left side of the display screen.

[0024] The display screen shown in Figure 4(a) also displays the word "Read" (403) to determine whether or not User B has viewed the message sent by User A. Additionally, a message input field 404 is displayed at the bottom of the display screen shown in Figure 4(s). In the message input field 404, users can select text input or image data stored in the memory unit 302 and send it to the chat server. This sends the message to User B.

[0025] Figure 4(b) is an explanatory diagram of the operation screen of a chat application that includes voice messages. Voice messages 405 are displayed separately from text messages, and a button (button icon) 406 for playing the audio is displayed. When the user presses the button to instruct the system to play the audio, the control unit 301 of mobile terminal 102 and mobile terminal 104 controls the audio playback. As a result, the audio is output from the speaker 306.

[0026] Figure 4(c) is an explanatory diagram of the operation screen for entering a voice message. When the voice message input field 407 is pressed, the chat application starts recording voice. The user inputs voice into the microphone 305 of mobile terminal 102 or mobile terminal 104. To finish inputting, press the voice message input field again. The chat application saves the voice message data to the storage unit 302 by stopping voice recording. Then, according to the user's instructions, the voice message is sent to the chat server, and the voice message is sent to the recipient.

[0027] <Figure 5: Operation of chat application server 107> Next, with reference to Figure 5, the operation of the message sending and receiving process in the chat application server 107 will be explained. The operation of the flowchart shown in Figure 5 is realized by the control unit 201 (CPU) reading the control program 203 stored in the memory unit 202 into RAM or the like and executing it. First, in S501, the control unit 201 maintains a chat room containing multiple users. Next, in S502, the control unit 201 receives a message from the chat application of the mobile terminal 102 (message sending terminal).

[0028] Next, in S503, the control unit 201 stores the received message in the storage unit 202. Then, in S504, the control unit 201 sends the received message to another user in the chat room. The chat application server 107 operates in this manner.

[0029] <Figure 6: Processing by mobile devices upon receiving a message> Next, with reference to Figure 6, the operation of the mobile terminal 104 when a message is received will be explained. The operation of the flowchart shown in Figure 6 is realized by the control unit 301 (CPU) reading the program stored in the memory unit 302 into RAM or the like and executing it. First, in S601, the control unit 301 receives a message from the chat application server 107. Next, in S602, the control unit 301 notifies the corresponding chat room of the message based on the chat rule ID that identifies the chat room contained in the message.

[0030] Next, in S603, the control unit 301 determines whether or not it is necessary to display the message received on the UI (user interface) and notify the user. The user has set in advance whether or not to send a notification, and this setting is stored in the storage unit 302.

[0031] Then, in S603, if the control unit 301 determines that notification is necessary (Yes), the process proceeds to S604. On the other hand, if the control unit 301 determines that notification is unnecessary (No), the process ends. Next, in S604, the control unit 301 creates the notification. Then, in S605, the control unit 301 displays the notification on the display unit 304 and, depending on the settings, notifies the user audibly using the speaker 306. The mobile terminal 104 that receives the message operates in this manner.

[0032] <Figure 7: Processing on mobile devices when a message is displayed> Next, with reference to Figure 7, the operation of the mobile terminal 104 when displaying messages in a chat room will be explained. The operation of the flowchart shown in Figure 7 is realized by the control unit 301 reading a program stored in the memory unit 302 into RAM or the like and executing it. First, in S701, the control unit 301 receives a request from the user to display a chat room. Next, in S702, the control unit 301 displays the selected chat room on the display unit 304. Specifically, the chat room is displayed as a UI.

[0033] Next, in S703, the control unit 301 stores the most recent unread message among the displayed messages in the storage unit 302 as a "read message". Then, in S704, the control unit 301 sends the read message stored in the storage unit 302 to the chat application server 107. The mobile terminal processing when a message is displayed is performed in this manner.

[0034] <Figure 8: Read receipt reception operation on mobile devices> Next, with reference to Figure 8, the operation of receiving read information in the mobile terminal 102 will be explained. The operation shown in the flowchart in Figure 8 is realized when the control unit 301 reads the program stored in the memory unit 302 into RAM or the like and executes it.

[0035] First, in S801, the control unit 301 receives read receipt information from the chat application server 107. Next, in S802, the control unit 301 updates the read receipt information for the corresponding chat room in the chat management table 901 using the chat room ID that identifies the chat room included in the read receipt information. The read receipt information reception operation is performed by these actions.

[0036] <Figure 9: Chat Management Table> Figure 9 is an explanatory diagram of an example of the "chat management table" updated in S802. The chat management table 901 is stored in the storage unit 302. The chat management table 901 stores the chat room ID (chat room identifier) ​​and read status information in association with each other.

[0037] In other words, the chat management table 901 manages information for each chat room and holds read status information 902, which indicates how many messages have been read by the recipient in each chat room. Based on this information, read status information 403 is displayed when the chat room is shown. For example, in chat room "A", since the read status information 902 is "100", it indicates that "100" messages have been read.

[0038] <Figure 10: Sequence of the entire chat system> Next, referring to Figure 10, the sequence of the entire chat system using the chat application server 107, mobile terminal 104, and mobile terminal 104 will be described. First, sender 100 operates mobile terminal 104 to send a "message" to the chat application server 107. In response, the chat application server 107 adds the "message" to the chat management table 901 and sends the "message" to recipient 103's mobile terminal 104.

[0039] Next, recipient 103 operates on mobile device 104 to display the chat room and confirm the "message". In response, mobile device 104 sends "read receipt information" to chat application server 107. Then, chat application server 107 sends "read receipt information" to mobile device 102. As a result, sender 100 can recognize that the "message" has been acknowledged. This completes the sequence of events for the entire chat system.

[0040] <First Embodiment> Next, referring to Figure 11, the process of transcribing (converting to text) voice messages in the chat room of the mobile terminal 104 and displaying them at the user's instruction will be explained. The operation shown in the flowchart of Figure 11 can be realized by the control unit 301 reading a program stored in the memory unit 302 into RAM or the like and executing it. First, in S1101, the control unit 301 receives a request from the user to display a chat room. Next, in S1102, the control unit 301 displays the selected chat room on the display unit 304.

[0041] Next, in S1103, the control unit 301 stores the most recent unread message among the displayed messages in the storage unit 302 as a read message. Then, in S1104, the control unit 301 sends the read messages stored in the storage unit 302 to the chat application server 107.

[0042] Next, in S1105, the control unit 301 determines whether or not there is an instruction to transcribe the voice message (text conversion instruction). If the control unit 301 determines that there is a transcription instruction (Yes), the process proceeds to S1106. On the other hand, if the control unit 301 determines that there is no transcription instruction (No), the operation shown in Figure 11 is terminated.

[0043] In S1105, the transcription instruction (instruction means) can be configured such that transcription is instructed when an action is performed on the touch panel, such as a long press or slide operation, on the relevant voice message. However, it is not limited to this, and for example, a dedicated instruction button such as the transcription button 1301 (Figure 13) may be displayed, and the user may give the transcription instruction by performing an action on this button. Figure 13 shows an example in which a dedicated transcription button 1301 (the word "text") for transcription is displayed within the display showing the voice message of the mobile terminal 102. Note that the dedicated button is not limited to the one shown in Figure 13.

[0044] Thus, the control unit 301 (instruction means) can be configured to give instructions to convert the content of a voice message into text by performing actions including long-press operations and slide operations on the voice message.

[0045] Next, in S1106, the control unit 301 converts the selected voice message into text by performing voice analysis. Then, in S1107, the control unit 301 displays the text string 1201 converted in S1106 on the display unit 304, linked to the voice message in the chat room, in a display mode different from the normal message display. Figure 12 shows how the text string (1201) is displayed on the chat room display screen shown on the mobile terminal 102. In the linked display example shown in Figure 12, a text display space is linked below the display item showing the voice message, and the text string is displayed.

[0046] Next, in S1108, the control unit 301 determines whether or not there is an "erasure instruction" from the erase instruction unit (erasure instruction means) for the transcript display 1201 shown in S1107. If the control unit 301 determines that there is an "erasure instruction" (Yes), the process proceeds to S1109. On the other hand, if the control unit 301 determines that there is no "erasure instruction" (No), the process returns to S1108 to check for a transcript erase instruction.

[0047] Thus, when the control unit 301 (control means) receives a command to erase the text display of the voice message, it erases the text display of the voice message shown on the display unit 304. The "erasure command" for the transcription in S1108 can be performed by an action on the touch panel, such as a long press or slide operation on the transcription display, just like the transcription command. However, it is not limited to this, and the "erasure command" can also be given by displaying a dedicated erase command button and having the user tap this button, or by tapping an area other than the transcription display. Then, in S1109, the control unit erases the transcription display 1201.

[0048] In the first embodiment, the mobile terminal 102 performs the process of converting voice messages to text, but it is also possible to request the conversion to be performed by the chat application server 107 and receive the text information from the chat application server 107, which has the function of converting to text. Then, in S1107, the converted text string may be displayed. Note that whether the voice message was sent by the user or received by the other party, it is possible to associate a "headline" with it and display it according to the operation in Figure 11.

[0049] If an audio message is long, the transcribed text may be lengthy. In such cases, displaying the entire text may reduce readability. In such cases, as shown in Figure 14, part of the text may be displayed by allowing the user to slide through the text within the transcription display. Figure 14 shows an example where the text displayed in the chat room of the mobile terminal 102 is long and does not fit in the text display space. In this case, if the text does not fit in the text display space, the user can slide the text to sequentially display the parts that are not currently displayed.

[0050] Thus, if the text string converted by the text conversion unit does not fit within the text display space, the control unit 301 moves the string within the text display space so that it is visible, in response to the acceptance of a slide operation (specific operation).

[0051] As explained above, displaying a transcript of an audio message allows users to check its content even when the audio message cannot be played back. Furthermore, since the transcription is performed at the user's instruction, it provides even greater user convenience.

[0052] <Summary of the First Embodiment> Based on the above, the following configuration is provided. First, the control unit 301 (communication means) sends and receives voice messages in a chat room where multiple users are chatting. Next, the control unit 310 (text conversion means) converts the content of the voice messages into text. Then, when a transcription instruction is given, the control unit 301 (control means) displays the string of text generated by the control unit 301 (text conversion means) in the chat room in a manner different from that of normal text messages in the chat room.

[0053] Furthermore, the control unit 301 (control means) can also be configured to associate the text strings converted into text by the text conversion unit with voice messages and display them on the display unit 304 (display means) that displays the chat room.

[0054] Alternatively, the system may be configured to send a request to the chat application server 107 to process the "text conversion," receive the "text data" information from the chat application server 107, and display the information on the mobile terminal 102. In other words, the control unit 301 (request sending means) sends information indicating a request to convert the voice message to text to the chat application server 107 (external server), which is capable of generating a string of characters that converts the voice message to text. In response, the control unit 301 (receiving means) receives the text conversion information sent by the chat application server 107 (external server). The control unit 301 (control means) can then link the text string indicated by the received text conversion information to the voice message and display it in the chat room. With this configuration, the chat application server 107, which has high processing power, can accurately generate text data and link it to the voice message.

[0055] <Second Embodiment> In the first embodiment, a method was described in which a text string derived from an audio message is displayed in a manner different from the normal method of displaying messages. In this second embodiment, an operation is described in which the transcribed text string is displayed in the same way as an existing message.

[0056] <Figure 15: Displaying the transcribed text as a regular message> Referring to Figure 15, the operation of transcribing voice messages in the chat room of the mobile terminal 102 and posting them to the message field according to the user's instructions will be explained. The operation of the flowchart shown in Figure 15 is realized by the control unit 301 reading a program stored in the memory unit 302 into RAM or the like and executing it. First, in S1501, the control unit 301 receives a request from the user to display a chat room. Next, in S1502, the control unit 301 displays the selected chat room on the display unit 304.

[0057] Next, in S1503, the control unit 301 stores the most recent unread message among the displayed messages in the storage unit 302 as a "read message". Next, in S1504, the control unit 301 sends the "read messages" stored in the storage unit 302 to the chat application server 107. Next, in S1505, the control unit 301 determines whether or not there is an instruction to transcribe the voice message. If the control unit 301 determines that there is an instruction to transcribe (Yes), the process proceeds to S1506. On the other hand, if the control unit 301 determines that there is no instruction to transcribe (No), the operation ends.

[0058] Next, in S1506, the control unit 301 performs speech analysis on the selected voice message and converts it to text. Then, in S1507, the control unit 301 displays the string converted to text in S1506 as a text message 1701 on the chat room display screen, just like a normal message. Figure 17 shows the display screen of the display unit 304 of the mobile terminal 102, and the string indicated by reference numeral 1701 is displayed like a normal chat message (post) without being linked to a voice message.

[0059] Furthermore, to distinguish the transcribed text message from other text messages, an indicator 1702 may be added to the message to indicate that it has been transcribed (converted to text). Alternatively, the color, shape, etc., of the transcribed message may be changed to make it easily distinguishable from text posted in a regular chat. In this way, the control unit 301 (control means) can display an indicator on the display unit 304 indicating that the corresponding voice message has been converted to text for each transcribed text string.

[0060] <Figure 16: When entering a text string into the message field> In the flowchart shown in Figure 15, the transcribed text is displayed directly in the message field. However, referring to Figure 16, we will explain the operation of first entering text into the message input field. The flowchart shown in Figure 16 can be realized by the control unit 301 reading the program stored in the memory unit 302 into RAM or the like and executing it. First, in S1601, the control unit 301 receives a request from the user to display a chat room. Next, in S1602, the control unit 301 displays the selected chat room on the display unit 304.

[0061] Next, in S1603, the control unit 301 stores the most recent unread message among the displayed messages in the storage unit 302 as a read message. Next, in S1604, the control unit 301 sends the "read messages" stored in the storage unit 302 to the chat application server 107. Next, in S1605, the control unit 301 determines whether or not there is an instruction to transcribe the voice message. If the control unit 301 determines that there is an instruction to transcribe (Yes), it proceeds to S1606. On the other hand, if the control unit 301 determines that there is no instruction to transcribe (No), it terminates the operation.

[0062] Next, in S1606, the control unit 301 performs speech analysis on the selected voice message and converts it into text. Then, in S1607, the control unit 301 inputs the text converted in S1606 into the message input field 404 (posting field: see Figure 4). Once the text is entered into the message input field 404, the content of the voice message becomes verifiable as text.

[0063] Next, in S1608, the control unit 301 determines whether or not there is an instruction to post a message. If the control unit 301 determines that there is an instruction to post (Yes), the process proceeds to S1609. On the other hand, if the control unit 301 determines that there is no instruction to post (No), the process ends. Then, in S1609, the control unit 301 sends the message to the chat application server 107. After the message is posted, other users who receive the message can also view the transcribed text.

[0064] In the flowcharts shown in Figures 15 and 16, the transcribed text is displayed in the message field. However, after the transcription is performed, the sender may delete the original voice message. In such cases, the voice message itself is deleted from the chat room, but the transcribed message remains, resulting in the retention of unnecessary or incorrect messages.

[0065] <Figure 18: Deletion of voice message> Therefore, referring to Figure 18, we will explain the operation of deleting the transcribed text message in conjunction with deleting the original audio message. The operation of the flowchart shown in Figure 18 is realized by the control unit 301 reading the program stored in the memory unit 302 into the RAM 302 or the like and executing it. First, in S1801, the control unit 301 receives a request from the user to display a chat room. Next, in S1802, the control unit 301 displays the selected chat room on the display unit 304.

[0066] Next, in S1803, the control unit 301 stores the most recent unread message among the displayed messages in the storage unit 302 as a "read message". Next, in S1804, the control unit 301 sends the "read messages" stored in the storage unit 302 to the chat application server 107. Next, in S1805, the control unit 301 determines whether or not there is a "delete instruction" for the voice message. If the control unit 301 determines that there is a "delete instruction" (Yes), it proceeds to S1806. On the other hand, if the control unit 301 determines that there is no "delete instruction" (No), it terminates the operation. The "delete instruction means" that gives a "delete instruction" for a voice message may include, but is not limited to, actions such as touching a dedicated button displayed on the display unit 304, or multiple taps or long presses on the displayed voice message.

[0067] Next, in S1806, the control unit 301 determines whether or not a text message exists that has been transcribed from the audio message for which a "deletion instruction" was given. If the control unit 301 determines that such a message exists (Yes), the process proceeds to S1809. On the other hand, if the control unit 301 determines that such a message does not exist (No), the process proceeds to S1807.

[0068] Next, in S1807, the control unit 301 deletes the selected voice message and displays the chat room from which the voice message was deleted on the display unit 304. Next, in S1808, the control unit 301 sends information about the deleted voice message to the chat application server 107.

[0069] Meanwhile, in S1809, the control unit 301 deletes the "text message" that was transcribed from the selected voice message and displays the chat room from which the transcribed message was deleted on the display unit 304. Then, in S1810, the control unit 301 sends information about the deleted transcribed message to the chat application server 107.

[0070] <Summary of the second embodiment> As explained above, the system is equipped with a message input field 404 (input means) for entering regular text and displaying it in the chat room. The control unit 301 (control means) automatically inputs the text string of the voice message into the message input field 404. This allows the text string corresponding to the voice message to be automatically posted in the chat room, enabling automatic posting in both voice and text. Furthermore, when a deletion instruction is given by the deletion instruction unit (deletion instruction means) that issues a deletion instruction for the voice message, the control unit 301 (control means) deletes the voice message and also deletes the text string corresponding to that voice message. This allows the transcribed text message to be deleted in conjunction with the deletion of the original voice message.

[0071] <Third Embodiment> In the second embodiment, a method for converting an audio message to text and displaying it all at once was described. However, if the audio message is long, it may take a long time to convert it to text, and a large amount of text may be displayed, making it difficult to read. Therefore, this third embodiment describes an operation in which transcription is performed for a predetermined time or a predetermined number of characters when a transcription instruction for an audio message is given.

[0072] <Figure 19: Transcription operation for only the set amount> Referring to Figure 19, the operation of converting a predetermined amount of text for a predetermined time or number of characters when a "transcription instruction" is received for the mobile terminal 104 will be explained. The operation of the flowchart shown in Figure 19 can be realized by the control unit 301 reading a program stored in the memory unit 302 into RAM or the like and executing it. First, in S1901, the control unit 301 receives a request from the user to display a chat room. Next, in S1902, the control unit 301 displays the selected chat room on the display unit 304.

[0073] Next, in S1903, the control unit 301 stores the most recent unread message among the displayed messages in the storage unit 302 as a "read message". Next, in S1904, the control unit 301 sends the "read messages" stored in the storage unit 302 to the chat application server 107. Next, in S1905, the control unit 301 determines whether or not there is an instruction to transcribe the voice message. If the control unit 301 determines that there is an instruction to transcribe (Yes), it proceeds to S1906. On the other hand, if the control unit 301 determines that there is no instruction to transcribe (No), it terminates the operation.

[0074] Next, in S1906, the control unit 301 performs speech analysis on the selected voice message for a set amount and converts it to text. The amount of text to be transcribed in S1906 is a roughly pre-set value stored in the storage unit 302. For example, it may be set to "10 (%): predetermined amount" of the message duration, or "10 (seconds): predetermined time". Alternatively, a number of characters, such as "30 characters", may be set. In this way, the storage unit 302 stores information for setting the amount of text to be transcribed in a way that allows for modification.

[0075] Next, in S1907, the control unit 301 displays a transcript of the text converted in S1906 on the display unit 304, linked to the voice message in the chat room, in a display mode different from the normal message display. Next, in S1908, the control unit 301 determines whether or not there is a "delete instruction" for the "transcript display (converted string of text)" displayed in S1907. If the control unit 301 determines that there is a transcript delete instruction (Yes), proceed to S1910. If the control unit 301 determines that there is no transcript "delete instruction" (No), proceed to S1909.

[0076] Next, in S1909, the control unit 301 determines whether or not there is a "transcription update instruction". If the control unit 301 determines that there is a "transcription update instruction" (Yes), the process proceeds to S1906. On the other hand, if the control unit 301 determines that there is no "transcription update instruction" (No), the process proceeds to S1908.

[0077] In S1909, the "transcription update instruction" can be performed by taking an action on the touch panel, such as long-pressing or sliding the transcription display. Alternatively, a dedicated instruction button may be displayed for the user to input, as shown in Figure 20 as the "transcription update button 2001 (update instruction unit)". Figure 20 is an explanatory diagram of the display screen of the display unit 304 of the mobile terminal 102, in which a triangular display object (2001) is displayed to the right of the text display space on the display screen, and this display object is the "transcription update button 2001". Then, in S1910, the control unit 301 erases the transcribed text from the display unit 304.

[0078] <Summary of the Third Embodiment> As explained above, the control unit 301 (text transcription means) transcribes the content of the voice message into text for a predetermined amount. Furthermore, when an update instruction is given by operating the transcription update button 2001 (update instruction means), the control unit 301 (text transcription means) again transcribes the content of the voice message for the predetermined amount. Therefore, it becomes possible to transcribe the voice message for the predetermined amount desired by the user. In addition, if there is sufficient time, the voice message will be transcribed again for the predetermined amount, making it highly convenient. Thus, by displaying the transcript for the predetermined amount of the voice message, it becomes possible to check the content of the voice message even if the entire voice message cannot be played back.

[0079] <Fourth Embodiment> There are times when you might want to save the transcribed text as a memo or copy it to use in another process. This fourth embodiment describes an operation in which another action is performed on the transcribed and displayed string.

[0080] <Figure 21: Editing Operation> Referring to Figure 21, the operation of the mobile terminal 102 when an action instruction is given for the transcription display will be explained. The operation shown in the flowchart in Figure 21 can be achieved by the control unit 301 reading the program stored in the memory unit 302 into RAM or the like and executing it.

[0081] First, in S2101, the control unit 301 receives a request from the user to display a chat room. Next, in S2102, the control unit 301 displays the selected chat room on the display unit 304. Next, in S2103, the control unit 301 stores the most recent unread message among the displayed messages as a "read message" in the storage unit 302. Next, in S2104, the control unit 301 sends the "read messages" stored in the storage unit 302 to the chat application server 107.

[0082] Next, in S2105, the control unit 301 determines whether or not there is an instruction to transcribe the voice message. If the control unit 301 determines that there is an instruction to transcribe (Yes), the process proceeds to S2106. On the other hand, if the control unit 301 determines that there is no instruction to transcribe (No), the process ends. Next, in S2106, the control unit 301 performs voice analysis on the selected voice message and converts it to text.

[0083] Next, in S2107, the control unit displays a transcript display 1201 (see Figure 12) of the text converted in S2106 on the display unit 304, linked to the voice message in the chat room, in a display mode different from the normal message display. Next, in S2108, the control unit 301 determines whether or not there is a "delete instruction" for the transcript display 1201 displayed in S2107. If the control unit 301 determines that there is a "delete instruction" for the transcript (Yes), the process proceeds to S2110. On the other hand, if the control unit 301 determines that there is no "delete instruction" for the transcript (No), the process proceeds to S2109.

[0084] Next, in S2109, the control unit 301 determines whether or not there is an "edit instruction" for the transcription display 1201. If the control unit 301 determines that there is an "edit instruction" (Yes), the process proceeds to S2111. On the other hand, if the control unit 301 determines that there is no "edit instruction" (No), the process proceeds to S2108.

[0085] In the S2109, "editing instructions" can be given by performing actions on the touch panel, such as long-pressing or sliding the corresponding transcript display. However, this is not the only option; a dedicated instruction button can also be displayed, and the user can give "editing instructions" by tapping this button.

[0086] Next, in S2110, the control unit 301 erases the transcript from the display unit 304. Then, in S2111, the control unit 301 displays the transcript editing popup 2201 on the display unit 304.

[0087] There are various editing actions for the displayed transcript. For example, the editing popup 2201 in Figure 22 includes Copy 2202, Memo 2203, and Message Post 2204. Copy 2202 copies the transcribed text, and Memo 2203 inputs into the memo area in the chat room. Furthermore, Message Post 2204 is an action that posts the transcribed string as a regular message. Popups are displayed for each action type in this way.

[0088] Next, in S2112, the control unit 301 determines whether or not an action has been selected for the edit popup 2201. If the control unit 301 determines that an action has been selected (Yes), the process proceeds to S2113. On the other hand, if the control unit 301 determines that no action has been selected (No), the process proceeds to S2108. Then, in S2113, the control unit 301 executes the selected action.

[0089] As explained above, by performing editing actions on the transcript display, the transcribed text can be used for other tasks. Figure 22 shows copy 2202, memo 2203, and message posting 2204, but it is not limited to these, and other actions can be added or the action types shown in Figure 22 can be replaced with other actions. For example, a "recall" action that retrieves all memos previously stored in the memo area, or a "summarize" action that summarizes the text of an audio message, could be used. To simplify the structure of the text, summarization can be done by, for example, extracting only keywords such as location and date / time, or by extracting a predetermined number of characters from the beginning of the text.

[0090] <Summary of the 4th Embodiment> As explained above, the control unit 301 (editing means) edits the text string when an editing instruction is given by operating the editing instruction unit (editing instruction means) for giving editing instructions for the text string. Furthermore, as shown in Figure 22, the text string editing performed by the control unit 301 (editing means) includes "copying" the text string, "memo" storing the text string in the memory unit, and "posting" the text string as a text message in the chat room. However, the editing methods are not limited to these. In this way, editing the voice message using the text string is provided for the convenience of the user.

[0091] <Other> (1) The above description has focused on an example of chatting between mobile terminal devices, but the operations described above can also be achieved between PCs such as desktop PCs and notebook PCs, or between a PC and a mobile terminal. In this case, the PC may be connected to a router via a wired LAN cable or to a wireless router. The notebook PC may also be portable. However, the PC must have the application program (AP) according to the present invention installed, and must have a built-in or external microphone for voice input and speakers for voice output. (2) The chat room itself may be displayed as a 3D image, and the text may also be displayed in 3D, allowing users to wear special glasses or goggles to recognize the chat room and the transcribed text in 3D. In this case, setting up one's own avatar and the other person's avatar in advance and displaying them in appropriate positions will enhance the chat experience. (3) The user assigns a color to each chat content (for friends, work, a specific meeting, etc.). The control unit 301 then extracts keywords (date, time, location, etc.) from the transcribed voice message, determines the chat content, and displays the text string with the corresponding color. This allows the user to grasp the general content. (4) By performing specific operations such as long-pressing the triangular playback icon of the voice message, it is possible to display the text string within the voice message display area without outputting the audio. In this case, if the string does not fit within the display area, it is possible to slide the string sequentially to make it visible by sliding the string. (5) Without having to make requests to the chat application server 107 each time, if a contract is concluded with the operator of the chat application server 107 that voice messages will always be converted to text, the various processes described above can be implemented with high accuracy on mobile devices.

[0092] Although preferred embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications and changes are possible within the scope of its gist. For example, the present invention can also be realized by supplying a program that implements one or more of the functions of the above embodiments to a system or device via a network or recording medium, and the processor of the computer in that system or device reads and executes the program. Furthermore, the present invention can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of Symbols]

[0093] 102 Mobile devices 104 Mobile devices 105 Communication base stations 106 Internet 107 Chat application server 201 Control Unit 202 Storage section 203 Control Program 209 Chat Processing Unit 301 Control Unit 302 Storage section 303 Input / output section 304 Display section 305 Mike 306 Speakers 404 Message Input Field 901 Chat Management Table

Claims

1. A program that causes a computer to execute a control method for an information processing device, The control method described above is A text conversion step that converts the content of the voice message received by the information processing device into text, A display step that displays the string converted into text by the text conversion step in association with the voice message, The process includes an object display step that displays an object that performs an action to post the string as a text message that is not associated with the voice message, A program characterized in that, based on the selection of the object, the string is added as the text message that is not associated with the voice message.

2. In the aforementioned display step, The program according to claim 1, characterized in that it distinguishes and displays the display format of the string converted into text by the above-mentioned text conversion process from the display format of a message different from the voice message.

3. In the aforementioned display step, The program according to claim 1, characterized in that it displays an indicator indicating that the corresponding voice message has been converted into text for the string converted into text by the above text conversion step.

4. The program according to claim 1 or 2, further comprising an instruction step of giving an instruction to convert the content of the voice message into text when a long press or slide operation is received on the voice message.

5. In the aforementioned display step, The program according to claim 1 or 2, characterized in that, if the text generated by the above-mentioned text generation process does not fit within the display area, the program updates the display so that the text can be viewed by accepting a specific operation.

6. The process further includes an erase instruction step for giving an instruction to erase the string of characters converted into text by the above text conversion step, In the aforementioned display step, further, The program according to claim 1 or 2, characterized in that when an erase command is given by the operation of the erase command step, the displayed string is erased.

7. In the aforementioned text conversion process, The program according to claim 1 or 2, characterized in that the content of the voice message is converted into a string based on a predetermined time.

8. The program according to claim 1 or 2, characterized in that the object is an action popup.

9. In the aforementioned text conversion process, The program according to claim 1 or 2, characterized in that it causes an external server to perform a process of converting the content of the voice message into text, and receives the converted text string.

10. The program according to claim 1 or 2, characterized in that the information processing device is a mobile device including a smartphone or tablet terminal.

11. In the object display step, The program according to claim 1, characterized in that it displays the object that performs the operation of posting the string as the text message.

12. The control method is: A receiving step for receiving instructions on how to use the aforementioned string, A popup display step in which a popup containing the object is displayed in response to the usage operation instruction, The program according to claim 1, further comprising the following:

13. The operation for posting the string is: This operation adds the aforementioned string as a regular text message in the chat room. The program according to claim 1, characterized in that the ordinary text message is displayed without being associated with the voice message.

14. A storage medium storing the program described in claim 1.

15. A control method for an information processing device, A text conversion step that converts the content of the voice message received by the information processing device into text, A display step that displays the string converted into text by the text conversion step in association with the voice message, The process includes an object display step that displays an object that performs an action to post the string as a text message that is not associated with the voice message, A control method for an information processing device, characterized in that, based on the selection of the object, the string is added as the text message that is not associated with the voice message.

16. An information processing device, A text conversion means that converts the content of an audio message received by the information processing device into text, A display means that displays a string of characters converted into text by the text conversion means in association with the voice message, The system includes an object display means that displays an object that performs an action to post the string as a text message that is not associated with the voice message, An information processing device characterized in that, based on the selection of the object, the string is added as the text message that is not associated with the voice message.

Images