Portable terminal device, information processing system, information processing method, and program

Abstract

To enable a portable terminal device owner to obtain only necessary text information.SOLUTION: A portable terminal device includes: a voice input unit 110; a storage unit 120 configured to store registration information; a determination unit 130 configured to determine whether at least a portion of voice data input by the voice input unit 110 is stored in the storage unit 120 as registration information; and a display unit 150 configured to display notification information corresponding to the registration information when the determination unit 130 determines that at least a portion of the voice data is stored in the storage unit 120 as registration information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a mobile terminal device, an information processing system, an information processing method, and a program. [Background technology]

[0002] A technology is being considered that, when a train a passenger is riding on suddenly comes to a halt, allows the portable terminal device carried by the passenger to convert the audio information of the in-car announcements into text information and display it, thereby enabling even hearing-impaired passengers to be aware of the situation around them (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-157776 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned technology, all voice information input to the mobile terminal device is converted into text information and displayed, which can lead to the problem that unnecessary text information may be displayed depending on the owner of the mobile terminal device, resulting in an ineffective display of the text information.

[0005] An object of the present invention is to provide a mobile terminal device, an information processing system, an information processing method, and a program that allow the owner of the mobile terminal device to obtain only the necessary text information. [Means for solving the problem]

[0006] The mobile terminal device of the present invention comprises: an audio input unit; a storage unit that stores registration information; a determination unit that determines whether at least a part of the sound data input by the voice input unit is stored in the storage unit as the registration information; The display unit displays notification information corresponding to the registration information when the determination unit determines that at least a portion of the sound data is stored in the storage unit as the registration information.

[0007] Further, the information processing system of the present invention comprises: The system includes a mobile terminal device, a server, and a storage unit that stores registration information, The mobile terminal device an audio input unit; a first communication unit that transmits the sound data input by the voice input unit to the server; a display unit, The server a determination unit that determines whether at least a part of the sound data transmitted from the first communication unit is stored in the storage unit as the registration information; a second communication unit that, when the determination unit determines that at least a part of the sound data is stored in the storage unit as the registration information, transmits notification information corresponding to the registration information to the mobile terminal device; The display unit displays the notification information transmitted from the second communication unit.

[0008] Further, the information processing method of the present invention comprises: a process of determining whether at least a part of the input sound data is stored in the storage unit as registration information; When at least a part of the sound data is stored in the storage unit as the registration information, the processing of displaying notification information corresponding to the registration information is performed.

[0009] The program of the present invention also includes: A program to be executed by a computer, On the computer, a step of determining whether at least a part of the input sound data is stored in the storage unit as registration information; If at least a part of the sound data is stored in the storage unit as the registration information, the step of displaying notification information corresponding to the registration information is executed. [Effects of the Invention]

[0010] In the present invention, the owner of the portable terminal device can obtain only the necessary text information. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 2 is a diagram illustrating an example of components included in a mobile terminal device according to a first embodiment. [Figure 2] 2 is a diagram showing an example of association between registration information and notification information stored in a storage unit shown in FIG. 1. FIG. [Figure 3] 10 is a flowchart illustrating an example of a registration process of registration information and notification information in the mobile terminal device shown in FIG. [Figure 4] 10 is a flowchart illustrating an example of a process for displaying notification information based on input sound data in the mobile terminal device shown in FIG. [Figure 5] FIG. 10 is a diagram showing an example of the display of screen data including notification information displayed on the display unit in step S15. [Figure 6] 10 is a flowchart illustrating another example of the registration process of registration information and notification information in the mobile terminal device shown in FIG. [Figure 7] 10 is a flowchart illustrating another example of the process of displaying notification information based on input sound data in the mobile terminal device shown in FIG. [Figure 8] FIG. 10 is a diagram showing an example of the display of screen data including notification information displayed on the display unit in step S34. [Figure 9] FIG. 10 illustrates an example of an information processing system according to a second embodiment. [Figure 10] 10 is a diagram illustrating an example of components included in the mobile terminal device illustrated in FIG. 9. FIG. [Figure 11]FIG. 10 is a diagram illustrating an example of components included in the server illustrated in FIG. 9. [Figure 12] 10 is a diagram showing an example of the association between user identification information, registration information, and notification information stored in the storage unit shown in FIG. 9. FIG. [Figure 13] 10 is a sequence diagram illustrating an example of a registration process of registration information and notification information in the information processing system shown in FIG. [Figure 14] 10 is a sequence diagram illustrating an example of a process for displaying notification information based on input sound data in the information processing system shown in FIG. 9. FIG. [Figure 15] 10 is a sequence diagram illustrating another example of the registration process of registration information and notification information in the information processing system shown in FIG. [Figure 16] 10 is a sequence diagram illustrating another example of the process of displaying notification information based on input sound data in the information processing system shown in FIG. 9. FIG. [Figure 17] FIG. 10 is a diagram illustrating an example of components included in a mobile terminal device according to a third embodiment. [Figure 18] 18 is a diagram showing an example of correspondence between notification information and voice messages stored in a storage unit shown in FIG. 17. FIG. [Figure 19] 18 is a flowchart illustrating an example of an information processing method in the mobile terminal device shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. (First embodiment)

[0013] FIG. 1 is a diagram showing an example of components included in a mobile terminal device according to a first embodiment. As shown in FIG. 1, the mobile terminal device 100 according to this embodiment includes a voice input unit 110, a storage unit 120, a determination unit 130, a screen creation unit 140, a display unit 150, and an information input unit 160. Furthermore, the mobile terminal device 100 may preferably include a vibration unit 170 and a light-emitting unit 180. The mobile terminal device 100 may also include a notification information generation unit 141. Note that FIG. 1 shows only the main components related to this embodiment among the components included in the mobile terminal device 100. Examples of the mobile terminal device 100 include a smartphone on which the program of the present invention is installed as software, and a mobile terminal device dedicated to the function of the present invention or for translation.

[0014] The audio input unit 110 inputs sound data from the outside (surroundings) of the mobile terminal device 100 to the mobile terminal device 100. The audio input unit 110 is, for example, a microphone. Note that this "audio" includes electronic sounds such as chimes, mechanical sounds, and natural sounds such as thunder, in addition to human voices. The audio input unit 110 outputs the input sound data to the determination unit 130. Furthermore, when registering registration information, which will be described later, the audio input unit 110 may store the input sound data in the storage unit 120.

[0015] The storage unit 120 stores registration information. The registration information is a keyword for notification and is information set by the user operating the mobile terminal device 100. The registration information is, for example, character data of nouns such as the user's first name, last name, first name, or nickname, the name of the station used, and numbers indicating waiting order (hereinafter referred to as "registered character data"). Note that the storage unit 120 may also register general nouns that represent disasters or events, adjectives such as "dangerous," and verbs such as "run away" as registration information. The storage unit 120 also stores sound data of recorded notification sounds (hereinafter referred to as "registered sound data") as registration information. When registration information corresponding to a notification sound is set, the storage unit 120 stores information indicating the location where the sound data of the notification sound is stored (such as a file name, a stored directory, or a URL) as registration information. The notification sound is a sound used to notify the occurrence of some event in daily life, such as a doorbell, a sound indicating that a microwave oven has finished heating food, a sound indicating that a rice cooker has finished cooking rice, a sound indicating that a bath is ready, or a warning sound emitted by a gas leak detector or a fire alarm. The storage unit 120 stores the registration information and the notification information input by the information input unit 160 in association with each other. The notification information is character data that is displayed on the display unit 150 when sound data (hereinafter referred to as "input sound data") is input by the voice input unit 110 so that the user of the mobile terminal device 100 can understand the meaning of the input sound data. The notification information may be set, for example, by the information input unit 160 selecting and setting notification information that matches the registration information from among pre-prepared notification information based on an operation by the user of the mobile terminal device 100. Alternatively, the notification information may be set by inputting character data by information input unit 160 based on an operation by the user of mobile terminal device 100 and setting the data as notification information. Alternatively, the notification information may be set by generating notification information that matches the set registration information by notification information generation unit 141, which will be described later. Note that when notification information generation unit 141 generates notification information, it is not necessary to store the notification information in storage unit 120 in association with the registration information.

[0016] FIG. 2 is a diagram illustrating an example of the association between registration information and notification information stored in the storage unit 120 illustrated in FIG. 1. As illustrated in FIG. 2, the storage unit 120 illustrated in FIG. 1 stores registration information and notification information in association with each other. In the example illustrated in FIG. 2, the registration information "XX" is the user's name, and indicates that when the user's name is called at a facility such as a hospital, the notification information is displayed as "Your name has been called!". The registration information "Number 2" is an example of registration when a reception number used to organize the order of service at a facility such as a restaurant is received, and indicates that when the number is called, the notification information is displayed as "You will be shown to your seat." If this registration information is included in the input sound data input by the voice input unit 110, the notification information associated with the registration information is read out. By storing the registration information and notification information in association with each other in the storage unit 120 in this manner, when the voice input unit 110 inputs input sound data, notification information corresponding to the input sound data can be obtained. The storage unit 120 may also store an identifier for identifying a set of data consisting of registration information and notification information in association with each other, and may also be able to store an explanatory text for the user to remember what the registered information is in association with each other. In particular, in the example shown in Fig. 2, in order to indicate what sound the registered notification sound is, it is possible to write "(sound of a doorbell)". In this case, the location where the sound data of the notification sound is stored is also registered in association with each other. It is also possible to register input sound data in a language that the user does not understand as registration information, and then register notification information translated into a language that the user can understand for the registration information.

[0017] The determination unit 130 determines whether at least a portion of the input sound data output from the voice input unit 110 is stored in the storage unit 120 as registration information. At this time, the determination unit 130 may convert the input sound data output from the voice input unit 110 into character data (hereinafter referred to as "input character data") and determine whether unit character data included in the converted input character data is stored in the storage unit 120 as registration information. This conversion may be performed using a general voice recognition function that converts human speech into character data (text data). In addition, to convert the input character data into unit character data, the input character data may be subjected to morphological analysis and broken down into individual morphemes (=unit character data), thereby creating unit character data. Instead of determining whether the entire converted input character data is stored in the storage unit 120 as registration information, the determination unit 130 determines whether unit character data, which is a part of the converted input character data, is stored in the storage unit 120 as registration information, thereby making it possible to handle multiple patterns of input character data containing key information. For example, determination unit 130 searches for registered character data stored in storage unit 120 and determines whether the input character data includes the registered character data stored in storage unit 120. Alternatively, when registered sound data is stored in storage unit 120 as registration information, determination unit 130 determines whether the sound data output from voice input unit 110 contains a sound that matches the registered sound data stored in storage unit 120 (a portion where the waveforms of each sound data match). Alternatively, for example, determination unit 130 compares unit character data obtained by breaking down input character data into morphemes with the registered character data stored in storage unit 120 as registration information, and determines whether there is matching unit character data and registered character data. Alternatively, for example, if the input sound data output from the voice input unit 110 is a sound that cannot be converted into text or a sound that cannot be distinguished from a human voice (for example, a sound that may be a notification sound as described above), the judgment unit 130 breaks down the input sound data into unit sound data for each sound waveform, compares it with the waveform of registered sound data registered as registration information, and judges whether the unit sound data, which is at least a part of the input sound data, matches the registered sound data, which is the registration information.In addition, the judgment unit 130 may judge whether or not the unit sound data, which is at least a part of the input sound data, and the registered sound data, which is the registration information, are similar (this is included in the judgment of whether or not they match in the following explanation) even if they do not completely match.

[0018] The determination unit 130 notifies the screen creation unit 140 of the determination result. The method by which the determination unit 130 notifies the screen creation unit 140 of the determination result may be, for example, a method of outputting registration information that matches the input sound data, a method of outputting an identifier indicating a pair of the matching registration information and notification information stored in association with the matching registration information, even if it is not the registration information itself, or a method of outputting the notification information stored in association with the matching registration information.

[0019] When the determination unit 130 notifies the screen creation unit 140 of a determination result that at least a portion of the input sound data input by the voice input unit 110 (for example, the above-described unit character data or unit sound data) is stored in the storage unit 120 as registration information (matches the registration information), the screen creation unit 140 creates screen data for displaying the notification information on the display unit 150 using the notification information stored in the storage unit 120 corresponding to the matching registration information. The screen data includes character data that is the notification information and is information for setting the position, size, color, format, etc., of the character data, as well as other characters, images, etc., to be arranged on the screen. For example, a predetermined format may be prepared in advance, to which the screen data can be completed simply by adding character data. Note that when the determination unit 130 outputs registration information or an identifier, the screen creation unit 140 reads out the notification information stored in the storage unit 120 corresponding to the registration information or the identifier, adds the notification information to the format of the screen data, and creates the screen data. Furthermore, when notification information is output from determination unit 130, screen creation unit 140 adds the output notification information to the format of the screen data to create screen data. Furthermore, when notification information generation unit 141 generates notification information, when determination unit 130 determines that the information matches the registered information, notification information generation unit 141 may generate the notification information, and screen creation unit 140 may use the notification information generated by notification information generation unit 141 to create screen data for displaying the notification information on display unit 150. Screen creation unit 140 outputs the created screen data including the notification information to display unit 150.

[0020] The notification information generation unit 141 generates notification information according to the registration information. The notification information generation unit 141 generates notification information according to the registration information to be stored when the registration information is stored in the storage unit 120. Alternatively, when at least a portion of the input sound data input by the voice input unit 110 matches the registration information stored in the storage unit 120, the notification information generation unit 141 generates notification information according to the matching registration information. Examples of methods by which the notification information generation unit 141 generates notification information according to the registration information include a method of registering a fixed phrase (e.g., "Someone called Mr. / Ms. XX!") to be added to the registration information and adding the fixed phrase to the registration information (e.g., "XX") to generate notification information (e.g., "Someone called Mr. / Ms. XX!"), and a method of generating notification information using a learning model that inputs registration information and outputs notification information. The learning model in this case is a learning model that machine-learns notification information for the registration information. Furthermore, the notification information generating unit 141 may obtain (generate) the notification information by inputting an instruction to generate the notification information using the registration information as a prompt to a generating AI (artificial intelligence).

[0021] The display unit 150 displays screen data including the notification information output from the screen creation unit 140. At this time, the display unit 150 may display the volume of the input sound data that matches the registration information. For example, when creating screen data, the screen creation unit 140 acquires a sound pressure value indicating the sound pressure (dB) of the input sound data that matches the registration information from the audio input unit 110, converts the sound pressure value into an easy-to-understand level unit (e.g., 10 levels), and adds the volume to the screen data. The display unit 150 then displays the screen data including the notification information and the volume. By displaying the volume of the input sound data input by the audio input unit 110, the user of the mobile terminal device 100 can determine whether the location of the audio data is close to the user. The display unit 150 may be a display that is included in a typical mobile terminal device.

[0022] The information input unit 160 inputs registration character data, which is registration information, based on an operation by a user operating the mobile terminal device 100. The information input unit 160 may also input or select notification information based on an operation by a user operating the mobile terminal device 100, and store the notification information and the input registration information in the storage unit 120 in association with each other. The user may use the information input unit 160 to associate the registration information with the notification information and store them in the storage unit 120, thereby enabling settings to be made according to how the user uses the device. The information input unit 160 may also input and record the notification sound described above using the voice input unit 110, store the recording in the storage unit 120, and register the registered sound data, which is the stored notification sound, as registration information. The information input unit 160 is, for example, a touch panel, hardware buttons, or a keyboard provided in the mobile terminal device 100.

[0023] The vibration unit 170 vibrates the mobile terminal device 100 when the display unit 150 displays notification information. The vibration unit 170 may vibrate the mobile terminal device 100 using a vibration function of a typical mobile terminal device. This allows the user to view the display unit 150 even when the user cannot view the display unit 150, such as when the user has the mobile terminal device 100 in a pocket or when the smartphone (i.e., the mobile terminal device 100) is in a closed wallet-style case. This allows the user to know that a notification has been received. For example, when the screen creation unit 140 outputs notification information to the display unit 150, the screen creation unit 140 outputs a command to the vibration unit 170 to vibrate the mobile terminal device 100. The vibration linked to the notification information may be turned on or off.

[0024] The light-emitting unit 180 emits light when the display unit 150 displays notification information. The light-emitting unit 180 may blink a lamp such as an LED provided in the mobile terminal device 100, or may alternately display, for example, a white screen and a black screen on at least a part of the display unit 150. This encourages the user to view the display unit 150 even when the mobile terminal device 100 is far from the user's location, and allows the user to know that some kind of notification has been received. For example, when the screen generating unit 140 outputs notification information to the display unit 150, the screen generating unit 140 outputs a command to the light-emitting unit 180 to cause it to emit light. Furthermore, the light emission linked to the notification information may be turned on / off.

[0025] The following describes an information processing method in the mobile terminal device 100 shown in Fig. 1. Fig. 3 is a flowchart illustrating an example of registration processing of registration information and notification information in the information processing method in the mobile terminal device 100 shown in Fig. 1. Here, the processing will be described when the target sound data is sound data that can be converted into character data.

[0026] First, a user carrying mobile terminal device 100 operates mobile terminal device 100 to input registration information and notification information in association with each other using information input unit 160 (step S1). This operation may be, for example, an operation following an input screen displayed on display unit 150 equipped with a touch panel function. This input screen may be a screen on which text is input by operating character input keys, or a screen on which multiple options are displayed and desired information is selected and input from the displayed options. Note that if the registration information input by information input unit 160 is unchanging information such as the user's own name, it is preferable to register the registration information in advance. However, if the registration information input by information input unit 160 is temporary information such as a reception number, it is preferable to make the registration information available for input using information input unit 160 as needed.

[0027] Next, the storage unit 120 stores the registration information and notification information input by the information input unit 160 in association with each other (step S2). The state of the registration information and notification information stored in the storage unit 120 is as shown in FIG. 2. At this time, a general template (e.g., "Matched with the registered information!") may be prepared as the notification information, and the input registration information may be stored in the storage unit 120 in association with the notification information of the general template. Alternatively, the registration information input by the user operating the mobile terminal device 100 to the notification information input by the information input unit 160 may be stored in association with the storage unit 120. Alternatively, the input registration information may be selected from a plurality of sentences prepared by the user operating the mobile terminal device 100 and stored in association with the storage unit 120. Alternatively, notification information may be generated by the notification information generation unit 141 based on the input registration information and stored in association with the storage unit 120. In addition to the registration process of the registration information and the notification information, the information input unit 160 can also be used to change or delete the registered registration information and the notification information.

[0028] Fig. 4 is a flowchart illustrating an example of a process for displaying notification information based on input sound data, among the information processing methods in the mobile terminal device 100 shown in Fig. 1. Here, the process will be described when the target input sound data is data that can be converted into character data.

[0029] After starting the process of displaying notification information based on input sound data, when the voice input unit 110 inputs input sound data (step S11), the determination unit 130 converts the input sound data input by the voice input unit 110 into input character data (step S12). Subsequently, the determination unit 130 determines whether or not the unit character data included in the converted input character data is stored as registered information in the storage unit 120 (whether or not the unit character data matches the registered information) (step S13). When the determination unit 130 determines that the unit character data included in the converted input character data is stored as registered information in the storage unit 120 (matches), the screen creation unit 140 reads out notification information associated with the matching registered information from the storage unit 120, and creates screen data including the read notification information (step S14). Note that when notification information generation unit 141 generates notification information, the notification information may not be stored in storage unit 120, but notification information generation unit 141 may generate notification information based on the matching registration information, and screen creation unit 140 may create screen data including the notification information generated by notification information generation unit 141. Display unit 150 displays the screen data including the notification information created by screen creation unit 140 (step S15). Furthermore, when displaying the screen data including the notification information on display unit 150, determination unit 130 or screen creation unit 140 may output a command to vibrate to vibration unit 170, or may output a command to emit light to light-emitting unit 180.

[0030] FIG. 5 is a diagram illustrating an example of the display of screen data including notification information displayed by display unit 150 in step S15. As shown in FIG. 5, display unit 150 displays notification information such as "Someone called you 'XXX'!" and a cloud-like illustration around it. The display shown in FIG. 5 illustrates an example in which, when input sound data including "XXX" is input by voice input unit 110 and the registered information "XXX" shown in FIG. 2 is stored in storage unit 120, screen creation unit 140 reads notification information "Someone called you 'XXX'!" associated with registered information "XXX" from storage unit 120, adds the read notification information "Someone called you 'XXX'!" to the format of screen data including the cloud-like illustration to create screen data, and display unit 150 displays the screen data. Note that in this example, display unit 150 may perform the function of light-emitting unit 180 by flashing the cloud-like illustration. After this process starts, external sound is always input to the voice input unit 110, and notification information is displayed when the input matches the registered information stored in the storage unit 120. Furthermore, the format (illustration) of the screen data displayed together with the notification information is not limited to the cloud-like illustration shown in Fig. 5, but may be any other illustration set by the user or an illustration prepared in advance by the system.

[0031] Fig. 6 is a flowchart illustrating another example of the registration process of registration information and notification information in the information processing method in the mobile terminal device 100 shown in Fig. 1. Here, the process when the target input sound data is the notification sound described above will be described. Note that the same descriptions as for steps S1 to S2 and steps S11 to S15 may be omitted.

[0032] First, a user of mobile terminal device 100 operates mobile terminal device 100 to record a notification sound, which is sound data, using voice input unit 110 (step S21). The notification sound input from voice input unit 110 is stored in storage unit 120. A method for recording a notification sound and registering it as registration information (storing it in storage unit 120) is, for example, a method in which, if the notification sound to be registered is a doorbell, the user starts recording, rings the doorbell, and then stops recording. When sound data is recorded using this method, the user operates mobile terminal device 100, which is located inside a room, then operates a doorbell push button or intercom handset installed outside the doorbell to ring the doorbell, and then returns inside the room. Therefore, the recorded sound data contains unnecessary silence or other sounds before and after the doorbell. When recording is finished, determination unit 130 displays, on display unit 150, the sound pressure level that changes over time and the text that converts the sound for the sound data from the start to the end of recording. When the user selects the character display (e.g., "ping pong") of the portion they want to register using the information input unit 160, the determination unit 130 extracts the sound data portion before character conversion that corresponds to the portion of the character display and registers it as registered sound data in the registration information. Note that, since it may be difficult for a hearing-impaired user to record sounds, the process of step S21 may be performed by a family member or friend of the user on their behalf, who may acquire a recording file of the notification sound on the user's mobile terminal device 100 using a messaging app or the like, or may download the recording file of the notification sound to the user's mobile terminal device 100 from the website of the manufacturer of the device that outputs the notification sound, and the acquired recording file may be stored in the storage unit 120.

[0033] Next, the user of the mobile terminal device 100 operates the mobile terminal device 100 to input or select notification information using the information input unit 160 (step S22). This operation may be, for example, an operation according to an input screen displayed on the display unit 150 equipped with a touch panel function. This input screen may be a screen on which text is input by operating character input keys, or a screen on which multiple options are displayed and desired information is selected and input from the displayed options. The notification information may be set as a general fixed phrase (e.g., "Matched with registered information!"), or may be input by the information input unit 160 based on the operation of the user operating the mobile terminal device 100, or may be selected from multiple prepared sentences based on the operation of the user operating the mobile terminal device 100, or the notification information may be generated by the notification information generation unit 141 based on the input registration information.

[0034] When a recorded notification sound is treated as registration information, it is not necessary to input the registration information as text data. Therefore, a process may be added in which information describing the registered notification sound is input by the information input unit 160 in association with the registration information. The determination unit 130 may have a function for identifying what the notification sound input by the audio input unit 110 is notifying. In this case, the result of the identification by the determination unit 130 may be stored in the storage unit 120 as information describing the registered notification sound in association with the registration information.

[0035] Next, the storage unit 120 associates the recorded notification sound or the notification information input or selected by the information input unit 160 or generated by the notification information generation unit 141 with the input registration information and stores them (step S23). In addition to the registration process of the registration information and the notification information, the information input unit 160 can also be used to change or delete the registered registration information and the notification information.

[0036] Fig. 7 is a flowchart illustrating another example of the process of displaying notification information based on input sound data, among the information processing methods in the mobile terminal device 100 shown in Fig. 1. Here, the process will be described when the target sound data is the notification sound described above.

[0037] After starting the process of displaying notification information based on input sound data, when the voice input unit 110 inputs input sound data (step S31), if the determination unit 130 determines that the input sound data is a sound that cannot be converted into text or a sound that cannot be distinguished from a human voice (for example, a sound that may be a notification sound), it determines whether the input sound data includes registered sound data (recorded notification sound) stored in the storage unit 120 as registration information (step S32). For example, when the registered sound data is stored in the storage unit 120 as registration information, the determination unit 130 determines whether the sound waveform of the sound data output from the voice input unit 110 includes a sound waveform that matches the registered sound data.

[0038] When the determination unit 130 determines that the input sound data contains registration information that is registration sound data (recorded notification sound) stored in the storage unit 120 (that is, that the input sound data contains matching registration sound data), the screen creation unit 140 reads out notification information associated with the registration information from the storage unit 120 and creates screen data including the read notification information (step S33). Note that when the notification information generation unit 141 generates notification information, the notification information may not be stored in the storage unit 120, but the notification information may be generated by the notification information generation unit 141 based on the matching registration information, and the screen creation unit 140 may create screen data including the notification information generated by the notification information generation unit 141. The display unit 150 displays the screen data including the notification information created by the screen creation unit 140 (step S34). Furthermore, when displaying screen data including notification information on the display unit 150, the determination unit 130 or the screen creation unit 140 may output a command to vibrate to the vibration unit 170, or may output a command to emit light to the light emission unit 180.

[0039] 8 is a diagram showing an example of the display of screen data including notification information displayed by display unit 150 in step S34. As shown in FIG. 8, display unit 150 displays notification information "Someone has visited you" and a scroll-like illustration around it. The display shown in FIG. 8 is an example in which, when the sound of a doorbell (notification sound) is stored in storage unit 120 as registered information (registered sound data) and notification information "Someone has visited you" is stored in storage unit 120 in association with the registered information, when voice input unit 110 inputs input sound data including the sound of the doorbell, screen creation unit 140 reads out the notification information "Someone has visited you" associated with the registered information from storage unit 120, adds the read notification information "Someone has visited you" to the format of screen data including the scroll-like illustration, creates screen data, and display unit 150 displays the screen data. In this example, the display unit 150 may flash a scroll-like illustration, thereby performing the function of the light-emitting unit 180. After the start of this process, external sounds are always input to the voice input unit 110, and notification information is displayed when the external sounds match the registered information stored in the storage unit 120.

[0040] In this manner, in this embodiment, notification information corresponding to the input sound data is displayed. Therefore, the mobile terminal device 100 displays notification information only for sound data that has been registered in advance. This allows a hearing-impaired user who owns the mobile terminal device 100 or a user who has difficulty understanding the language spoken around them to obtain only the necessary text information, and the user can take action based on the text information. (Second embodiment)

[0041] Fig. 9 is a diagram showing an example of an information processing system according to the second embodiment. As shown in Fig. 9, the information processing system according to the second embodiment includes a mobile terminal device 101, a server 200, and a storage unit 301. The mobile terminal device 101 and the server 200 are connected via a communication network 400. The server 200 and the storage unit 301 may be connected directly or via the communication network 400. The server 200 may also include the storage unit 301. Note that descriptions similar to those in the first embodiment may be omitted.

[0042] The mobile terminal device 101 is a communication device carried and operated by a user who uses the present system. FIG. 10 is a diagram illustrating an example of components included in the mobile terminal device 101 shown in FIG. 9. As shown in FIG. 10, the mobile terminal device 101 in this embodiment includes a voice input unit 110, a display unit 150, an information input unit 160, a control unit 191, and a communication unit 192. Furthermore, the mobile terminal device 101 preferably includes a vibration unit 170 and a light-emitting unit 180. The voice input unit 110, the display unit 150, the information input unit 160, the vibration unit 170, and the light-emitting unit 180 are the same as those in the first embodiment. Note that FIG. 10 illustrates only the main components related to this embodiment among the components included in the mobile terminal device 101 shown in FIG. 9. An example of the mobile terminal device 101 is a smartphone on which the program of the present invention is installed as software.

[0043] The control unit 191 controls the voice input unit 110, the display unit 150, the information input unit 160, the vibration unit 170, the light emitting unit 180, and the communication unit 192 that are included in the mobile terminal device 101. The control unit 191 outputs input sound data output from the voice input unit 110 to the communication unit 192. The control unit 191 outputs screen data including the notification information output from the communication unit 192 to the display unit 150. The display unit 150 displays the screen data including the notification information output from the control unit 191.

[0044] Communication unit 192 is a first communication unit that transmits input sound data output from control unit 191 to server 200. Communication unit 192 outputs screen data including notification information transmitted from server 200 to control unit 191. When transmitting information such as input sound data to server 200, communication unit 192 transmits user identification information uniquely assigned to mobile terminal device 101 or the user operating mobile terminal device 101 to server 200 together with the information.

[0045] Fig. 11 is a diagram showing an example of components included in server 200 shown in Fig. 9. As shown in Fig. 11, server 200 shown in Fig. 9 includes control unit 210 and communication unit 240. Control unit 210 includes determination unit 220 and screen creation unit 230. Control unit 210 may also include notification information generation unit 250. Note that Fig. 11 shows only the main components related to this embodiment among the components included in server 200 shown in Fig. 9.

[0046] The determination unit 220 determines whether at least a portion of the input sound data transmitted from the communication unit 192 is stored in the storage unit 301 as registered information (whether at least a portion of the input sound data matches the registered information). When making this determination, the determination unit 220 determines whether at least a portion of the input sound data transmitted from the communication unit 192 and user identification information are stored in the storage unit 301 in association with each other. At this time, the determination unit 220 may convert the input sound data transmitted from the communication unit 192 into input character data and determine whether unit character data included in the converted input character data is stored in the storage unit 301 as registered information. This conversion may be performed using a general voice recognition function that converts human speech into character data (text data). Furthermore, to convert the input character data into unit character data, the input character data may be subjected to morphological analysis and broken down into individual morphemes (=unit character data), thereby creating unit character data. Instead of determining whether the entire converted input character data is stored in storage unit 301 as registered information, determination unit 220 determines whether unit character data, which is a part of the converted input character data, is stored in storage unit 301 as registered information, thereby making it possible to handle a plurality of patterns of input character data containing key information. For example, determination unit 220 searches each registered character data stored in storage unit 301 to determine whether registered character data is included in the input character data and determines whether there is a match. Alternatively, when registered sound data is stored in storage unit 301 as registered information, determination unit 220 determines whether the sound data transmitted from communication unit 192 contains a sound that matches the registered sound data (a portion where the waveforms of each sound data match). Alternatively, for example, determination unit 220 compares unit character data obtained by breaking down input character data into morphemes with registered character data stored as registered information, and determines whether there is matching unit character data and registered character data.Alternatively, for example, if the input sound data transmitted from communication unit 192 is a sound that cannot be converted into text or that cannot be distinguished from a human voice (e.g., a sound that may be a notification sound as described above), determination unit 220 decomposes the input sound data into unit sound data for each waveform of the sound, compares the waveform of the registered sound data registered as registration information, and determines whether the unit sound data, which is at least a part of the input sound data, matches the registered sound data, which is the registered information. Note that determination unit 220 may determine whether the unit sound data, which is at least a part of the input sound data, and the registered sound data, which is the registered information, are similar (this determination is included in the determination of whether they match in the following description) even if they do not completely match. When making this determination, determination unit 220 determines whether the user identification information transmitted together with the input sound data transmitted from communication unit 192 matches the user identification information stored in storage unit 301 in association with the registered character data or registered sound data determined to match. Determination unit 220 notifies screen creation unit 230 of the determination result. The method by which the determination unit 220 notifies the screen creation unit 230 of the determination result may be, for example, a method of outputting registration information that matches the input sound data, a method of outputting an identifier indicating a pair of the matching registration information and notification information stored in association with the matching registration information, or a method of outputting the notification information stored in association with the matching registration information. At this time, the determination unit 220 also outputs the user identification information transmitted from the communication unit 192 to the screen creation unit 230.

[0047] When the determination unit 220 notifies the screen creation unit 230 of a determination result that at least a part of the input sound data (for example, the above-mentioned unit character data or unit sound data) is stored in the storage unit 301 as registered information (matches the registered information), the screen creation unit 230 creates screen data for displaying the notification information on the mobile terminal device 101, using the notification information stored in the storage unit 301 in correspondence with the matching registered information. The screen creation unit 230 may generate the notification information according to the registered information. The method by which the screen creation unit 230 generates the notification information according to the registered information may be the same as the method in the first embodiment. The screen creation unit 230 outputs the created screen data including the notification information to the communication unit 240.

[0048] The communication unit 240 is a second communication unit that transmits the notification information output from the screen creation unit 230 to the mobile terminal device 101. At this time, the communication unit 240 transmits the notification information to the user identification information transmitted together with the sound data as the transmission destination.

[0049] The notification information generation unit 250 generates notification information according to the registration information. As for the timing at which the notification information generation unit 250 generates notification information, when the registration information is stored in the storage unit 301, the notification information generation unit 250 generates notification information according to the stored registration information. Alternatively, when the determination unit 220 determines that there is a match with the registration information, the notification information generation unit 250 generates notification information according to the matching registration information. Other operations of the notification information generation unit 250 may be similar to those performed by the notification information generation unit 141 in the first embodiment.

[0050] The storage unit 301 stores registration information. The registration information is a keyword for making a notification, and is information that is set by the user operating the information input unit 160 of the mobile terminal device 101. The storage unit 301 also stores sound data of a notification sound recorded using the mobile terminal device 101 (hereinafter referred to as "registration sound data") as registration information. This notification sound is the same as that in the first embodiment. The storage unit 301 stores the registration information in association with user identification information. As described above, the user identification information is information uniquely assigned to the mobile terminal device 101 or the user operating the mobile terminal device 101. The storage unit 301 may also store the registration information in association with notification information. The notification information is the same as that in the first embodiment. The storage unit 301 may be provided within the server 200.

[0051] 12 is a diagram showing an example of the correspondence between user identification information, registration information, and notification information stored in storage unit 301 shown in FIG. 9. As shown in FIG. 12, storage unit 301 shown in FIG. 9 stores a user ID, which is user identification information, registration information, and notification information, in association with each other. The example shown in FIG. 12 shows that a user with user ID "A001" has registered a pair of registration information "XX" (user's name) and notification information "Someone called you!", a pair of registration information "No. 2" (an example registered when receiving a reception number at a restaurant, etc.) and notification information "You will be shown to your seat," and a pair of registration information "(sound of doorbell)" notification sound and notification information "Someone has come to visit." When input sound data transmitted from mobile terminal device 101 in association with user identification information includes registration information stored in association with the user identification information (user ID), the notification information associated with the registration information is read out. In this way, by storing in storage unit 301 the registration information and notification information corresponding to the user identification information in association with each other, when input sound data is transmitted from mobile terminal device 101, notification information corresponding to the input sound data can be obtained for mobile terminal device 101. Note that, as in the first embodiment, an identifier or an explanation for identifying a set of data, namely, registration information and notification information, may also be stored in association with each other.

[0052] The following describes an information processing method in the information processing system shown in Fig. 9. Fig. 13 is a sequence diagram illustrating an example of the registration process of registration information and notification information in the information processing method in the information processing system shown in Fig. 9. Here, the process will be described when the target sound data is sound data that can be converted into character data.

[0053] First, a user carrying the mobile terminal device 101 operates the mobile terminal device 101 to input registration information and notification information in association with each other using the information input unit 160 (step S41). This operation may be, for example, an operation following an input screen displayed on the display unit 150 equipped with a touch panel function. This input screen may be a screen on which text is input by operating character input keys, or a screen on which multiple options are displayed and desired information is selected and input from the displayed options. The input or selection methods for the registration information and notification information are the same as those in the first embodiment. If the notification information is predetermined or if the notification information generation unit 250 generates the notification information, the input or selection of the notification information and the subsequent transmission of the notification information are omitted. If the registration information input by the information input unit 160 is unchanging information such as the user's own name, it is preferable to register the registration information in advance. However, if the registration information input by the information input unit 160 is temporary information such as a reception number, it is preferable to make the registration information available for input using the information input unit 160 as needed. Next, the communication unit 192 associates the user identification information uniquely assigned to the mobile terminal device 101 or the user operating the mobile terminal device 101 with the registration information and notification information input by the information input unit 160 and transmits them to the server 200 (step S42).

[0054] When communication unit 240 receives the user identification information, registration information, and notification information transmitted from communication unit 192, control unit 210 associates these pieces of information with each other and stores them in storage unit 301 (step S43). At this time, when notification information generation unit 250 generates notification information, the notification information may be generated based on the transmitted registration information. The user identification information, registration information, and notification information stored in storage unit 301 are as shown in FIG. 12.

[0055] Fig. 14 is a flowchart illustrating an example of a process for displaying notification information based on input sound data, among the information processing methods in the information processing system shown in Fig. 9. Here, the process will be described in the case where the target input sound data is data that can be converted into character data.

[0056] After starting the process of displaying notification information based on input sound data, the voice input unit 110 inputs input sound data (step S51), and the communication unit 192 associates the user identification information with the input sound data input by the voice input unit 110 and transmits them to the server 200 (step S52). When the communication unit 240 receives the user identification information and input sound data transmitted from the communication unit 192, the determination unit 220 converts the input sound data received by the communication unit 240 into input character data (step S53). Next, the determination unit 220 determines whether or not unit character data included in the converted input character data is associated with the user identification information and stored in the storage unit 301 as registered information (whether or not the unit character data matches the registered information) (step S54). When the determination unit 220 determines that the unit character data included in the converted input character data is stored in the storage unit 301 as registration information associated with the user identification information (matches), the screen creation unit 230 reads out notification information associated with the registration information matching the user identification information from the storage unit 301, and creates screen data including the read notification information (step S55). Note that when the notification information generation unit 250 generates notification information, the notification information generation unit 250 may generate notification information based on the matching registration information without storing the notification information in the storage unit 301, and the screen creation unit 230 may create screen data including the notification information generated by the notification information generation unit 250. The communication unit 240 transmits the screen data including the notification information created by the screen creation unit 230 to the mobile terminal device 101 (step S56).

[0057] When communication unit 192 receives the screen data including the notification information transmitted from communication unit 240, control unit 191 causes display unit 150 to display the screen data including the notification information (step S57). The display mode of the notification information on display unit 150 may be the same as that of the first embodiment (FIG. 5). Furthermore, when displaying the screen data including the notification information on display unit 150, control unit 191 may output a command to vibrate to vibration unit 170, or may output a command to emit light to light-emitting unit 180.

[0058] Fig. 15 is a sequence diagram illustrating another example of the registration process of registration information and notification information in the information processing method in the information processing system shown in Fig. 9. Here, the process when the target sound data is the notification sound described above will be described. Note that the same descriptions as for steps S41 to S43 and steps S51 to S56 may be omitted.

[0059] First, a user who owns the mobile terminal device 101 operates the mobile terminal device 101 to record a notification sound, which is sound data, using the voice input unit 110 (step S61). The notification sound input from the voice input unit 110 is temporarily stored in the memory of the mobile terminal device 101. As in the first embodiment, for hearing-impaired users, a recording file of the notification sound may be obtained from another person, a website, or the like. Next, the user who owns the mobile terminal device 101 operates the mobile terminal device 101 to input or select notification information using the information input unit 160 (step S62). This operation may be performed, for example, according to an input screen displayed on the display unit 150 equipped with a touch panel function. This input screen may be a screen for inputting text by operating character input keys, or a screen that displays multiple options and allows the user to select and input desired information from the displayed options. As in the first embodiment, the notification information may be prepared as a general template, input to information input unit 160, or selected from a plurality of prepared sentences. The notification information input or selected in this manner is temporarily stored in the memory of mobile terminal device 101. Note that when the notification information is predetermined or when notification information generation unit 250 generates the notification information, the input or selection of the notification information and the subsequent transmission of the notification information are omitted.

[0060] When the recorded notification sound is treated as registration information, it is not necessary to input the registration information as text data. Therefore, a process for inputting information explaining the input notification sound using information input unit 160 may be added.

[0061] Next, control unit 191 associates the registration information input by information input unit 160, which is the notification sound (registration sound data) temporarily stored in memory, with the input or selected notification information, and transmits them from communication unit 192 to server 200 (step S63). At this time, communication unit 192 also transmits user identification information uniquely assigned to mobile terminal device 101 or the user operating mobile terminal device 101 to server 200.

[0062] When the communication unit 240 receives the user identification information, the registration information which is the notification sound (registration sound data), and the input or selected notification information transmitted from the communication unit 192, the control unit 210 associates these pieces of information with each other and stores them in the memory unit 301 (step S64).

[0063] Note that determination unit 220 may have a function for identifying what the notification sound transmitted from communication unit 192 is intended to notify. In this case, the result of determination unit 220's identification may be stored in storage unit 301 in association with the registration information as information explaining the registered notification sound. At this time, when notification information generation unit 250 generates notification information, notification information generation unit 250 may generate the notification information based on the transmitted registration information. In addition to the registration process of registration information and notification information, information input unit 160 may also be used to change or delete the registered registration information and notification information.

[0064] Fig. 16 is a flowchart illustrating another example of the process of displaying notification information based on input sound data, among the information processing methods in the information processing system shown in Fig. 9. Here, the process will be described in the case where the target input sound data is the above-mentioned notification sound.

[0065] After starting the process of displaying notification information based on input sound data, when the voice input unit 110 inputs input sound data (step S71), the control unit 191 transmits the input sound data and user identification information from the communication unit 192 to the server 200 (step S72). The communication unit 240 receives the user identification information and input sound data transmitted from the communication unit 192, and if it determines that the received input sound data is a sound that cannot be converted into text or a sound that cannot be distinguished from a human voice (for example, a sound that may be a notification sound), the determination unit 220 determines whether registration information corresponding to the input sound data (notification sound) is stored in the storage unit 301 in association with the user identification information (step S73). For example, when the registration sound data is stored in the storage unit 310 as registration information, the determination unit 220 determines whether the sound waveform of the input sound data includes a sound waveform that matches the registration sound data.

[0066] When the determination unit 220 determines that the registration information, which is the corresponding registration sound data (recorded notification sound), is stored in the storage unit 301 in association with the user identification information (i.e., the input sound data contains matching registration sound data), the screen creation unit 230 reads out notification information associated with the user identification information and the registration information from the storage unit 301 and creates screen data including the read notification information (step S74). Note that when the notification information generation unit 250 generates notification information, the notification information may not be stored in the storage unit 301, but the notification information may be generated by the notification information generation unit 250 based on the matching registration information, and the screen creation unit 230 may create screen data including the notification information generated by the notification information generation unit 250. The communication unit 240 transmits the screen data including the notification information created by the screen creation unit 230 to the mobile terminal device 101 (step S75).

[0067] When the communication unit 192 receives the screen data including the notification information transmitted from the communication unit 240, the control unit 191 causes the display unit 150 to display the notification information (step S76). The display mode of the notification information on the display unit 150 may be the same as that of the first embodiment (FIG. 8). Furthermore, when displaying screen data including notification information on the display unit 150, the control unit 191 may output a command to the vibration unit 170 to vibrate, or may output a command to the light emission unit 180 to emit light.

[0068] In this embodiment, the portable terminal device 101 owned by the user transmits input sound data to the server 200, and the server 200 presents notification information corresponding to the received sound data to the portable terminal device 101, which then displays the notification information. Therefore, the server 200 presents notification information only for sound data pre-registered by the user who owns the portable terminal device 101. This allows a hearing-impaired user who owns the portable terminal device 101 or a user who has difficulty understanding the language spoken around them to obtain only the necessary text information. Furthermore, by storing the registered information in an external storage device of the portable terminal device 101 and providing the server 200 with a notification information determination function, the user can receive notification of the notification information using an already installed web browser without adding any new functions to the portable terminal device 101. The registration information and notification information may be pre-registered in the storage unit 301 by someone other than the user, for example, by someone operating the system. In this case, the storage unit 301 may store, as registered information, words used to notify the occurrence of disasters or troubles that may threaten human life, such as "earthquake," "fire," "heavy rain," "typhoon," "flood," "river overflow," "tsunami," "landslide," "gas leak," "danger," "accident," "incident," and "malfunction." The notification information stored in association with the registered information is preferably information that allows the user to quickly understand the occurrence of each event. Furthermore, the display unit 150 may display predetermined notification information depending on the volume of the sound data input by the voice input unit 110. By displaying such information, it is possible to notify the user that some event has occurred even when the specific event that has occurred cannot be identified. (Third embodiment)

[0069] FIG. 17 is a diagram illustrating an example of components included in a mobile terminal device according to a third embodiment. As shown in FIG. 17, the mobile terminal device 102 according to this embodiment includes a voice input unit 110, a storage unit 122, a determination unit 130, a screen creation unit 142, a display unit 150, an information input unit 162, and a voice output unit 193. Furthermore, the mobile terminal device 102 preferably includes a vibration unit 170 and a light-emitting unit 180. The voice input unit 110, the determination unit 130, the display unit 150, the vibration unit 170, and the light-emitting unit 180 are the same as those in the first embodiment. Of the components included in the mobile terminal device 102, FIG. 17 illustrates only the main components related to this embodiment. Descriptions similar to those in the first embodiment may be omitted.

[0070] The storage unit 122 stores notification information and voice messages in association with each other, in addition to the information stored in the storage unit 120 in the first embodiment. FIG. 18 is a diagram showing an example of the association between notification information and voice messages stored in the storage unit 122 shown in FIG. 17. As shown in FIG. 18, the storage unit 122 shown in FIG. 17 stores notification information and voice messages in association with each other. The notification information is the same as the notification information in the first embodiment. For convenience of explanation, the voice message shown in FIG. 18 is shown as text information, but it may also be information indicating the storage location of the file of the voice data output by the voice output unit 193 (such as a file name, a stored directory, or a URL). The voice message may be a message recorded using the voice input unit 110 and stored in the storage unit 122, a message recorded in advance by a service provider and stored in the storage unit 122, or a voice message obtained by converting text data input using the information input unit 162 into voice. Furthermore, the correspondence between notification information and voice messages may be established by accepting a predetermined operation from the user on the mobile terminal device 102, allowing the information input unit 162 to select a voice message to be associated with the notification information, or the notification information and voice messages may be associated in advance. Furthermore, the voice message may be stored in association with a set of registration information and notification information, rather than being stored in association with the notification information.

[0071] In addition to the processing of the information input unit 160 in the first embodiment, the information input unit 162 accepts operations for selecting a voice message or issuing an instruction to output a voice message based on operations by the user operating the mobile terminal device 102. These operations are predetermined operations, such as, for example, when the display unit 150 has a touch panel function, touching a part of the screen displayed on the display unit 150 (for example, the characters of the voice message or a predetermined button display) or pressing a hardware button provided on the mobile terminal device 102. Other functions and processing provided by the information input unit 162 are the same as those of the information input unit 160 in the first embodiment.

[0072] In the first embodiment, after display unit 150 displays notification information, when audio output unit 193 receives an operation for instructing to output a voice message, audio output unit 193 outputs a voice message corresponding to the notification information. At this time, audio output unit 193 may use the notification information included in the screen data displayed by display unit 150 as a search key, read out from storage unit 122 a voice message associated with the notification information, and output the voice message. Alternatively, audio output unit 193 may cause screen creation unit 142 to read out from storage unit 122 a voice message associated with the notification information included in the screen data created by screen creation unit 142 and displayed by display unit 150, and output the voice message read by screen creation unit 142. Audio output unit 193 may be a speaker or the like. 18 is stored in the storage unit 122, and the display unit 150 is displaying notification information such as "Someone called 'X-san'!", when the information input unit 162 receives an instruction to output a voice message, the voice output unit 193 reads out the voice message "Did they call you?" associated with the notification information "Someone called 'X-san'!" from the storage unit 122 and outputs it. Also, when the display unit 150 is displaying notification information such as "We're making an emergency stop!", when the information input unit 162 receives an instruction to output a voice message, the voice output unit 193 reads out the voice message "Is something wrong?" associated with the notification information "We're making an emergency stop!" from the storage unit 122 and outputs it. Note that when the voice output unit 193 outputs the voice message, the screen creation unit 142 may create screen data including text information of the voice message and display it on the display unit 150.

[0073] It is also possible that a plurality of voice messages are stored in the storage unit 122 in advance, and when the display unit 150 displays the notification information, the screen creation unit 142 creates display screen data that prompts the user to select a voice message that the user wants to output to the voice output unit 193 from the voice messages, and the screen data is displayed by the display unit 150. In this case, the voice output unit 193 reads out the voice message that the user selected using the information input unit 162 from the plurality of voice messages displayed on the display unit 150 from the storage unit 122 and outputs it.

[0074] The following describes an information processing method in the mobile terminal device 102 shown in Fig. 17. Fig. 19 is a flowchart illustrating an example of an information processing method in the mobile terminal device 102 shown in Fig. 17.

[0075] The display unit 150 displays the notification information (step S81). This processing corresponds to the processing in step S15 or step S34 in the first embodiment. Thereafter, when the information input unit 162 accepts an operation to instruct to output a voice message (step S82), the voice output unit 193 reads out a voice message associated with the notification information displayed on the display unit 150 from the storage unit 122, using the notification information as a search key (step S83). Subsequently, the voice output unit 193 outputs the voice message read out from the storage unit 122 (step S84). At this time, the screen creation unit 142 may create screen data including text information of the voice message and display the screen data on the display unit 150.

[0076] Furthermore, after audio output unit 193 outputs a voice message, when audio input unit 110 inputs sound data, determination unit 130 converts the input sound data into input character data, and display unit 150 displays the input character data. When information input unit 162 inputs character data, determination unit 130 converts it into audio data, and audio output unit 193 outputs the audio data.

[0077] As described above, in this embodiment, after the mobile terminal device 102 displays the notification information, if an operation to output a voice message is accepted, a voice message is output. The voice message that is output is a voice message corresponding to the notification information or a voice message that can be selected by the user. Therefore, after the user is notified of necessary text information by inputting sound data into the voice input unit 110, the user can further obtain detailed information such as the situation from others around them and can smoothly communicate with people who have spoken to them or who have come to visit them.

[0078] Although the above description has been given with each component assigned to a respective function (process), this assignment is not limited to the above. Furthermore, the configuration of the components is also merely an example and is not limited to this. Multiple components may be combined into a single component, a single component may be divided into multiple components, or the function (process) of one component may be performed by another component. Furthermore, each embodiment may be combined.

[0079] The processing performed by each of the above-described components may be performed by a logic circuit that is individually manufactured depending on the purpose. Alternatively, a computer program (hereinafter referred to as a program) that describes the processing procedures may be recorded on a recording medium that is readable by each of the mobile terminal devices 100-102 and the server 200, and the program recorded on the recording medium may be read and executed by each of the mobile terminal devices 100-102 and the server 200. Examples of recording media that are readable by each of the mobile terminal devices 100-102 and the server 200 include removable recording media such as floppy disks, magneto-optical disks, DVDs (Digital Versatile Discs), CDs (Compact Discs), Blu-ray (registered trademark) Discs, USB (Universal Serial Bus) memories, and SD cards, as well as memories such as ROMs (Read Only Memory) and RAMs, and HDDs (Hard Disc Drives) that are built into each of the mobile terminal devices 100-102 and the server 200. The program recorded on this recording medium is read by the CPU provided in each of the mobile terminal devices 100-102 and the server 200, and the same processing as described above is performed under the control of the CPU. Here, the CPU operates as a computer that executes the program read from the recording medium on which the program is recorded. [Explanation of symbols]

[0080] 100~102 Portable terminal device 110 Audio input section 120,122,301 Storage section 130,220 Judgment section 140,142,230 Screen Creation Department 141,250 Notification information generation unit 150 Display section 160,162 Information input section 170 Vibration unit 180 Light-emitting part 191,210 Control unit 192,240 Communications Department 193 Audio output section 200 servers 400 Communication Network

Claims

1. an audio input unit; a storage unit that stores registration information; a determination unit that determines whether at least a part of the sound data input by the voice input unit is stored in the storage unit as the registration information; a display unit that displays notification information corresponding to the registration information when the determination unit determines that at least a portion of the sound data is stored in the memory unit as the registration information.

2. 2. The mobile terminal device according to claim 1, the determination unit converts the sound data input by the voice input unit into input character data, and determines whether unit character data included in the converted input character data is stored in the storage unit as the registration information; The display unit displays the notification information corresponding to the registration information when the determination unit determines that the unit character data is stored in the storage unit as the registration information.

3. 3. The mobile terminal device according to claim 1, an information input unit that inputs the registration information and selects the notification information to be associated with the input registration information; the storage unit stores the notification information in association with the registration information; The display unit displays the notification information associated with the registration information in the storage unit.

4. 3. The mobile terminal device according to claim 1, an information input unit for inputting the registration information and the notification information; the storage unit stores the registration information and the notification information input by the information input unit in association with each other; The display unit displays the notification information associated with the registration information in the storage unit.

5. 3. The mobile terminal device according to claim 1, an information input unit for inputting the registration information; a notification information generating unit that generates notification information according to the registration information, The display unit displays the notification information generated by the notification information generation unit.

6. 3. The mobile terminal device according to claim 1, The mobile terminal device has a vibration unit that vibrates the mobile terminal device when the display unit displays the notification information, or a light emitting unit that emits light when the display unit displays the notification information.

7. 3. The mobile terminal device according to claim 1, The display unit displays the volume level of the sound data input by the voice input unit.

8. 3. The mobile terminal device according to claim 1, A mobile terminal device having an audio output unit that outputs a voice message corresponding to the notification information when the display unit displays the notification information and the mobile terminal device receives an instruction to output a voice message.

9. The system includes a mobile terminal device, a server, and a storage unit that stores registration information, The mobile terminal device an audio input unit; a first communication unit that transmits the sound data input by the voice input unit to the server; a display unit, The server a determination unit that determines whether at least a part of the sound data transmitted from the first communication unit is stored in the storage unit as the registration information; a second communication unit that, when the determination unit determines that at least a part of the sound data is stored in the storage unit as the registration information, transmits notification information corresponding to the registration information to the mobile terminal device; The display unit displays notification information transmitted from the second communication unit.

10. a process of determining whether at least a part of the input sound data is stored in the storage unit as registration information; When at least a part of the sound data is stored in the storage unit as the registration information, the information processing method displays notification information corresponding to the registration information.

11. On the computer, a step of determining whether at least a part of the input sound data is stored in the storage unit as registration information; and when at least a portion of the sound data is stored in the storage unit as the registration information, displaying notification information corresponding to the registration information.

Images