Speech processing method, speech processing apparatus, and speech processing program

Abstract

To provide an improved speech processing apparatus, speech processing method, and speech processing system.SOLUTION: A speech processing apparatus comprises a control unit. The control unit acquires a result of speech recognition processing of recognizing a speech from speech data. When the control unit detects a speech satisfying a set condition set in advance to a speech on the basis of the result of the speech recognition processing, it notifies a user of the detection of the speech satisfying the set condition according to a notification condition set in the set condition and corresponding to the detected speech.SELECTED DRAWING: Figure 2

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to Patent Application No. 2022-008227, filed in Japan on January 21, 2022, the entire disclosure of which is incorporated herein by reference. [Technical field]

[0002] The present disclosure relates to an audio processing device, an audio processing method, and an audio processing system. [Background technology]

[0003] Conventionally, there is known a technology that enables a user to hear surrounding sounds while wearing an audio output device such as headphones or earphones. In such a technology, a portable music playback device is known that is equipped with a notification means that notifies the user through headphones when an external sound matches a predetermined phrase (Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2001-256771 A Summary of the Invention

[0005] A voice processing device according to an embodiment of the present disclosure includes: The control unit obtains the results of a voice recognition process that recognizes voice from voice data, and when a voice that satisfies a preset setting condition for the voice is detected based on the result of the voice recognition process, notifies the user that a voice that satisfies the set condition has been detected, in accordance with a notification condition corresponding to the detected voice that is set in the set condition.

[0006] A speech processing method according to an embodiment of the present disclosure includes: Obtaining a result of a speech recognition process for recognizing speech from the speech data; When a voice that satisfies a preset setting condition regarding the voice is detected based on the result of the voice recognition processing, the method includes notifying a user that a voice that satisfies the set condition has been detected, in accordance with a notification condition corresponding to the detected voice that is set in the set condition.

[0007] A voice processing system according to an embodiment of the present disclosure includes: A sound collector that collects surrounding sounds; The voice processing device includes: a voice recognition processing result that recognizes voice from the voice data collected by the sound collector; and when a voice that satisfies a preset setting condition for the voice is detected based on the result of the voice recognition processing, the voice processing device notifies a user that a voice that satisfies the set condition has been detected, in accordance with a notification condition corresponding to the detected voice that is set in the set condition. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating a schematic configuration of a voice processing system according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a block diagram of the voice processing system shown in FIG. [Diagram 3] FIG. 13 is a diagram showing an example of a search list. [Figure 4] FIG. 13 is a diagram showing an example of a notification sound list. [Diagram 5] FIG. 11 is a diagram for explaining notification means and notification timing according to priority. [Figure 6] FIG. 13 is a diagram showing an example of a main screen. [Figure 7] FIG. 13 illustrates an example of a setting screen. [Figure 8] FIG. 13 is a diagram illustrating an example of a notification screen. [Figure 9] 3 is a diagram illustrating an example of processing by a section detection unit illustrated in FIG. 2. [Figure 10] 3 is a block diagram of the utterance storage unit shown in FIG. 2. [Figure 11] 3 is a flowchart showing an operation of an event detection process executed by the voice processing device shown in FIG. 2. [Figure 12] 3 is a flowchart showing an operation of a reproduction data output process executed by the audio processing device shown in FIG. 2; [Figure 13] 3 is a flowchart showing an operation of a reproduction data output process executed by the audio processing device shown in FIG. 2; [Figure 14] FIG. 13 is a diagram illustrating a schematic configuration of a voice processing system according to another embodiment of the present disclosure. [Figure 15] FIG. 13 is a diagram showing a schematic configuration of a voice processing system according to still another embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] There is room for improvement in the conventional technology. For example, depending on the contents of the detected voice, a user may preferentially want to be notified that a voice has been detected in some cases, and may preferentially not want to be notified in other cases. According to an embodiment of the present disclosure, it is possible to provide an improved voice processing device, a voice processing method, and a voice processing system.

[0010] In the present disclosure, "audio" includes any sound. For example, audio includes human voices, machine output sounds, animal cries, environmental sounds, and the like.

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0012] 1, the sound processing system 1 includes a sound collector 10 and a sound processing device 20. The sound collector 10 and the sound processing device 20 can communicate with each other via a communication line. The communication line includes at least one of a wired line and a wireless line.

[0013] In this embodiment, the sound collector 10 is an earphone. However, the sound collector 10 is not limited to an earphone. The sound collector 10 may be a headphone or the like. The sound collector 10 is worn by a user. The sound collector 10 can output music or the like. The sound collector 10 may include an earphone unit that is worn on the left ear of the user and an earphone unit that is worn on the right ear of the user.

[0014] The sound collector 10 collects sounds around the sound collector 10. When worn by a user, the sound collector 10 collects sounds around the user. The sound collector 10 outputs the collected sounds around the user under the control of the sound processing device 20. With this configuration, the user can hear sounds around them while wearing the sound collector 10.

[0015] In this embodiment, the voice processing device 20 is a terminal device. The terminal device that is the voice processing device 20 is, for example, a mobile phone, a smartphone, a tablet, a personal computer (PC), etc. However, the voice processing device 20 is not limited to a terminal device.

[0016] The sound processing device 20 is operated by a user. The user can operate the sound processing device 20 to set the sound collector 10 and the like.

[0017] The voice processing device 20 controls the sound collector 10 to collect voices around the user. When the voice processing device 20 detects a voice that satisfies a preset condition from among the collected voices around the user, it notifies the user that the voice that satisfies the set condition has been detected. Details of this process will be described later.

[0018] Fig. 2 is a block diagram of the voice processing system 1 shown in Fig. 1. In Fig. 2, main flows of data etc. are indicated by solid lines.

[0019] The sound collector 10 includes a microphone 11, a speaker 12, a communication unit 13, a storage unit 14, and a control unit 15.

[0020] The microphone 11 is capable of collecting sounds around the sound collector 10. The microphone 11 includes a left microphone and a right microphone. The left microphone may be included in an earphone unit included in the sound collector 10 that is worn on the left ear of the user. The right microphone may be included in an earphone unit included in the sound collector 10 that is worn on the right side of the user. For example, the microphone 11 is a stereo microphone or the like.

[0021] The speaker 12 is capable of outputting sound. The speaker 12 includes a left speaker and a right speaker. The left speaker may be included in an earphone unit included in the sound collector 10 and worn on the left ear of the user. The right speaker may be included in an earphone unit included in the sound collector 10 and worn on the right side of the user. For example, the speaker 12 may be a stereo speaker.

[0022] The communication unit 13 includes at least one communication module capable of communicating with the audio processing device 20 via a communication line. The communication module is a communication module compatible with the standard of the communication line. The standard of the communication line is, for example, a wired communication standard or a short-range wireless communication standard including Bluetooth (registered trademark), infrared, and NFC (Near Field Communication), etc.

[0023] The storage unit 14 is configured to include at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a RAM (Random Access Memory) or a ROM (Read Only Memory). The RAM is, for example, a SRAM (Static Random Access Memory) or a DRAM (Dynamic Random Access Memory). The ROM is, for example, an EEPROM (Electrically Erasable Programmable Read Only Memory). The storage unit 14 may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 14 stores data used in the operation of the sound collector 10 and data obtained by the operation of the sound collector 10. For example, the storage unit 14 stores a system program, an application program, embedded software, and the like.

[0024] The control unit 15 is configured to include at least one processor, at least one dedicated circuit, or a combination of these. The processor is a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for specific processing. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 15 executes processing related to the operation of the sound collector 10 while controlling each part of the sound collector 10.

[0025] In this embodiment, the control unit 15 includes an audio acquisition unit 16, an audio playback unit 17, and an accumulation unit 18. The accumulation unit 18 is configured to include components that are the same as or similar to those of the memory unit 14. At least a part of the accumulation unit 18 may be a part of the memory unit 14. The operation of the accumulation unit 18 is executed by a processor of the control unit 15 or the like.

[0026] The voice acquisition unit 16 acquires digital data of the voice from the analog data of the voice collected by the microphone 11. In this embodiment, the voice acquisition unit 16 acquires voice sampling data as digital data of the voice by sampling the analog data of the voice at a preset sampling rate.

[0027] The voice acquisition unit 16 outputs the voice sampling data to the voice reproduction unit 17. The voice acquisition unit 16 also transmits the voice sampling data to the voice processing device 20 via the communication unit 13.

[0028] When the microphone 11 includes a left microphone and a right microphone, the voice acquisition unit 16 may acquire left voice sampling data from analog data of the voice collected by the left microphone. The voice acquisition unit 16 may also acquire right voice sampling data from analog data of the voice collected by the right microphone. The voice acquisition unit 16 may transmit the left voice sampling data and the right voice sampling data to the voice processing device 20 via the communication unit 13. Hereinafter, when there is no particular distinction between the left voice sampling data and the right voice sampling data, they are also simply referred to as "voice sampling data".

[0029] The audio replay unit 17 acquires audio sampling data from the audio acquisition unit 16. The audio replay unit 17 receives a replay flag from the audio processing device 20 via the communication unit 13.

[0030] The replay flag is set to True or False. When the replay flag is False, the voice processing system 1 operates in through mode. The through mode is a mode in which the voice data collected by the sound collector 10 is output from the sound collector 10 without going through the sound processing device 20. When the replay flag is True, the voice processing system 1 operates in playback mode. The playback mode is a mode in which the sound collector 10 outputs playback data acquired from the sound processing device 20. The conditions under which the replay flag is set to True or False will be described later.

[0031] When the replay flag is False, that is, when the audio processing system 1 is in the through mode, the audio replay unit 17 causes the speaker 12 to output the audio sampling data acquired from the audio acquisition unit 16 .

[0032] When the replay flag is True, that is, when the audio processing system 1 is in the playback mode, the audio playback unit 17 causes the speaker 12 to output the playback data stored in the storage unit 18 .

[0033] The audio playback unit 17 receives the notification sound file from the audio processing device 20 via the communication unit 13. When the audio processing device 20 detects audio that satisfies the set conditions, the notification sound file is transmitted from the audio processing device 20 to the audio playback unit 17. When the audio playback unit 17 receives the notification sound file, it outputs the notification sound to the speaker 12. With this configuration, the user can know that audio that satisfies the set conditions has been detected.

[0034] The storage unit 18 stores playback data. The playback data is data transmitted from the audio processing device 20 to the sound collector 10. When the control unit 15 receives playback data from the audio processing device 20 via the communication unit 13, it stores the received playback data in the storage unit 18. The control unit 15 can receive a playback stop instruction and a replay stop instruction, which will be described later, from the audio processing device 20 via the communication unit 13. When the control unit 15 receives a playback stop instruction or a replay stop instruction, it erases the playback data stored in the storage unit 18.

[0035] The control unit 15 may receive the left playback data and the right playback data from the audio processing device 20 and store them in the storage unit 18. In this case, the audio playback unit 17 may output the left playback data stored in the storage unit 18 to the left speaker of the speaker 12, and output the right playback data stored in the storage unit 18 to the right speaker of the speaker 12.

[0036] The voice processing device 20 includes a communication unit 21, an input unit 22, a display unit 23, a vibration unit 24, a storage unit 26, and a control unit 27.

[0037] The communication unit 21 is configured to include at least one communication module capable of communicating with the sound collector 10 via a communication line. The communication module is a communication module that complies with the standard of the communication line. The standard of the communication line is, for example, a wired communication standard, or a short-range wireless communication standard including Bluetooth (registered trademark), infrared, NFC, etc.

[0038] The communication unit 21 may further include at least one communication module that can be connected to any network including a mobile communication network and the Internet, etc. The communication module is, for example, a communication module compatible with a mobile communication standard such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation).

[0039] The input unit 22 can receive input from a user. The input unit 22 includes at least one input interface that can receive input from a user. The input interface is, for example, a physical key, a capacitive key, a pointing device, a touch screen that is integrated with a display, a microphone, or the like.

[0040] The display unit 23 is capable of displaying data. The display unit 23 is, for example, a display. The display is, for example, a Liquid Crystal Display (LCD) or an organic EL (Electro Luminescence) display.

[0041] The vibration unit 24 is capable of vibrating the sound processing device 20. The vibration unit 24 includes a vibration element. The vibration element is, for example, a piezoelectric element.

[0042] The light emitting unit 25 is capable of emitting light. The light emitting unit 25 is, for example, an LED (Light Emitting Diode).

[0043] The storage unit 26 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a RAM or a ROM. The RAM is, for example, an SRAM or a DRAM. The ROM is, for example, an EEPROM. The storage unit 26 may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 26 stores data used in the operation of the audio processing device 20 and data obtained by the operation of the audio processing device 20. For example, the storage unit 26 stores a system program, an application program, embedded software, and the like.

[0044] The storage unit 26 stores, for example, a search list as shown in Fig. 3 described later, and a notification sound list and a notification sound file as shown in Fig. 4 described later. The storage unit 26 stores, for example, a notification list described later.

[0045] The control unit 27 includes at least one processor, at least one dedicated circuit, or a combination of these. The processor is a general-purpose processor such as a CPU or a GPU, or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, an FPGA or an ASIC. The control unit 27 executes processes related to the operation of the audio processing device 20 while controlling each part of the audio processing device 20.

[0046] The control unit 27 executes a voice recognition process for recognizing voice from the voice data. However, the control unit 27 may obtain the result of the voice recognition process executed by an external device. When the control unit 27 detects a voice that satisfies a set condition based on the result of the voice recognition process, the control unit 27 notifies the user that a voice that satisfies the set condition has been detected. The set condition is a condition that is set in advance regarding the voice. The control unit 27 notifies the user that a voice that satisfies the set condition has been detected according to a notification condition corresponding to the detected voice that is set in the set condition.

[0047] The notification condition is a condition for determining the priority of notifying the user that a voice has been detected. The higher the priority, the earlier the timing of notifying the user may be. The higher the priority, the earlier the notification may be made to the user by a notification means that is easier for the user to notice. For example, as described above, the user wears an earphone that is the sound collector 10. Therefore, when a sound such as a notification sound is output from the sound collector 10, the user can immediately notice the sound. In other words, a notification means using a sound such as a notification sound has a higher priority than a notification means using visual information presentation or the like. As will be described later, the user may be notified that a voice has been detected by playing the detected sound. In this case, the higher the priority, the earlier the timing of playing the detected sound may be. In addition, when the priority is low, the detected sound may be played at any timing.

[0048] The notification condition includes a first condition and a second condition. When the notification condition satisfies the second condition, the priority of notifying the user is lower than when the notification condition satisfies the first condition. The first condition includes a third condition and a fourth condition. When the notification condition satisfies the fourth condition, the priority of notifying the user is lower than when the notification condition satisfies the third condition.

[0049] In this embodiment, the notification condition is set by priority. The priority indicates the priority order for notifying the user of the voice that satisfies the set condition. The priority may be set in multiple stages. The higher the priority, the higher the priority for notifying the user. In this embodiment, the priority is set in three stages including "high", "medium" and "low", as shown in Figures 3 and 4. The "high" priority is the highest priority among the three priority stages. The "medium" priority is the middle priority among the three priority stages. The "low" priority is the lowest priority among the three priority stages.

[0050] In this embodiment, the notification condition satisfies the first condition when the priority is "high" or "medium". The notification condition satisfies the second condition when the priority is "low". The notification condition satisfies the third condition when the priority is "high". The notification condition satisfies the fourth condition when the priority is "medium".

[0051] When the notification condition corresponding to the detected voice satisfies the first condition, i.e., when the priority corresponding to the detected voice is "medium" or "high," the control unit 27 may notify the user that a voice satisfying the set condition has been detected by playing a notification sound. With this configuration, the user can immediately notice that a voice has been detected.

[0052] When the notification condition corresponding to the detected voice satisfies the first condition, i.e., when the priority corresponding to the detected voice is "medium" or "high", the control unit 27 may play the notification sound and then play the voice that satisfies the set condition. With this configuration, when the priority is "medium" or "high", the detected voice is automatically played. When the priority is "medium" or "high", there is a high possibility that the user would like to immediately check the contents of the detected voice. By automatically playing the voice that satisfies the set condition when the notification condition satisfies the first condition, the user can immediately check the detected voice. This can improve the user's convenience.

[0053] When the notification condition corresponding to the detected sound satisfies the second condition, that is, when the priority corresponding to the detected sound is "low", the control unit 27 may notify the user that a sound satisfying the set condition has been detected by presenting visual information to the user. When the priority is "low", a notification appropriate for the low priority can be given by presenting visual information to the user instead of playing a notification sound.

[0054] In this embodiment, the set condition is that the speech contains a preset search word. A priority is set for each search word. The search word includes, for example, at least one of letters and numbers. The search word may be any information that can be processed as text data. In this embodiment, when the control unit 27 detects an utterance that includes the search word as a voice that satisfies the set condition, the control unit 27 notifies the user that the utterance has been detected.

[0055] FIG. 3 shows a search list. The search list associates search words with priorities set for the search words. A "high" priority is set for the search word "flight 153." A "medium" priority is set for the search word "hello." A "low" priority is set for the search word "good morning." For example, the control unit 27 generates the search list based on a user's input to a setting screen 50 as shown in FIG. 7 described below.

[0056] FIG. 4 shows a notification sound list. The notification sound list associates a priority with a notification sound set for the priority. The notification sound is used when notifying a user that an utterance has been detected. In this embodiment, the notification sound is also used when an utterance corresponding to a priority of "low" is detected. However, when an utterance corresponding to a priority of "low" is detected, the notification sound does not have to be used. In FIG. 4, the priority is associated with a notification sound file. The notification sound file is a file for storing the notification sound on a computer. The notification sound file "ring.wav" is associated with the priority of "high". The notification sound file "alert.wav" is associated with the priority of "medium". The notification sound file "notify.wav" is associated with the priority of "low".

[0057] A notification according to priority according to this embodiment will be described with reference to Fig. 5. In Fig. 5, the notification means is a means for notifying the user that an utterance including a search word has been detected. The notification timing is the timing for notifying the user that an utterance including a search word has been detected. The playback timing is the timing for playing back the detected utterance.

[0058] As shown in FIG. 5, when the control unit 27 detects an utterance corresponding to the priority level of "high", the control unit 27 uses a notification sound and vibration by the vibration unit 24 as a notification means. The control unit 27 sets the notification timing to the timing immediately after detecting the search word. The control unit 27 sets the playback timing to the timing immediately after notifying that the utterance has been detected. That is, when the notification condition satisfies the third condition, the control unit 27 controls so that the notification sound is played and the playback of the utterance is started immediately after detecting the search word. For example, the priority level of "high" is set for the search word "flight 153". In this case, the control unit 27 sets the notification timing to the timing immediately after detecting the search word "flight 153". That is, the control unit 27 vibrates the vibration unit 24 and plays the notification sound immediately after detecting the search word "flight 153". In addition, the control unit 27 sets the playback timing to the timing immediately after playing the notification sound, and controls so that the utterance "Flight 153 is scheduled to depart 20 minutes late" is played. With this configuration, immediately after detecting the search word "Flight 153," a notification sound is played and playback of the speech "Flight 153 is scheduled to depart 20 minutes late" begins. In addition, speech containing the search word is automatically played.

[0059] As shown in FIG. 5, when the control unit 27 detects an utterance corresponding to the priority level of "medium", the control unit 27 uses a notification sound and vibration by the vibration unit 24 as a notification means. The control unit 27 sets the notification timing to the timing immediately after the utterance including the search word is ended. The control unit 27 sets the playback timing to the timing immediately after the notification that the utterance has been detected. That is, when the notification condition satisfies the fourth condition, the control unit 27 controls so that the notification sound is played and the playback of the utterance is started immediately after the utterance including the search word is ended. For example, the priority level of "medium" is set for the search word "flight 153". In this case, the control unit 27 sets the notification timing to the timing immediately after the utterance "flight 153 is scheduled to depart 20 minutes late" is ended, vibrates the vibration unit 24, and plays the notification sound. The control unit 27 also controls so that the playback timing is immediately after the playback of the notification sound, and the utterance "flight 153 is scheduled to depart 20 minutes late" is played. With this configuration, immediately after the utterance "Flight 153 is scheduled to depart 20 minutes late" is finished, a notification sound is played and playback of the utterance "Flight 153 is scheduled to depart 20 minutes late" begins. In addition, utterances that include search words are automatically played.

[0060] As shown in FIG. 5, when the control unit 27 detects an utterance corresponding to a priority level of "low", the control unit 27 uses a screen display by the display unit 23 and light emission by the light emission unit 25 as a notification means. The screen display and light emission are examples of notification means for presenting visual information to the user. The control unit 27 causes the display unit 23 to display a notification list as a screen display. The notification list is a list of audio information that satisfies the detected setting condition. In this embodiment, the notification list is a list of event information. The event is an utterance that includes a search word. Details of the notification list will be described later. The control unit 27 sets the playback timing to immediately after the user instructs playback of the utterance. In other words, when the notification condition satisfies the second condition, the control unit 27 plays back the utterance that includes the search word based on the user's input. With this configuration, the utterance that includes the search word is manually played back.

[0061] <Input / output processing> The control unit 27 receives an input from a user through the input unit 22. The control unit 27 selects a screen to be displayed on the display unit 23 based on the input received through the input unit 22. For example, the control unit 27 displays a screen as shown in Fig. 6, Fig. 7 or Fig. 8 based on the input received through the input unit 22. In the configurations shown in Figs. 6 to 8, the input unit 22 is a touch screen provided integrally with the display of the display unit 23.

[0062] A main screen 40 as shown in FIG. 6 includes an area 41, an area 42, an area 43, and an area 44.

[0063] The state of the sound collector 10 is displayed in the area 41. In Fig. 6, the area 41 displays information "Replaying..." which indicates that the sound collector 10 is replaying.

[0064] When the audio processing system 1 is in the through mode, the text "REPLAY START" is displayed in the area 42. When the audio processing system 1 is in the playback mode, the text "REPLAY STOP" is displayed in the area 42. The control unit 27 can accept an input to the area 42 via the input unit 22.

[0065] When the words "REPLAY START" are displayed in area 42, that is, when the voice processing system 1 is in the through mode, the control unit 27 can accept the start of replay by accepting an input to area 42 via the input unit 22. When accepting the start of replay, the control unit 27 sets a replay flag to True and outputs a replay instruction to the utterance accumulation unit 32 described below.

[0066] When the words "Stop Replay" are displayed in the area 42, that is, when the sound processing system 1 is in the playback mode, the control unit 27 can accept a replay stop by accepting an input to the area 42 via the input unit 22. When accepting a replay stop, the control unit 27 sets the replay flag to False and transmits a replay stop instruction to the sound collector 10 via the communication unit 21.

[0067] The words "Notification List" are displayed in area 43. Control unit 27 can accept input to area 43 via input unit 22. When control unit 27 accepts input to area 43 via input unit 22, control unit 27 causes display unit 23 to display a notification screen 60 as shown in FIG.

[0068] The word "Settings" is displayed in area 44. Control unit 27 can accept input to area 44 via input unit 22. When control unit 27 accepts input to area 44 via input unit 22, it causes display unit 23 to display a setting screen 50 as shown in FIG.

[0069] 7 is a screen for a user to perform various settings. The setting screen 50 includes an area 51, an area 52, an area 53, an area 54, an area 55, and an area 56.

[0070] The words "Add search word" are displayed in area 51. Control unit 27 can accept input to area 51 via input unit 22. Control unit 27 accepts, from area 51, input of a search word and an input of a priority corresponding to the search word.

[0071] The set search word is displayed in area 52. In Fig. 7, the search words "Flight 153", "Hello", and "Good morning" are displayed in area 52. The control unit 27 can accept an input to area 52 by the input unit 22. When the control unit 27 accepts an input to area 52 by the input unit 22, it causes the display unit 23 to display a search list as shown in Fig. 3.

[0072] The area 53 displays the words "Recording Buffer Setting." Area 53 is used to set the length of time for recording the sound collected by the sound collector 10. In this embodiment, audio sampling data for the recording time is accumulated in a ring buffer 34 as shown in FIG. 10 described below. The control unit 27 can accept input to area 53 via the input unit 22. The control unit 27 accepts input of recording times such as 5 seconds, 10 seconds, and 15 seconds. The control unit 27 stores the accepted recording time information in the memory unit 26.

[0073] The word "speed setting" is displayed in the area 54. The area 54 is used to set the playback speed of the sound output from the sound collector 10. The control unit 27 can accept input to the area 54 via the input unit 22. The control unit 27 accepts input of sound speeds such as 1x speed, 1.1x speed, and 1.2x speed. The control unit 27 stores the accepted sound speed information in the storage unit 26.

[0074] The words "audio threshold setting" are displayed in area 55. Area 55 is used to set an audio threshold at which the audio collected by sound collector 10 is cut off as noise. In this embodiment, audio below the audio threshold is cut off as noise. Control unit 27 can receive input to area 55 via input unit 22. Control unit 27 receives input of an audio threshold ranging from -50 [dBA] to -5 [dBA], for example. Control unit 27 stores the received audio threshold information in memory unit 26.

[0075] The words "SETTING END" are displayed in area 56. Control unit 27 is capable of accepting input to area 56 via input unit 22. When control unit 27 accepts input to area 56 via input unit 22, it causes main screen 40 as shown in FIG.

[0076] 8 is a screen for notifying the user of various information. The notification screen 60 includes an area 61, an area 62, an area 63, and an area 64.

[0077] A notification list is displayed in area 61. As described above, the notification list is a list of event information. As described above, an event is an utterance including a search word. The control unit 27 displays event information of events included in the notification list that have a "low" priority in area 61. However, the control unit 27 may display all event information included in the notification list in area 61 regardless of priority. The control unit 27 can accept input for each event in the notification list displayed in area 61 through the input unit 22. The control unit 27 accepts selection of an event in the notification list by accepting input for each event in the notification list from area 61 through the input unit 22.

[0078] The words "display details" are displayed in area 62. The control unit 27 can accept input to area 62 via the input unit 22. The control unit 27 can accept a selection of an event included in the notification list from area 61, and can further accept input to area 62 via the input unit 22. In this case, the control unit 27 causes the display unit 23 to display details of the event information selected from area 61 in area 61. For example, the control unit 27 causes the left voice recognition result and the right voice recognition result, which will be described later, to be displayed as details of the event information.

[0079] The area 63 displays the words "Start playback / Stop playback". The control unit 27 can accept a playback start or playback stop by accepting an input to the area 63 by the input unit 22. When an utterance is not being played back, the control unit 27 accepts a selection of an event included in the notification list from the area 61, and further accepts an input to the area 63 by the input unit 22 to accept an event playback start. When an event playback start is accepted, the control unit 27 controls so that the event selected from the area 61, that is, the utterance, is played back. In this embodiment, the control unit 27 refers to the notification list in the storage unit 26 and acquires an event ID (described later) of the event selected from the area 61. The control unit 27 outputs the event ID and a playback start instruction to the utterance storage unit 36 ​​(described later) and controls so that the utterance is played back. Furthermore, when an utterance is being played back, the control unit 27 accepts an input to the area 63 by the input unit 22 to accept an event playback stop. When the control unit 27 receives a command to stop the playback of the event, the control unit 27 controls the playback of the speech to stop. In this embodiment, the control unit 27 transmits a playback stop command to the sound collector 10 via the communication unit 21, and controls the playback of the speech to stop.

[0080] The word "Back" is displayed in area 64. Control unit 27 is capable of accepting input to area 64 via input unit 22. When control unit 27 accepts input to area 64 via input unit 22, control unit 27 causes main screen 40 as shown in FIG.

[0081] <Audio processing> 2, control unit 27 includes a section detection unit 28, a voice recognition unit 29, an event detection unit 30, an utterance notification unit 31, an utterance accumulation unit 32, a voice modulation unit 35, and an utterance holding unit 36. Utterance holding unit 36 ​​is configured to include components that are the same as or similar to those of memory unit 26. At least a part of utterance holding unit 36 ​​may be a part of memory unit 26. The operation of utterance holding unit 36 ​​is executed by a processor of control unit 27, etc.

[0082] The section detection unit 28 receives the voice sampling data from the sound collector 10 via the communication unit 21. The section detection unit 28 detects speech sections from the voice sampling data. A speech section is a section in which a speech state continues. The section detection unit 28 can also detect non-speech sections by detecting a speech section from the voice sampling data. A non-speech section is a section in which a non-speech state continues. The start point of a speech section is also referred to as the "speech start time". The start point of a speech section is the end point of a non-speech section. The end point of a speech section is also referred to as the "speech end time". The end point of a speech section is the start point of a non-speech section.

[0083] An example of the processing of the section detection unit 28 will be described with reference to Fig. 9. However, the processing of the section detection unit 28 is not limited to the processing described with reference to Fig. 9. The section detection unit 28 may detect the speech section from the voice sampling data by any method. As another example, the section detection unit 28 may detect the speech section from the voice sampling data by a machine learning model generated using any machine learning algorithm.

[0084] In Fig. 9, the horizontal axis indicates time. The voice sampling data shown in Fig. 9 is acquired by the voice acquisition unit 16 of the sound collector 10. The section detection unit 28 acquires voice section detection data from the voice sampling data. The voice section detection data is data obtained by averaging the power of the voice sampling data over a preset time width. The time width of the voice section detection data may be set based on the specifications of the voice processing device 20, etc. In Fig. 9, one piece of voice section detection data is shown as one square. The time width of this one square, i.e., the time width of one piece of voice section detection data, is, for example, 200 [ms].

[0085] The section detection unit 28 acquires information on the voice threshold from the storage unit 26, and classifies the voice section detection data into voice data and non-voice data. In FIG. 9, voice data is data that is darkly colored among the voice section detection data shown as squares. Non-voice data is data that is white among the voice section detection data shown as squares. If the value of the voice section detection data is null, the section detection unit 28 classifies the voice section detection data into non-voice data. If the value of the voice section detection data is not null and the voice section detection data is less than the voice threshold, the section detection unit 28 classifies the voice section detection data into non-voice data. If the value of the voice section detection data is not null and the value of the voice section detection data is equal to or greater than the voice threshold, the section detection unit 28 classifies the voice section detection data into voice data.

[0086] The section detection unit 28 detects a section in which voice data continues without a set time gap as a speech section. The set time may be set based on the language processed by the voice processing device 20. When the language to be processed is Japanese, the set time is, for example, 500 [ms]. In FIG. 9, when the section detection unit 28 detects voice data after non-voice data continues beyond the set time, the section detection unit 28 specifies the time when the voice data is detected as the speech start time. For example, the section detection unit 28 specifies time t1 as the speech start time. After specifying the speech start time, if the section detection unit 28 determines that the non-voice data has continued beyond the set time, the section detection unit 28 specifies the time when the section detection unit 28 determines that the non-voice data has continued beyond the set time, the section detection unit 28 specifies the time when the section detection unit 28 determines that the non-voice data has continued as the speech end time. For example, the section detection unit 28 specifies time t2 as the speech end time. The section detection unit 28 detects the section from the speech start time to the speech end time as the speech section.

[0087] The section detection unit 28 may receive left voice sampling data and right voice sampling data from the sound collector 10. In this case, when non-voice data continues for a set time in both the left and right voice sampling data and then voice data is detected in either the left or right voice sampling data, the section detection unit 28 may determine the time when the voice data is detected as the speech start time. Furthermore, when the section detection unit 28 determines that non-voice data has continued for a set time in both the left and right voice sampling data, it may determine the time when the section detection unit 28 determines that the non-voice data has continued for a set time in both the left and right voice sampling data as the speech end time.

[0088] When the section detection unit 28 identifies the speech start time from the voice sampling data, it generates an utterance ID. The utterance ID is identification information that can uniquely identify each utterance. The section detection unit 28 outputs the utterance start time information and the utterance ID to the voice recognition unit 29 and the utterance storage unit 32, respectively.

[0089] When the section detection unit 28 identifies the end point of the utterance from the voice sampling data, the section detection unit 28 outputs information on the end point of the utterance to the voice recognition unit 29 and the utterance storage unit 32, respectively.

[0090] The section detection unit 28 sequentially outputs the voice sampling data received from the sound collector 10 to the voice recognition unit 29 and the speech accumulation unit 32, respectively.

[0091] The voice recognition unit 29 acquires information on the start time of an utterance and an utterance ID from the section detection unit 28. When the voice recognition unit 29 acquires the information on the start time of an utterance and the like, it executes a voice recognition process for recognizing voice on the voice sampling data successively acquired from the section detection unit 28. In this embodiment, the voice recognition unit 29 recognizes voice by converting the voice data included in the voice sampling data into text data by the voice recognition process.

[0092] The voice recognition unit 29 outputs the information on the start time of the utterance and the utterance ID acquired from the section detection unit 28 to the event detection unit 30. After outputting the information on the start time of the utterance and the like to the event detection unit 30, the voice recognition unit 29 sequentially outputs text data as a voice recognition result to the event detection unit 30.

[0093] The voice recognition unit 29 acquires information on the end of the speech from the section detection unit 28. When the voice recognition unit 29 acquires the information on the end of the speech, it ends the voice recognition process. The voice recognition unit 29 outputs the information on the end of the speech acquired from the section detection unit 28 to the event detection unit 30. After that, the voice recognition unit 29 can acquire information on a new start of the speech and an utterance ID from the section detection unit 28. When the voice recognition unit 29 acquires the information on a new start of the speech and the like, it executes the voice recognition process again on the voice sampling data successively acquired from the section detection unit 28.

[0094] The voice recognition unit 29 may acquire the left voice sampling data and the right voice sampling data from the section detection unit 28. In this case, the voice recognition unit 29 may convert each of the left voice sampling data and the right voice sampling data into text data. Hereinafter, the text data acquired from the left voice sampling data is also referred to as "left text data" or "left voice recognition result". The text data acquired from the right voice sampling data is also referred to as "right text data" or "right voice recognition result".

[0095] The event detection unit 30 acquires information on the start time of an utterance and an utterance ID from the voice recognition unit 29. After acquiring the information on the start time of an utterance, the event detection unit 30 sequentially acquires text data from the voice recognition unit 29. The event detection unit 30 refers to a search list as shown in Fig. 3 and determines whether or not the text data sequentially acquired from the voice recognition unit 29 includes any of the search words in the search list.

[0096] When the event detection unit 30 determines that the text data contains a search word, it detects an utterance containing the search word as an event. When the event detection unit 30 detects an event, it acquires the utterance ID acquired from the voice recognition unit 29 as an event ID. Furthermore, the event detection unit 30 refers to a search list as shown in Fig. 3 and acquires a priority corresponding to the search word included in the text data. When the event detection unit 30 acquires the priority, it executes a notification process according to the priority.

[0097] In the case where the priority is "high", when the event detection unit 30 determines that the text data contains a search word, it outputs an event ID and an output instruction to the speech storage unit 32 and outputs a priority of "high" to the speech notification unit 31. The output instruction is an instruction to cause the speech storage unit 32 to output the voice sampling data corresponding to the event ID as playback data to the voice modulation unit 35. When the event detection unit 30 outputs the output instruction, it sets the replay flag to True. In this way, when the priority is "high", the output instruction etc. are output to the speech storage unit 32 etc. immediately after the search word included in the text data is detected. With this configuration, when the priority is "high" as shown in FIG. 5, a notification sound is played and playback of the speech is started immediately after the search word is detected.

[0098] When the priority is "medium", upon acquiring information on the end of the utterance from the voice recognition unit 29, the event detection unit 30 outputs an event ID and an output instruction to the utterance storage unit 32, and outputs a priority of "medium" to the utterance notification unit 31. When the event detection unit 30 outputs the output instruction, it sets the replay flag to True. In this way, when the priority is "medium", an output instruction etc. is output to the utterance storage unit 32 etc. at the end of the utterance. With this configuration, when the priority is "medium" as shown in FIG. 5, a notification sound is played and playback of the utterance begins immediately after the utterance including the search word ends.

[0099] When the priority is "low", the event detection unit 30, upon acquiring information on the end time of the utterance from the voice recognition unit 29, outputs an event ID and a storage instruction to the utterance storage unit 32, and outputs a priority of "low" to the utterance notification unit 31. The storage instruction is an instruction to the utterance storage unit 32 to output the voice sampling data corresponding to the event ID to the utterance storage unit 36. The voice sampling data stored in the utterance storage unit 36 ​​is played back when the user instructs playback, as described above with reference to FIG. 8. With this configuration, when the priority is "low" as shown in FIG. 5, the utterance including the search word is played back immediately after the user instructs playback.

[0100] The event detection unit 30 updates the notification list stored in the storage unit 26 based on the event ID, the priority, the detection date and time when the event was detected, and the search word included in the text data. The notification list in the storage unit 26 includes, for example, associations between the event ID, the priority, the detection date and time when the event was detected, the search word, and the text data. As an example of the update process, the event detection unit 30 associates the event ID, the priority, the detection date and time, the search word, and the text data. The event detection unit 30 updates the notification list by including this association in the notification list.

[0101] The event detection unit 30 determines whether or not the search word is included in the text data until it acquires information on the end of the utterance from the voice recognition unit 29. If the event detection unit 30 determines that the search word is not included in the text data sequentially acquired from the voice recognition unit 29 at the time when it acquires the information on the end of the utterance, it acquires the utterance ID acquired from the voice recognition unit 29 as a clear event ID. The event detection unit 30 outputs the clear event ID to the utterance accumulation unit 32.

[0102] The event detection unit 30 may acquire information on the start time of a new utterance and an utterance ID from the voice recognition unit 29. When the event detection unit 30 acquires the information on the start time of a new utterance, etc., it determines whether or not any of the search words in the search list is included in the text data successively newly acquired from the voice recognition unit 29.

[0103] The event detection unit 30 may acquire left text data and right text data from the voice recognition unit 29. In this case, when the event detection unit 30 determines that the search word is included in either the left text data or the right text data, the event detection unit 30 may detect an utterance including the search word as an event. When the event detection unit 30 determines that the search word is not included in either the left text data or the right text data, the event detection unit 30 may acquire an utterance ID corresponding to the text data as a clear event ID.

[0104] The speech notification unit 31 acquires the priority from the event detection unit 30. The speech notification unit 31 acquires a notification sound file corresponding to the priority from the storage unit 26. The speech notification unit 31 transmits the acquired notification sound file to the sound collector 10 via the communication unit 21.

[0105] 4, and acquires a notification sound file named "ring.wav" associated with the priority level "high" from the storage unit 26. The speech notification unit 31 transmits the acquired notification sound file to the sound collector 10 via the communication unit 21.

[0106] 4, and acquires a notification sound file named "alert.wav" associated with the priority level "medium" from the storage unit 26. The speech notification unit 31 transmits the acquired notification sound file to the sound collector 10 via the communication unit 21.

[0107] When the priority is "low", the speech notification unit 31 refers to the notification sound list as shown in Fig. 4 and acquires the notification sound file "notify.wav" associated with the priority of "low" from the storage unit 26. The speech notification unit 31 transmits the acquired notification sound file to the sound collector 10 via the communication unit 21.

[0108] 10, the utterance storage unit 32 has a data buffer 33 and a ring buffer 34. The data buffer 33 and the ring buffer 34 are configured to include components that are the same as or similar to those of the storage unit 26. At least a part of the data buffer 33 and the ring buffer 34 may be a part of the storage unit 26. The operation of the utterance storage unit 32 is executed by a processor of the control unit 27 or the like.

[0109] The utterance accumulation unit 32 acquires information on the start time of an utterance and an utterance ID from the section detection unit 28. When the utterance accumulation unit 32 acquires the information on the start time of an utterance and the like, the utterance accumulation unit 32 associates the voice sampling data successively acquired from the section detection unit 28 with the utterance ID and accumulates it in the data buffer 33. When the utterance accumulation unit 32 acquires information on a new start time of an utterance and a new utterance ID from the section detection unit 28, the utterance accumulation unit 32 associates the voice sampling data successively acquired from the section detection unit 28 with the new utterance ID and accumulates it in the data buffer 33. In FIG. 10, the data buffer 33 accumulates a plurality of voice sampling data corresponding to utterance ID1, a plurality of voice sampling data corresponding to utterance ID2, and a plurality of voice sampling data corresponding to utterance ID3.

[0110] The speech accumulation unit 32 receives the voice sampling data from the sound collector 10 via the communication unit 21. The speech accumulation unit 32 accumulates the voice sampling data received from the sound collector 10 in the ring buffer 34. The speech accumulation unit 32 refers to the recording time information stored in the memory unit 26, and accumulates the voice sampling data for the recording time in the ring buffer 34. The speech accumulation unit 32 accumulates the voice sampling data in chronological order in the ring buffer 34.

[0111] The utterance accumulation unit 32 can acquire a clear event ID from the event detection unit 30. When the utterance accumulation unit 32 acquires a clear event ID, the utterance accumulation unit 32 deletes, from the voice sampling data accumulated in the data buffer 33, the voice sampling data associated with the utterance ID that matches the clear event ID.

[0112] The utterance accumulation unit 32 may acquire an event ID and an output instruction from the event detection unit 30. When the utterance accumulation unit 32 acquires the output instruction, it identifies an utterance ID that matches the event ID acquired together with the output instruction from among the voice sampling data accumulated in the data buffer 33. The utterance accumulation unit 32 outputs the voice sampling data corresponding to the identified utterance ID as playback data to the voice modulation unit 35. The utterance accumulation unit 32 outputs the voice sampling data to the voice modulation unit 35 so that the voice sampling data is played back from the top voice sampling data. The top voice sampling data refers to the voice sampling data with the oldest time among multiple voice sampling data in chronological order.

[0113] The utterance storage unit 32 may acquire an event ID and a storage instruction from the event detection unit 30. When the utterance storage unit 32 acquires the storage instruction, it identifies an utterance ID that matches the event ID acquired together with the storage instruction from among the voice sampling data accumulated in the data buffer 33. The utterance storage unit 32 outputs the voice sampling data associated with the identified utterance ID to the utterance storage unit 36 ​​together with the event ID.

[0114] The utterance accumulation unit 32 may acquire a replay instruction. When the utterance accumulation unit 32 acquires a replay instruction, the utterance accumulation unit 32 outputs the voice sampling data accumulated in the ring buffer 34 to the voice modulation unit 35 as playback data so that the voice sampling data is played back from the leading voice sampling data.

[0115] 2, the audio modulation unit 35 acquires playback data from the speech accumulation unit 32. When the replay flag is True, the audio modulation unit 35 refers to the voice speed information stored in the storage unit 26, and modulates the playback data so that the playback data is played back as voice at that voice speed. The audio modulation unit 35 transmits the modulated playback data to the sound collector 10 via the communication unit 21.

[0116] The utterance storage unit 36 ​​acquires the event ID and the voice sampling data from the utterance accumulation unit 32. The utterance storage unit 36 ​​stores the acquired voice sampling data in association with the acquired event ID.

[0117] The speech storage unit 36 ​​may acquire an event ID and a playback start instruction. When the speech storage unit 36 ​​acquires the playback start instruction, it identifies the voice sampling data associated with the event ID. The speech storage unit 36 ​​transmits the identified voice sampling data to the sound collector 10 as playback data via the communication unit 21.

[0118] Fig. 11 is a flowchart showing the operation of the event detection process executed by the sound processing device 20 shown in Fig. 2. This operation corresponds to an example of the sound processing method according to the present embodiment. For example, when the transmission of sound sampling data from the sound collector 10 to the sound processing device 20 is started, the sound processing device 20 starts the process of step S1.

[0119] The section detection unit 28 receives the sound sampling data from the sound collector 10 via the communication unit 21 (step S1).

[0120] In the process of step S2, the section detection unit 28 sequentially outputs the voice sampling data acquired in the process of step S1 to the voice recognition unit 29 and the utterance accumulation unit 32, respectively.

[0121] In the process of step S2, the section detection unit 28 identifies the speech start time from the voice sampling data acquired in the process of step S1. When the section detection unit 28 identifies the speech start time, it generates an utterance ID. The section detection unit 28 outputs the information on the speech start time and the utterance ID to the voice recognition unit 29 and the utterance storage unit 32, respectively.

[0122] In the process of step S2, the section detection unit 28 identifies the end point of the utterance from the voice sampling data acquired in the process of step S1. When the section detection unit 28 identifies the end point of the utterance, the section detection unit 28 outputs information on the end point of the utterance to the voice recognition unit 29 and the utterance accumulation unit 32.

[0123] In the process of step S3, when the voice recognition unit 29 acquires information on the start of an utterance, etc. from the section detection unit 28, it sequentially converts the voice sampling data sequentially acquired from the section detection unit 28 into text data. When the voice recognition unit 29 outputs information on the start of an utterance, etc. to the event detection unit 30, it sequentially outputs text data as a voice recognition result to the event detection unit 30. When the voice recognition unit 29 acquires information on the end of an utterance from the section detection unit 28, it ends the voice recognition process. However, when the voice recognition unit 29 acquires information on a new start of an utterance, etc. from the section detection unit 28, it sequentially converts the voice sampling data sequentially acquired from the section detection unit 28 into text data.

[0124] In the process of step S4, the event detection unit 30 refers to a search list as shown in FIG. 3 and determines whether or not the text data successively acquired from the voice recognition unit 29 includes any of the search words in the search list.

[0125] When the event detection unit 30 determines that the search word is not included in the successively acquired text data at the time when the information on the end of the utterance is acquired from the voice recognition unit 29 (step S4: NO), the event detection unit 30 proceeds to the process of step S5. When the event detection unit 30 determines that the search word is included in the successively acquired text data from the voice recognition unit 29 before acquiring the information on the end of the utterance (step S4: YES), the event detection unit 30 proceeds to the process of step S6.

[0126] In the process of step S5, the event detection unit 30 obtains the utterance ID obtained from the voice recognition unit 29 as a clear event ID. The event detection unit 30 outputs the clear event ID to the utterance accumulation unit 32.

[0127] In the process of step S6, the event detection unit 30 detects an utterance including the search word as an event.

[0128] In the process of step S7, the event detection unit 30 acquires, as an event ID, the utterance ID acquired from the voice recognition unit 29. Furthermore, the event detection unit 30 refers to the search list as shown in Fig. 3 and acquires a priority corresponding to the search word included in the text data.

[0129] In the process of step S8, the event detection unit 30 executes a notification process according to the priority acquired in the process of step S7.

[0130] In the process of step S9, the event detection unit 30 updates the notification list stored in the storage unit 26 based on the event ID, the priority, the detection date and time when the event was detected, and the search word included in the text data.

[0131] 12 and 13 are flowcharts showing the operation of the playback data output process executed by the sound processing device 20 shown in Fig. 2. This operation corresponds to an example of the sound processing method according to the present embodiment. For example, when the transmission of sound sampling data from the sound collector 10 to the sound processing device 20 is started, the sound processing device 20 starts the process of step S11 as shown in Fig. 12.

[0132] In the process of step S11, the sound processing device 20 operates in the through mode. In the sound collector 10, the sound replay unit 17 outputs the sound sampling data acquired from the sound acquisition unit 16 to the speaker 12. In the process of step S11, the replay flag is set to False.

[0133] In the process of step S12, the control unit 27 determines whether or not a replay start has been accepted by accepting an input to an area 42 as shown in Fig. 6 from the input unit 22. If the control unit 27 determines that a replay start has been accepted (step S12: YES), the control unit 27 proceeds to the process of step S13. If the control unit 27 does not determine that a replay start has been accepted (step S12: NO), the control unit 27 proceeds to the process of step S18.

[0134] In the process of step S13, the control unit 27 sets the replay flag to True, and outputs a replay instruction to the utterance accumulation unit 32.

[0135] In the process of step S14, the utterance accumulation unit 32 acquires a replay instruction. Upon acquiring the replay instruction, the utterance accumulation unit 32 starts outputting the reproduction data from the ring buffer to the audio modulation unit .

[0136] In the process of step S15, the control unit 27 judges whether or not all the reproduced data has been output from the ring buffer 34 to the audio modulation unit 35. If the control unit 27 judges that all the reproduced data has been output (step S15: YES), the control unit 27 proceeds to the process of step S17. If the control unit 27 does not judge that all the reproduced data has been output (step S15: NO), the control unit 27 proceeds to the process of step S16.

[0137] In the process of step S16, the control unit 27 determines whether or not a replay stop has been received by receiving an input to an area 42 as shown in Fig. 6 from the input unit 22. If the control unit 27 determines that a replay stop has been received (step S16: YES), the control unit 27 proceeds to the process of step S17. If the control unit 27 does not determine that a replay stop has been received (step S16: NO), the control unit 27 returns to the process of step S15.

[0138] In the process of step S17, the control unit 27 sets the replay flag to False. After executing the process of step S17, the control unit 27 returns to the process of step S11.

[0139] In the process of step S18, the control unit 27 determines whether or not an event playback start has been received by receiving an input to an area 63 as shown in Fig. 8 via the input unit 22. If the control unit 27 determines that an event playback start has been received (step S18: YES), the control unit 27 proceeds to the process of step S19. If the control unit 27 does not determine that an event playback start has been received (step S18: NO), the control unit 27 proceeds to the process of step S24 as shown in Fig. 13.

[0140] In the process of step S19, the control unit 27 sets the replay flag to True. The control unit 27 also refers to the notification list in the storage unit 26, and acquires the event ID of the selected event from the area 61 as shown in Fig. 8. The control unit 27 outputs the event ID and a playback start instruction to the utterance storage unit 36.

[0141] In the process of step S20, the utterance storage unit 36 ​​acquires an event ID and a playback start instruction. When the utterance storage unit 36 ​​acquires the playback start instruction, it identifies the voice sampling data associated with the event ID. The utterance storage unit 36 ​​starts transmitting the identified voice sampling data, i.e., playback data, to the sound collector 10.

[0142] In the process of step S21, the control unit 27 judges whether or not all of the playback data has been transmitted from the speech storage unit 36 ​​to the sound collector 10. If the control unit 27 judges that all of the playback data has been transmitted (step S21: YES), the control unit 27 proceeds to the process of step S23. If the control unit 27 does not judge that all of the playback data has been transmitted (step S21: NO), the control unit 27 proceeds to the process of step S22.

[0143] In the process of step S22, the control unit 27 determines whether or not a request to stop playing the event has been received by receiving an input to an area 63 as shown in Fig. 8 via the input unit 22. If the control unit 27 determines that a request to stop playing the event has been received (step S22: YES), the control unit 27 proceeds to the process of step S23. If the control unit 27 does not determine that a request to stop playing the event has been received (step S22: NO), the control unit 27 returns to the process of step S21.

[0144] In the process of step S23, the control unit 27 sets the replay flag to False. After executing the process of step S23, the control unit 27 returns to the process of step S11.

[0145] 13, in the process of step S24, the utterance accumulation unit 32 judges whether or not an event ID and an output instruction are acquired from the event detection unit 30. If the utterance accumulation unit 32 judges that an event ID and an output instruction are acquired (step S24: YES), the utterance accumulation unit 32 proceeds to the process of step S25. If the utterance accumulation unit 32 does not judge that an event ID and an output instruction are acquired (step S24: NO), the utterance accumulation unit 32 proceeds to the process of step S30.

[0146] In the process of step S25, the replay flag is set to True. This replay flag is set to True by the event detection unit 30 when the event detection unit 30 outputs the output instruction in the process of step S24 to the utterance accumulation unit 32.

[0147] In the process of step S26, the utterance accumulation unit 32 identifies an utterance ID that matches the event ID acquired in the process of step S24 from among the voice sampling data accumulated in the data buffer 33. The utterance accumulation unit 32 acquires the voice sampling data corresponding to the identified utterance ID as playback data. The utterance accumulation unit 32 starts outputting the playback data from the data buffer 33 to the voice modulation unit 35.

[0148] In the process of step S27, the control unit 27 judges whether or not all of the reproduced data has been output from the data buffer 33 to the audio modulation unit 35. If the control unit 27 judges that all of the reproduced data has been output (step S27: YES), the control unit 27 proceeds to the process of step S29. If the control unit 27 does not judge that all of the reproduced data has been output (step S27: NO), the control unit 27 proceeds to the process of step S28.

[0149] In the process of step S28, the control unit 27 judges whether or not a replay stop has been received by receiving an input to an area 42 as shown in Fig. 6 from the input unit 22. If the control unit 27 judges that a replay stop has been received (step S28: YES), the control unit 27 proceeds to the process of step S29. If the control unit 27 does not judge that a replay stop has been received (step S28: NO), the control unit 27 returns to the process of step S27.

[0150] In the process of step S29, the control unit 27 sets the replay flag to False. After executing the process of step S29, the control unit 27 returns to the process of step S11 as shown in FIG.

[0151] In the process of step S30, the utterance storage unit 32 judges whether or not the event ID and the holding instruction are acquired from the event detection unit 30. If the utterance storage unit 32 judges that the event ID and the holding instruction are acquired (step S30: YES), the utterance storage unit 32 proceeds to the process of step S31. If the utterance storage unit 32 does not judge that the event ID and the holding instruction are acquired (step S30: NO), the control unit 27 returns to the process of step S11 as shown in FIG.

[0152] In the process of step S31, the utterance accumulation unit 32 identifies an utterance ID that matches the event ID acquired in the process of step S30 from among the voice sampling data accumulated in the data buffer 33. The utterance accumulation unit 32 outputs the voice sampling data associated with the identified utterance ID to the utterance holding unit 36 ​​together with the event ID.

[0153] After executing the process of step S31, the control unit 27 returns to the process of step S11 as shown in FIG.

[0154] In this way, in the voice processing device 20, when the control unit 27 detects a voice that satisfies the set condition, it notifies the user that the voice that satisfies the set condition has been detected according to the notification condition. In this embodiment, when the control unit 27 detects an utterance including a search word as a voice that satisfies the set condition, it notifies the user that the utterance has been detected according to the priority set for the search word.

[0155] Here, the user may preferentially want to be notified of the detection of a voice that satisfies a set condition depending on the content of the voice, or may not preferentially want to be notified of the detection. In this embodiment, the user can set a priority as a notification condition to separate voices into voices that are preferentially notified of their detection and voices that are not preferentially notified of their detection. Therefore, the voice processing device 20 can improve the convenience of the user.

[0156] In addition, when a voice satisfying a set condition is detected, if the voice is simply played back, the user may miss the played back voice. The voice processing device 20 notifies the user that a voice has been detected, thereby reducing the possibility that the user misses the played back voice.

[0157] Therefore, according to the present embodiment, an improved voice processing device 20, a voice processing method, and a voice processing system 1 can be provided.

[0158] Furthermore, when the notification condition corresponding to the detected voice satisfies the first condition, the control unit 27 of the voice processing device 20 may notify the user that the voice satisfying the set condition has been detected by playing a notification sound. As described above, by playing the notification sound, the user can immediately notice that the voice has been detected.

[0159] In addition, when the notification condition corresponding to the detected voice satisfies the first condition, the control unit 27 of the voice processing device 20 may play a voice that satisfies the set condition after playing the notification sound. With such a configuration, it is possible to improve the convenience for the user as described above.

[0160] Furthermore, when the notification condition corresponding to the detected voice satisfies the second condition, the control unit 27 of the voice processing device 20 may notify the user that a voice satisfying the set condition has been detected by presenting visual information to the user. As described above, the second condition has a lower priority for notifying the user than the first condition, the third condition, and the fourth condition. When the priority is low, a notification appropriate for the low priority can be given by presenting visual information to the user instead of playing a notification sound.

[0161] Furthermore, when the notification condition corresponding to the detected voice satisfies the second condition, the control unit 27 of the voice processing device 20 may present visual information to the user by presenting a notification list to the user. By looking at the notification list, the user can understand the date and time when the voice was detected and the circumstances of the voice detection.

[0162] Furthermore, the control unit 27 of the voice processing device 20 may play the detected voice based on the user's input when the notification condition corresponding to the detected voice satisfies the second condition. When the priority is low, it is highly likely that the user wants to check the detected voice later. With such a configuration, it is possible to improve the convenience for the user.

[0163] Furthermore, the control unit 27 of the voice processing device 20 may perform control so that, when the notification condition corresponding to the detected voice satisfies the third condition, the notification sound is reproduced and the reproduction of the utterance is started immediately after the search word is detected. With this configuration, when the priority is high, the user can immediately check the content of the utterance.

[0164] In addition, the control unit 27 of the voice processing device 20 may control so that, when the notification condition corresponding to the detected voice satisfies the fourth condition, the notification sound is reproduced and the reproduction of the utterance is started immediately after the utterance ends. By starting the reproduction of the utterance immediately after the utterance ends, the utterance uttered in real time and the reproduced utterance do not overlap. With such a configuration, the user can more accurately grasp the content of the reproduced utterance.

[0165] Furthermore, the control unit 27 of the voice processing device 20 may perform control so that the voice data of the speech section including the detected utterance is played back. As described above with reference to FIG. 9, the speech section is a section in which the voice data continues without a set time gap. By playing back the voice data of such a speech section, a group of utterances including the search word is played back. With this configuration, the user can understand the meaning of the utterance including the search word.

[0166] Furthermore, regardless of the priority, the control unit 27 of the voice processing device 20 may display, from among the information included in the notification list, the detection date and time when the event, i.e., the utterance, was detected and the search word included in the utterance on the display unit 23. With this configuration, the user can understand how the detected utterance was made.

[0167] (Other embodiments) 14 can provide a monitoring service for a baby, etc. The voice processing system 101 includes a sound collector 110 and a voice processing device 20.

[0168] The sound collector 110 and the sound processing device 20 are located farther away from each other than the sound collector 10 and the sound processing device 20 as shown in Fig. 1. For example, the sound collector 110 and the sound processing device 20 are located in separate rooms. The sound collector 110 is located in a room where a baby is present. The sound processing device 20 is located in a room where a user is present.

[0169] In another embodiment, the setting condition is a condition that the voice characteristic matches a previously set voice characteristic. The user may input the voice characteristic to be set as the setting condition from a microphone of the input unit 22 of the voice processing device 20 as shown in Fig. 2, and set the voice characteristic as the setting condition in the voice processing device 20. For example, the user sets the characteristic of a baby's crying as the setting condition in the voice processing device 20.

[0170] 2, the sound collector 110 includes a microphone 11, a speaker 12, a communication unit 13, a storage unit 14, and a control unit 15. The sound collector 110 does not necessarily need to include the speaker 12.

[0171] The audio processing device 20 may further include a speaker 12 as shown in Fig. 2. The control unit 27 of the audio processing device 20 may further include an audio playback unit 17 and a storage unit 18 as shown in Fig. 2.

[0172] In another embodiment, the storage unit 26 as shown in Fig. 2 stores a search list in which data indicating voice features are associated with priorities, instead of the search list as shown in Fig. 3. The data indicating voice features may be data of voice features that can be processed by a machine learning model used by the voice recognition unit 29. The voice features are, for example, Mel-Frequency Cepstral Coefficients (MFCC) or Perceptual Linear Prediction (PLP). For example, the storage unit 26 stores a search list in which data indicating a baby's cry is associated with a priority of "high".

[0173] In another embodiment, the control unit 27 as shown in FIG. 2 detects a voice whose characteristics match those of a preset voice as a voice that satisfies a set condition.

[0174] In the same or similar manner as in the above-described embodiment, the voice recognition unit 29 acquires information on the start time of an utterance, information on the end time of an utterance, an utterance ID, and voice sampling data from the section detection unit 28. In another embodiment, the voice recognition unit 29 determines whether or not the voice characteristics in the utterance section match the voice characteristics set in advance by a voice recognition process using a learning model generated by an arbitrary machine learning algorithm.

[0175] When the voice recognition unit 29 determines that the voice features in the speech section match the preset voice features, it outputs a result indicating the match as a voice recognition result, the utterance ID of the speech section, and data indicating the voice features to the event detection unit 30. When the voice recognition unit 29 determines that the voice features in the speech section do not match the preset voice features, it outputs a result indicating a mismatch as a voice recognition result and the utterance ID of the speech section to the event detection unit 30.

[0176] The event detection unit 30 may obtain, from the voice recognition unit 29, a result indicating a match as a voice recognition result, an utterance ID, and data indicating a preset voice feature. When the event detection unit 30 obtains a result indicating a match, it detects a voice whose features match the preset voice feature as an event. When the event detection unit 30 detects an event, it obtains the utterance ID obtained from the voice recognition unit 29 as an event ID. Furthermore, the event detection unit 30 refers to the search list and obtains a priority corresponding to the data indicating the voice feature obtained from the voice recognition unit 29. The event detection unit 30 executes a notification process according to the obtained priority in the same or similar manner as in the above-mentioned embodiment.

[0177] The event detection unit 30 may obtain a result indicating a mismatch as a voice recognition result and an utterance ID from the voice recognition unit 29. When the event detection unit 30 obtains a result indicating a mismatch, the event detection unit 30 obtains the utterance ID obtained from the voice recognition unit 29 as a clear event ID. The event detection unit 30 outputs the clear event ID to the utterance accumulation unit 32.

[0178] The processing of the voice processing device 20 according to the other embodiments is not limited to the above-mentioned processing. As another example, the control unit 27 may construct a classifier capable of classifying a plurality of types of voice. Furthermore, the control unit 27 may determine which priority the voice collected by the sound collector 110 corresponds to based on the result of inputting the voice data collected by the sound collector 110 to the constructed classifier.

[0179] Other effects and configurations of the voice processing system 101 according to the other embodiment are the same as or similar to those of the voice processing system 1 shown in FIG.

[0180] Although the present disclosure has been described based on the drawings and examples, it should be noted that a person skilled in the art can easily make various modifications or corrections based on the present disclosure. Therefore, it should be noted that these modifications or corrections are included in the scope of the present disclosure. For example, the functions, etc. included in each functional unit can be rearranged so as not to be logically inconsistent. Multiple functional units, etc. may be combined into one or divided. Each embodiment of the present disclosure described above is not limited to being implemented faithfully to each of the embodiments described, and may be implemented by combining each feature as appropriate or omitting a part. In other words, the contents of the present disclosure can be modified and corrected in various ways by a person skilled in the art based on the present disclosure. Therefore, these modifications and corrections are included in the scope of the present disclosure. For example, in each embodiment, each functional unit, each means, each step, etc. can be added to other embodiments so as not to be logically inconsistent, or replaced with each functional unit, each means, each step, etc. of other embodiments. In addition, in each embodiment, multiple functional units, each means, each step, etc. can be combined into one or divided. Furthermore, each of the above-described embodiments of the present disclosure is not limited to being implemented faithfully according to each of the described embodiments, but may be implemented by combining each feature or omitting some features as appropriate.

[0181] For example, the control unit 27 of the voice processing device 20 may detect a plurality of voices each satisfying different setting conditions from one speech section. In this case, the control unit 27 may notify the user that a voice satisfying the setting conditions has been detected according to each of a plurality of notification conditions set for each of the different setting conditions. Alternatively, the control unit 27 may notify the user that a voice satisfying the setting conditions has been detected according to a part of a plurality of notification conditions set for each of the different setting conditions. Some of the plurality of notification conditions may be notification conditions that satisfy a selection condition. The selection condition is a condition selected in advance from among a first condition, a second condition, a third condition, and a fourth condition based on a user operation or the like. Alternatively, some of the plurality of notification conditions may be notification conditions included in the Nth (N is an integer of 1 or more) notification conditions counting from the highest priority of notifying the user determined by the plurality of notification conditions. The Nth may be set in advance based on a user operation or the like. For example, the control unit 27 may detect a plurality of different search words from one speech section. In this case, the control unit 27 may execute a process according to each of the multiple priorities set for each of the different search words. Alternatively, the control unit 27 may execute a process according to a part of the multiple priorities set for each of the different search words. The part of the multiple priorities may be, for example, the priorities included in the top N priorities counting from the highest priority.

[0182] For example, the control unit 27 of the voice processing device 20 may detect voice satisfying the same set condition multiple times from one speech section. In this case, the control unit 27 may execute the process of notifying the user according to the notification condition only once in one speech section, or may execute the process as many times as the voice satisfying the set condition is detected. For example, the control unit 27 may detect the same search word multiple times from one speech section. In this case, the control unit 27 may execute the process according to the priority only once in one speech section, or may execute the process as many times as the search word is detected.

[0183] For example, the section detection unit 28 as shown in FIG. 2 may stop detecting a speech section while the replay flag is set to True.

[0184] For example, in the voice processing system 101 as shown in Fig. 14, the voice characteristics preset as the setting conditions are described as being the characteristics of a baby's cry. However, the voice characteristics preset as the setting conditions are not limited to the characteristics of a baby's cry. Any voice characteristics may be set as the setting conditions depending on the usage status of the voice processing system 101. As another example, the characteristics of a boss's voice, an intercom ringtone, or a telephone ringtone may be set as the setting conditions.

[0185] For example, in the above-described embodiment, the priority is described as being set in three stages including "high", "medium" and "low". However, the priority is not limited to being set in three stages. The priority may be set in a plurality of stages. For example, the priority may be set in two stages or four or more stages.

[0186] For example, in the above-described embodiment, the sound collector 10 and the sound processing device 20 have been described as separate devices. However, the sound collector 10 and the sound processing device 20 may be configured as one device. An example of this will be described with reference to FIG. 15. The sound processing system 201 shown in FIG. 15 includes a sound collector 210. The sound collector 210 is an earphone. The sound collector 210 is configured to execute the processing of the sound processing device 20. In other words, the sound collector 210 as an earphone is the sound processing device of the present disclosure. The sound collector 210 includes a microphone 11, a speaker 12, a communication unit 13, a storage unit 14, and a control unit 15 as shown in FIG. 2. The control unit 15 of the sound collector 210 includes components corresponding to the control unit 27 of the sound processing device 20. The storage unit 14 of the sound collector 210 stores a notification list. The sound collector 210 may use another terminal device such as a user's smartphone to perform screen display and light emission as a notification means as shown in Fig. 5. For example, the control unit 15 of the sound collector 210 transmits the notification list in the storage unit 14 to the user's smartphone or the like via the communication unit 13, and displays it on the user's smartphone or the like.

[0187] For example, in the above-described embodiment, the voice processing device 20 has been described as executing the voice recognition process. However, an external device other than the voice processing device 20 may execute the voice recognition process. The control unit 27 of the voice processing device 20 may acquire the result of the voice recognition process executed by the external device. The external device may be, for example, a dedicated computer configured to function as a server, a general-purpose personal computer, or a cloud computing system. In this case, the communication unit 13 of the sound collector 10 may be configured to further include at least one communication module that can be connected to any network including a mobile communication network and the Internet, in the same or similar manner as the communication unit 21. In the sound collector 10, the control unit 15 may transmit the voice sampling data to the external device via the network by the communication unit 13. When the external device receives the voice sampling data from the sound collector 10 via the network, it may execute the voice recognition process. The external device may transmit the result of the voice recognition process to the voice processing device 20 via the network. In the voice processing device 20, the control unit 27 may acquire the result of the voice recognition process by receiving it from the external device via the network by the communication unit 21.

[0188] For example, in the above-mentioned embodiment, the voice processing device 20 has been described as a terminal device. However, the voice processing device 20 is not limited to a terminal device. As another example, the voice processing device 20 may be a dedicated computer configured to function as a server, a general-purpose personal computer, a cloud computing system, or the like. In this case, the communication unit 13 of the sound collector 10 may be configured to further include at least one communication module that can be connected to any network including a mobile communication network and the Internet, in the same or similar manner as the communication unit 21. The sound collector 10 and the voice processing device 20 may communicate with each other via a network.

[0189] For example, an embodiment is also possible in which a general-purpose computer functions as the voice processing device 20 according to the above-described embodiment. Specifically, a program describing the processing contents for realizing each function of the voice processing device 20 according to the above-described embodiment is stored in the memory of the general-purpose computer, and the program is read and executed by a processor. Therefore, the configuration according to the above-described embodiment can also be realized as a program executable by a processor or a non-transitory computer-readable medium that stores the program. [Explanation of symbols]

[0190] 1,101,201 Voice Processing System 10,110,210 Sound collector 11. Mike 12 Speaker 13. Communications Department 14 Storage section 15 Control section 16 Voice Acquisition Unit 17 Audio playback section 18 Storage section 20 Audio processing device 21 Communications Department 22 Input section 23 Display section 24 Vibration section 25 Light emitting part 26 Memory section 27 Control Unit 28 Section detection section 29 Voice Recognition Unit 30 Event detection section 31 Speech notification unit 32 Speech storage unit 33 Data Buffer 34 Ring Buffer 35 Audio Modulation Section 40 Main Screen 41,42,43,44 area 50 Setting screen 51,52,53,54,55,56 area 60 Notification screen 61,62,63,64 area

Claims

1. The storage unit stores sounds around the user collected by the microphone; When the control unit detects an utterance satisfying a condition in the ambient sound, the control unit outputs the ambient sound from a start point of an utterance section including the utterance according to a notification condition. Sound processing methods.

2. The speech section is a section in which sound continues without a set time gap. The sound processing method according to claim 1 .

3. The control unit outputs the ambient sound up to an end point of the speech section. The sound processing method according to claim 2 .

4. The control unit outputs the chunk of utterance. The sound processing method according to claim 1 .

5. A storage unit that stores sounds collected by a microphone around the user; a control unit that, when an utterance satisfying a condition is detected in the ambient sound, outputs the ambient sound from a start point of an utterance section including the utterance in accordance with a notification condition, Sound processing device.

6. storing sounds around the user collected by a microphone in a storage unit; When an utterance satisfying a condition is detected in the ambient sound, outputting the ambient sound from a start point of an utterance section including the utterance in accordance with a notification condition. causing the control unit to execute Sound processing program.

7. The storage unit stores sounds around the user collected by the microphone; When the control unit detects a sound that satisfies a condition in the surrounding sounds, the control unit outputs the sound that satisfies the condition from a start point of the surrounding sounds according to a notification condition. Sound processing methods.

Images