Voice recognition device, voice recognition system, voice recognition method and program

Abstract

To provide a voice recognition device which more satisfactorily maintains accuracy of voice recognition in an utterance section including voices of a plurality of speakers.SOLUTION: A voice recognition device includes: an utterance section detection unit for detecting an utterance section of voice data including voices of a plurality of speakers; a speaker information application unit for applying speaker information for identifying each speaker from a plurality of speakers to voice data so as to identify timing of speaker changes in the detected utterance section; a voice data division unit for dividing voice data for each speaker information at timing when speaker information is changed when speaker information applied within the utterance section is changed; and a voice recognition unit for recognizing voice for each voice data divided at every speaker information.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a voice recognition device, a voice recognition system, a voice recognition method, and a program. [Background technology]

[0002] In order to improve speech recognition performance, a technique has been known in the art that detects sections where speech exists (speech sections) from speech data to be recognized using voice activity detection and then performs speech recognition on the detected speech sections.

[0003] For example, Patent Document 1 discloses a technology for determining the speech section of each speaker from audio data containing a mixture of the voices of multiple speakers, extracting speaker features that can identify the speaker, and performing voice recognition of the speech section using the extracted speaker features. Summary of the Invention [Problem to be solved by the invention]

[0004] In speech activity detection, speech activities including the voices of multiple speakers may be detected as a single speech activity, for example, when one speaker finishes speaking and another speaker starts speaking at the same time.

[0005] However, conventional speech recognition has a problem in that the accuracy of speech recognition decreases when a detected speech section contains voices from multiple speakers. Patent Document 1 does not describe how to maintain good accuracy of speech recognition even when a detected speech section contains voices from multiple speakers.

[0006] An object of one embodiment of the present invention is to provide a speech recognition device that maintains better speech recognition accuracy for speech segments that include speech from multiple speakers. [Means for solving the problem]

[0007] A speech recognition device according to one embodiment of the present invention includes a speech interval detection unit that detects speech intervals in speech data containing speech from multiple speakers; a speaker information assignment unit that assigns speaker information identifying each speaker from the multiple speakers to the speech data in order to identify timings at which speakers switch within the detected speech interval; a speech data division unit that, when the assigned speaker information switches within the speech interval, divides the speech data for each piece of speaker information at the timing at which the speaker information switches; and a speech recognition unit that performs speech recognition on each piece of the speech data divided for each piece of speaker information. [Effects of the Invention]

[0008] According to the present invention, it is possible to maintain a high level of accuracy in speech recognition for speech segments containing speech from multiple speakers. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of a configuration of a voice recognition system according to a first embodiment. [Figure 2] FIG. 1 is a diagram illustrating a hardware configuration of an example of a smartphone. [Figure 3] FIG. 2 is a diagram illustrating a hardware configuration of an example of a computer. [Figure 4] 1 is a functional configuration diagram of an example of a voice recognition system according to a first embodiment. [Figure 5] 1 is a flowchart illustrating an example of the procedure of a voice recognition method performed by a general voice recognition system. [Figure 6] 3 is a flowchart illustrating an example of the procedure of a voice recognition method performed by the voice recognition system according to the first embodiment. [Figure 7] FIG. 10 is an explanatory diagram of an example of the speaker diarization processing in step S24. [Figure 8] FIG. 3 is an explanatory diagram of an example of processing for extracting speaker features according to the first embodiment. [Figure 9] FIG. 2 is a diagram illustrating an example of a user interface of the speech recognition system according to the first embodiment. [Figure 10]FIG. 2 is a diagram illustrating an example of a user interface of the speech recognition system according to the first embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of a configuration of a voice recognition system according to a second embodiment. [Figure 12] FIG. 10 is a functional configuration diagram of an example of a voice recognition device according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0011] [First embodiment] The first embodiment is an example of a speech recognition system used to create meeting minutes. In the first embodiment, speech recognition is performed on audio data in which the voices of multiple participants (speakers) who have participated in a meeting are recorded, and the results of the speech recognition are output to support the creation of meeting minutes.

[0012] <System configuration> 1 is a configuration diagram of an example of a speech recognition system 1 according to the first embodiment. In the speech recognition system 1 according to the first embodiment, a speech recognition device 10, an online conference control device 12, and a conference communication terminal 14 are communicably connected via a network 18. The network 18 is the Internet, a LAN (Local Area Network), or the like.

[0013] The conference communication terminal 14 is used by users such as participants in an online conference, the organizer, and the person creating the conference minutes. The conference communication terminal 14 is a communication terminal such as a PC (Personal Computer), a smartphone, a tablet terminal, a mobile phone, a wearable device, or an HMD (Head Mounted Display). By connecting the conference communication terminal 14 to the online conference control device 12, the user can participate in an online conference that uses an online conference service provided by the online conference control device 12.

[0014] Voice data during an online conference is recorded in the voice recognition system 1. The voice data may be recorded by the online conference control device 12 or by the conference communication terminal 14. The voice recognition system 1 may also acquire and use voice data recorded by a recording device such as an IC recorder.

[0015] The online conference control device 12 provides an online conference service to users who use the conference communication terminals 14. For example, the online conference control device 12 performs control so that the voices, images, videos, etc. of multiple speakers (users) participating in the same online conference are shared among the conference communication terminals 14 of the multiple speakers (users) participating in the same online conference, thereby realizing the online conference.

[0016] The voice recognition device 10 receives from the user the specification of the voice data to be recognized, performs the voice recognition as described below, and outputs the voice recognition result. The voice data to be recognized may be specified by uploading the voice data to the voice recognition device 10, or by specifying a storage location for the voice data.

[0017] The speech recognition device 10 may function as a web server and receive, from the user, a specification of speech data to be recognized by displaying a web page for receiving the specification of speech data on the conference communication terminal 14. Here, the web page functions as an example of a receiving unit for receiving the specification of speech data.

[0018] The speech recognition device 10 is, for example, a PC or a workstation, etc. The speech recognition device 10 may also be an output device such as a printer, a scanner, a facsimile machine, a multifunction peripheral, a projector, a display device with an electronic whiteboard function, a digital signage, a head-up display device, an industrial machine, an imaging device, a sound collection device, a medical device, a network home appliance, a connected car, or a game console, etc.

[0019] The configuration of the speech recognition system 1 shown in FIG. 1 is an example. The configuration of the speech recognition system 1 varies depending on the application, purpose, etc. For example, the speech recognition device 10 and the online conference control device 12 may be configured as one or more computers. The speech recognition device 10 and the online conference control device 12 may be realized as a cloud computing service. The speech recognition device 10 and the online conference control device 12 may be integrated into one device. Furthermore, the function of the speech recognition device 10 may be provided as one function of the online conference control device 12.

[0020] <Hardware configuration> Smartphone The conference communication terminal 14 may be realized by using a smartphone 400 having the hardware configuration shown in Fig. 2. Fig. 2 is a diagram showing an example of the hardware configuration of the smartphone 400.

[0021] The smartphone 400 includes a CPU (Central Processing Unit) 401, a ROM (Read Only Memory) 402, a RAM (Random Access Memory) 403, an EEPROM 404, a CMOS (Complementary Metal Oxide Semiconductor) sensor 405, an image sensor I / F 406, an acceleration / azimuth sensor 407, a media I / F 409, and a GPS receiver 411.

[0022] The CPU 401 controls the overall operation of the smartphone 400. The ROM 402 stores the CPU 401 and programs used to drive the CPU 401, such as the IPL. The RAM 403 is used as a work area for the CPU 401. The EEPROM 404 reads and writes various data, such as smartphone programs, under the control of the CPU 401.

[0023] The CMOS sensor 405 is a type of built-in imaging means that captures an image of a subject (mainly a self-portrait) under the control of the CPU 401 to obtain image data. Note that instead of a CMOS sensor, an imaging means such as a CCD (Charge Coupled Device) sensor may also be used. The imaging element I / F 406 is a circuit that controls the driving of the CMOS sensor 405. The acceleration / azimuth sensor 407 is one of various sensors, such as an electronic magnetic compass that detects geomagnetism, a gyrocompass, and an acceleration sensor. The media I / F 409 controls the reading and writing (storage) of data from and to a recording medium 408, such as a flash memory. The GPS receiver 411 receives GPS signals from GPS satellites.

[0024] The smartphone 400 also includes a long-distance communication circuit 412, a CMOS sensor 413, an image sensor I / F 414, a microphone 415, a speaker 416, an audio input / output I / F 417, a display 418, an external device connection I / F (Interface) 419, a short-distance communication circuit 420, an antenna 420a for the short-distance communication circuit 420, and a touch panel 421. The long-distance communication circuit 412 is a circuit that communicates with other devices via the network 18. The CMOS sensor 413 is a type of built-in imaging means that captures an image of a subject under the control of the CPU 401 to obtain image data. The image sensor I / F 414 is a circuit that controls the driving of the CMOS sensor 413. The microphone 415 is a built-in circuit that converts sound into an electrical signal. The speaker 416 is a built-in circuit that converts the electrical signal into physical vibrations to produce sound such as music or voice.

[0025] The audio input / output I / F 417 is a circuit that processes input and output of audio signals between the microphone 415 and the speaker 416 under the control of the CPU 401. The display 418 is a type of display means such as a liquid crystal or organic EL (Electro Luminescence) that displays an image of a subject, various icons, etc.

[0026] The external device connection I / F 419 is an interface for connecting various external devices. The short-range communication circuit 420 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The touch panel 421 is a type of input means that allows a user to operate the smartphone 400 by pressing the display 418. The smartphone 400 includes a bus line 410. The bus line 410 is an address bus, a data bus, etc. for electrically connecting the components such as the CPU 401 shown in FIG. 2.

[0027] The hardware configuration shown in FIG. 2 is an example, and it is not necessary to include all of the components shown in FIG. 2, or it may include components other than those shown in FIG.

[0028] "computer" The speech recognition device 10 and the online conference control device 12 shown in Fig. 1 may be realized by using a computer 500 having a hardware configuration shown in Fig. 3. Also, the conference communication terminal 14 may be realized by using a computer 500 having a hardware configuration shown in Fig. 3. Fig. 3 is a hardware configuration diagram of an example of the computer 500.

[0029] The computer 500 includes a CPU 501, a ROM 502, a RAM 503, a HD 504, an HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F 508, a network I / F 509, a data bus 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I / F 516.

[0030] The CPU 501 controls the overall operation of the computer 500 in accordance with a program. The ROM 502 stores programs used to drive the CPU 501, such as the IPL. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading and writing of various data from and to the HD 504 in accordance with the control of the CPU 501.

[0031] The display 506 displays various types of information such as a cursor, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various types of external devices. In this case, the external device is, for example, a USB (Universal Serial Bus) memory. The network I / F 509 is an interface for data communication using the network 18. The data bus 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501.

[0032] The keyboard 511 is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 514 controls reading and writing of various data from a DVD-RW 513, which is an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R, etc. The media I / F 516 controls reading and writing (storing) of data from a recording medium 515, such as a flash memory.

[0033] The hardware configuration shown in FIG. 3 is an example, and it is not necessary to include all of the components shown in FIG. 3, or it may include components other than those shown in FIG.

[0034] <Functional configuration> Fig. 4 is a functional configuration diagram of an example of a speech recognition system 1 according to the first embodiment. The functional configuration diagram of Fig. 4 omits configurations that are not necessary for explaining the first embodiment. The speech recognition device 10, the online conference control device 12, and the conference communication terminal 14 realize the functional configuration of Fig. 4 by, for example, the smartphone 400 shown in Fig. 2 or the computer 500 shown in Fig. 3 executing an OS (Operating System) and a program.

[0035] 4 includes a transmitting / receiving unit 30, a voice data acquiring unit 32, a speech segment detecting unit 34, a speaker information adding unit 36, a voice data dividing unit 38, a voice recognizing unit 40, a storing unit 42, a result output unit 48, and an edit receiving unit 49. The online conference control device 12 includes a transmitting / receiving unit 50, an online conference control unit 52, a recording processing unit 54, and a storing unit 56. The conference communication terminal 14 includes a receiving unit 70, a transmitting / receiving unit 72, an online conference processing unit 74, a recording processing unit 76, and a storing unit 78.

[0036] The reception unit 70 of the conference communication terminal 14 receives various operations from the user. For example, the reception unit 70 receives various operations from the user, such as an operation to join an online conference, an operation to start recording an online conference, an operation to start recording an online conference, or an operation to specify audio data for speech recognition.

[0037] The transmitting / receiving unit 72 transmits and receives data between the speech recognition device 10 and the online conference control device 12. The online conference processing unit 74 transmits and receives data to and from the online conference control device 12, and performs processing to share the voices, images, videos, etc. of multiple speakers (users) participating in the same online conference, thereby allowing the user operating the conference communication terminal 14 to participate in the online conference.

[0038] The recording processing unit 76 records audio data by recording during the online conference. The storage unit 78 stores the audio data recorded by the recording processing unit 76 during the online conference. The recording processing unit 76 may also record video data by recording during the online conference. Video data is an example of data from which audio data can be extracted. The storage unit 78 stores the video data recorded by the recording processing unit 76 during the online conference.

[0039] The transmitting / receiving unit 50 of the online conference control device 12 transmits and receives data between the speech recognition device 10 and the conference communication terminal 14. The online conference control unit 52 transmits and receives data to and from the conference communication terminal 14 of a user participating in the same online conference, and performs processing to share the voices, images, videos, etc. of multiple speakers (users) participating in the same online conference, thereby allowing the user operating the conference communication terminal 14 to participate in the online conference.

[0040] The recording processing unit 54 records audio data by recording during the online conference. The storage unit 56 stores the audio data recorded by the recording processing unit 54 during the online conference. The recording processing unit 54 may record video data by recording during the online conference. The storage unit 56 may store the video data recorded by the recording processing unit 54 during the online conference.

[0041] The transmitter / receiver 30 of the voice recognition device 10 transmits and receives data between the online conference control device 12 and the conference communication terminal 14. The voice data acquisition unit 32 acquires voice data to be recognized as specified by the user. The voice data acquisition unit 32 acquires the voice data specified by the user from, for example, the storage unit 42. The voice data acquisition unit 32 may acquire the voice data from, for example, the storage unit 56 of the online conference control device 12. The voice data acquisition unit 32 may also acquire the voice data from the storage unit 78 of the conference communication terminal 14.

[0042] The speech interval detection unit 34 detects speech intervals in the audio data including the voices of multiple speakers. Specifically, the speech interval detection unit 34 detects intervals where speech exists (speech intervals) from the audio data including the voices of multiple speakers acquired by the audio data acquisition unit 32 by speech interval detection (VAD: Voice Activity Detection). The speech interval detection technology is an existing technology, and therefore a description thereof will be omitted.

[0043] The speaker information assigning unit 36 ​​assigns speaker information for identifying each speaker from among multiple speakers to the audio data in order to identify the timing at which the speaker switches within the speech section detected by the speech section detection unit 34. For example, the speaker information assigning unit 36 ​​can assign speaker information to the audio data in order to identify the timing at which the speaker switches within the speech section by using a speaker diarization processing technique.

[0044] Speaker diarization is a technique for estimating who is speaking what part of speech data containing the voices of multiple speakers. The speaker information to be assigned is information that identifies the speaker. A speaker label is an example of speaker information that identifies a speaker.

[0045] The speaker information assignment unit 36 ​​has a speaker feature extraction unit 44 and a speaker label determination unit 46. The speaker feature extraction unit 44 divides the speech period detected by the speech period detection unit 34 into predetermined units (for example, units of predetermined time intervals) and extracts speaker features for each predetermined unit. The extraction of speaker features will be described later. The speaker label determination unit 46 determines speaker information for each predetermined unit based on the extracted speaker features representing the characteristics of the speaker, and assigns the speaker information (speaker label) of the determined speaker to the audio data. An example in which the speaker information is a speaker label will be described below. The assignment of speaker labels will be described in detail later.

[0046] When the assigned speaker information changes within a speech section, the audio data division unit 38 divides the audio data for each piece of speaker information at the timing when the speaker information changes. The timing when the speaker information changes within a speech section can be determined by the speaker label assigned by the speaker label determination unit 46, as will be described later. Even if the speech section detected by the speech section detection unit 34 includes voices of multiple speakers, the audio data division unit 38 divides the audio data at the timing when the speaker changes within the speech section. Therefore, the audio data divided by the audio data division unit 38 includes the speech of one speaker.

[0047] The speech recognition unit 40 performs speech recognition on each piece of speech data divided by the speech data division unit 38 for each piece of speaker information. The speech recognition unit 40 performs speech recognition on speech data that the speech data division unit 38 has divided so as to include the speech of one speaker and that has consistent acoustic features (voice quality, speaking style, etc.), thereby maintaining better speech recognition accuracy even for speech segments that include speech from multiple speakers. The storage unit 42 stores speech data acquired from the online conference control device 12 or the conference communication terminal 14. The storage unit 42 also stores the results of speech recognition of the speech data. The result output unit 48 outputs the results of speech recognition and speaker information, for example, by displaying them on a screen. The edit receiving unit 49 receives an operation from the user to edit the output speech recognition results and speaker information.

[0048] <Processing> Fig. 5 is a flowchart showing an example of the procedure of a speech recognition method performed by a general speech recognition system. Fig. 6 is a flowchart showing an example of the procedure of a speech recognition method performed by the speech recognition system 1 according to the first embodiment. The flowchart in Fig. 5 shows the speech recognition method of a general speech recognition device that is the target of improvement by the speech recognition system 1 according to the first embodiment.

[0049] In step S10, a general speech recognition device acquires speech data to be recognized, as specified by a user. In step S12, the general speech recognition device detects speech segments from the acquired speech data using voice activity detection (VAD). In step S14, the general speech recognition device divides the speech data to be recognized into the speech segments detected in step S12. In step S16, the general speech recognition device performs speech recognition on each of the pieces of speech data divided in step S14 (divided speech data).

[0050] In the speech recognition method of a general speech recognition device shown in the flowchart of Figure 5, speech recognition processing is performed after dividing the speech into speech segments, which makes it possible to prevent errors such as transcribing non-speech segments during speech recognition processing.

[0051] In recent years, various speech recognition modules (speech recognizers) that utilize deep learning to achieve advanced speech recognition have come to be used for the speech recognition performed in step S16. For example, the speech recognition in step S16 can utilize technologies such as Whisper (A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, "Robust speech recognition via large-scale weak supervision," Tech. Rep., OpenAI, 2022.) and Conformer (A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang. Conformer: Convolution-augmented transformer for speech recognition. arXiv, 2020.).

[0052] However, when a single speech segment contains the voices of multiple speakers, typical speech recognition devices are unable to recognize speech data with consistent acoustic characteristics (such as voice quality and speaking style), which raises concerns that this may have a negative impact on the results of speech recognition.

[0053] For example, the speech interval detected by the speech interval detection in step S12 may include speech from multiple speakers, such as when one speaker finishes speaking and another speaker starts speaking at the same time. In particular, when speech recognition in step S16 is performed taking into account the relationship between units of a predetermined time interval (frames) within the input speech interval, switching speakers within the same speech interval can significantly change the acoustic characteristics, potentially adversely affecting the recognition results. Furthermore, when a single speech interval contains speech from two speakers, the speech recognition in step S16 may only recognize the speech of one speaker.

[0054] The speech recognition system 1 according to the first embodiment solves the above-mentioned problems of the speech recognition procedure performed by the general speech recognition system shown in Fig. 5 by implementing the speech recognition method procedure shown in Fig. 6. The speech recognition method procedure shown in Fig. 6 includes a speaker diarization process in step S24 between the speech activity detection process in step S22 and the speech data division process in step S26. Note that a description of processes similar to those in the flowchart of Fig. 5 will be omitted where appropriate.

[0055] In step S20, the speech data acquisition unit 32 of the speech recognition device 10 according to the first embodiment acquires speech data for speech recognition designated by the user. In step S22, the speech segment detection unit 34 of the speech recognition device 10 detects speech segments from the acquired speech data.

[0056] In step S24, the speaker information assigning unit 36 ​​assigns one or more speaker labels to the speech section detected in step S22. Here, the processing of step S24 performed by the speaker information assigning unit 36 ​​will be described with reference to Fig. 7. Fig. 7 is an explanatory diagram of an example of the speaker diarization processing of step S24. The speaker diarization processing of step S24 is realized by processing of the speaker information assigning unit 36 ​​that combines the processing of the speaker feature extracting unit 44 and the processing of the speaker label determining unit 46.

[0057] As a method for extracting speaker features, well-known methods such as i-vector (N. Dehak, P. Kenny, R. Dehak, P. Dumouchel, P. Ouellet, Front-end factor analysis for speaker verification, IEEE Transactions on Audio, Speech, and Language Processing 19 (2011)) and x-vector (D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, S. Khudanpur, X-vectors: Robust DNN embeddings for speaker recognition, in: Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, 2018, pp. 5329-5333) can be applied.

[0058] A widely known method for determining speaker labels is to classify the extracted speaker features into multiple groups by clustering, and then assign each group to the same speaker. Examples of clustering methods include k-means clustering (J. MacQueen, et al., Some methods for classification and analysis of multivariate observations, in: Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, volume 14, Oakland, CA, USA, 1967, pp. 281-297.) and spectral clustering (U. Luxburg, "A Tutorial on Spectral Clustering," Statistics and Computing, pp. 395-416, 2007.).

[0059] The speaker information assigning unit 36 ​​may be configured to receive from the user a specification of the number of speakers included in the speech data. The speaker information assigning unit 36 ​​clusters speaker features in accordance with the specification of the number of speakers included in the speech data received from the user. Specifically, the speaker label determining unit 46 constituting the speaker information assigning unit 36 ​​determines the number of groups in accordance with the specification of the number of speakers included in the speech data received from the user, and assigns speaker information to the speech data based on the determined number of groups. This allows the speech recognition device 10 to improve the accuracy of clustering. The speaker label determining unit 46 may also be configured to receive from the user a specification of a range of the number of speakers included in the speech data. The speaker label determining unit 46 clusters speaker features in accordance with the specification of the range of the number of speakers included in the speech data received from the user. Specifically, the speaker label determining unit 46 determines the number of groups in accordance with the specification of the range of the number of speakers included in the speech data received from the user, and assigns speaker information to the speech data based on the determined number of groups. This allows the speech recognition device 10 to improve the accuracy of clustering. For example, specifying a range of the number of speakers included in the voice data is effective for voice data of a meeting (such as a regular meeting) where the number of participants is somewhat fixed.

[0060] The speaker information assigning unit 36 ​​is not limited to the configuration of the first embodiment as long as it can assign a speaker label to the speech period detected by the speech period detecting unit 34.

[0061] 7 shows an example of the speech data of the speech section detected in step S22. The speaker feature extraction unit 44 of the speaker information attachment unit 36 ​​divides the speech section from the start to the end into units (frames) of a predetermined time interval, and extracts a speaker feature X i (i=1 to the number of frames) are extracted. In Fig. 7, a certain speech section is divided into eight frames, and speaker features X1 to X8 corresponding to the eight frames are extracted.

[0062] The speaker label determination unit 46 uses clustering to classify the entire set of speaker features extracted from all speech segments, including the speaker features X1 to X8 extracted as shown in FIG. 7, into multiple groups. Clustering is a technique for dividing a set of classification targets into subsets called clusters. Using this technique, the speaker label determination unit 46, when given a large amount of data, classifies the data into groups by classifying them as follows: "Objects in the same cluster are similar to each other" and "Objects in different clusters are dissimilar." In other words, the speaker label determination unit 46 divides the set of extracted speaker features into multiple groups, associates speaker information with each group, determines speaker information from the speaker features, and assigns the determined speaker information to the speech data. Specifically, the speaker label determination unit 46 determines frames of speaker features classified into the same group as frames spoken by the same speaker and assigns a speaker label to each frame. Hereinafter, the frames in FIG. 7 from which speaker features X1 to X8 are extracted are referred to as frames 1 to 8. In Figure 7, two speakers, "A" and "B," are shown as an example, but even when speech is given by three or more speakers, the speaker label determination unit 46 can group the speakers by performing the above-mentioned clustering on the entire set of speaker features extracted from all the speeches.

[0063] In FIG. 7, the speaker label "A" is assigned to the first to sixth frames. Also, in FIG. 7, the speaker label "B" is assigned to the seventh and eighth frames. Different speaker labels assigned to frames indicate that different speakers are speaking in those frames. For example, the example in FIG. 7 indicates that the speaker changed between the sixth and seventh frames within the speech section.

[0064] The speaker label "A" assigned to the first to sixth frames and the speaker label "B" assigned to the seventh and eighth frames are examples of speaker information assigned to identify the timing at which the speaker switches within a speech section.

[0065] The speech recognition device 10 according to the first embodiment determines that one speaker has spoken in an utterance section when the same speaker label is assigned to that utterance section, whereas the speech recognition device 10 determines that multiple speakers have spoken in that utterance section when different speaker labels are assigned to that utterance section.

[0066] 6, the speech data division unit 38 divides the speech data to be recognized into speech intervals detected by the speech interval detection unit 34 and timings at which the speaker label changes within the speech interval. The timings at which the speaker label changes within a speech interval can be determined from the position of the frame within the speech interval at which the speaker label assigned by the speaker label determination unit 46 changes (switches).

[0067] The timing at which speaker information changes within a certain speech section is, for example, the end point of consecutive frames to which the same speaker label is assigned. In the example of Figure 7, the speech section is divided between the sixth and seventh frames within the speech section. In this way, the speech data division unit 38 can further divide the speech section based on consecutive frames with the same speaker label, thereby dividing the speech data so that the speech data includes voices from the same speaker.

[0068] Also, in step S24, if the speaker label determination unit 46 assigns speaker labels such as [(start time), (end time)] = [5, 10] to a speech section such as [(start time), (end time)] = [5, 10], the audio data division unit 38 further divides the speech section [5, 10] into [5, 8] and [8, 10].

[0069] In step S28, the speech recognition unit 40 performs speech recognition on each piece of speech data divided in step S26 (post-division speech data). The speech data divided in step S26 is speech data with consistent acoustic characteristics, which has been divided by the speech data division unit 38 so as to include speech from one speaker. Therefore, even if the speech data includes a speech section including speech from multiple speakers, the speech recognition unit 40 can recognize speech data with consistent acoustic characteristics, thereby maintaining better speech recognition accuracy.

[0070] Moreover, the processing for extracting speaker features described with reference to Fig. 7 may be performed, for example, as shown in Fig. 8. Fig. 8 is an explanatory diagram of an example of processing for extracting speaker features according to the first embodiment.

[0071] 8 shows an example of the audio data of the speech section detected in step S22. The speaker feature extraction unit 44 of the speaker information attachment unit 36 ​​divides each speech section from its start to its end into predetermined units (frames) so that each frame overlaps the preceding and succeeding speech sections (frames) in time, as shown in FIG. i Then, the speaker label determining unit 46 determines speaker information for each frame based on the extracted speaker features, and assigns the determined speaker information to the speech data.

[0072] In the example shown in Fig. 8, more speaker features can be obtained while maintaining the frame length required for feature extraction. The example shown in Fig. 8 also has the advantage of being more sensitive to speaker label changes within a speech section. The length of the temporal overlap with the preceding and following frames may be fixed (for example, 50% of the frame length) or may be user-configurable.

[0073] 7, the process of assigning a speaker label to each frame (the process of the speaker label determination unit 46 in the speaker information assignment unit 36) may assign multiple speaker labels to one frame by using information indicating whether each speaker is speaking at each time. When multiple speaker labels are assigned to one frame, the audio data division unit 38 determines the position between frames at which to divide the speech section, taking into account the continuity with the speaker labels assigned to the previous and following frames, etc.

[0074] A user who specifies the voice data to be recognized inputs a command from a prompt screen such as that shown in Fig. 9. Fig. 9 is a diagram illustrating an example of a user interface of the voice recognition system 1 of the first embodiment.

[0075] Figure 9(A) shows an example of a speech recognition request made by entering an API command from a prompt screen. As shown in Figure 9(A), the user can specify request parameters in the speech recognition request. The request parameter "d" specifies the engine name to be used for speech recognition. The request parameter "u" specifies authentication information. The request parameter "a" specifies the audio data to be recognized. The user may also specify the number of speakers included in the audio data or the range of the number of speakers included in the audio data in the request parameters for use in the clustering described above.

[0076] Figure 9(B) is an example of a response to the request in Figure 9(A). The results of transcribing the specified voice data (the results of voice recognition) are written in the "..." part of "text". Note that the response in Figure 9(B) is just an example, and may include information such as the start time of the utterance, the end time of the utterance, and the reliability of the voice recognition results in addition to the results of voice recognition.

[0077] A user who specifies speech data to be recognized may use the speech recognition system 1 of the first embodiment from, for example, a web screen as shown in Fig. 10. Fig. 10 is a diagram illustrating an example of a user interface of the speech recognition system 1 of the first embodiment. The web screen shown in Fig. 10 is displayed by the result output unit 48 of the speech recognition device 10. User operations on the web screen shown in Fig. 10 are accepted by the edit accepting unit 49 of the speech recognition device 10.

[0078] Fig. 10 shows an example of a request for speech recognition from a web screen displayed in a browser. For example, a user can select speech data to be recognized from the speech data list 1000 by pressing the execute button in the speech data list 1000 in Fig. 10.

[0079] Furthermore, the user can specify request parameters for the speech recognition request by, for example, pressing a setting button in the request parameter field 1002 in Fig. 10. The request parameter field 1002 is an example of a request parameter field 1002 in which the number of speakers, the range of the number of speakers, and the names of the speakers can be specified as request parameters.

[0080] The request parameter may specify whether to enable or disable speaker diarization processing. Specifically, the speaker label determination unit 46 enables or disables the assignment of speaker information based on the enable or disable specification received from the user. For example, if the speech data to be recognized does not contain multiple speakers, disabling the speaker diarization processing can omit the speaker diarization processing and improve processing speed.

[0081] Furthermore, the speech recognition result field 1004 displays the speech recognition result of the speech data selected by the user from the speech data list 1000. The user may be able to edit the speech recognition result by pressing an edit button in the speech recognition result field 1004. Even if there is an error in the speech recognition result, the user can edit it as desired. Furthermore, the user may be able to edit the unspecified speaker A and the unspecified speaker B to personal names, etc.

[0082] 10 may be changed to the names input by the user. For example, after the user inputs the names of the unspecified speaker A and the unspecified speaker B in the request parameter field 1002, the speech recognition device 10 performs a process of associating the unspecified speaker A and the unspecified speaker B with the names input by the user, using commonly known speaker recognition. In this way, the speech recognition device 10 may change the names of the unspecified speaker A and the unspecified speaker B in the speech recognition result field 1004 to the names input by the user and display them.

[0083] The speech recognition system 1 according to the first embodiment can also support the creation of lecture notes by performing speech recognition on speech data in which the speech of multiple speakers during an online lecture, such as the speech of a teacher or students in the online lecture, is recorded, and outputting the speech recognition results.Furthermore, the speech recognition system 1 according to the first embodiment can also support the creation of a record of the meeting by performing speech recognition on speech data in which the speech of multiple speakers during a meeting, such as the speech of meeting participants, is recorded, and outputting the speech recognition results.

[0084] As described above, the speech recognition system 1 according to this embodiment can maintain high speech recognition accuracy even when the detected speech section contains speech from multiple speakers. Furthermore, the speech recognition system 1 according to this embodiment executes a speaker diarization process (S24), which includes adding speaker information for identifying speakers to speech data, between the speech section detection process (S22), the speech data segmentation process (S26), and the speech recognition process (S28). In other words, the speaker diarization process, including adding speaker information, is executed independently before the speech data segmentation process and speech recognition process are executed. This allows existing speech section detection process, speech data segmentation process, and speech recognition process to be used without modification. This also has the effect of increasing the reuse of design assets of speech recognition devices and reducing design man-hours.

[0085] [Second embodiment] The second embodiment is an example of a voice recognition system used for creating medical examination results, handovers, daily nursing care reports, etc. In the second embodiment, voice recognition is performed on voice data containing recorded voices of multiple speakers, such as the voices of medical staff or patients, the voices of nursing care staff or facility users, and the voice recognition results are output to support the creation of medical examination results.

[0086] The second embodiment is similar to the first embodiment except for some parts, and therefore the description will be omitted as appropriate.

[0087] 11 is a configuration diagram of an example of a voice recognition system 2 according to the second embodiment. In the voice recognition system 2 according to the second embodiment, a voice recognition device 10, a medical examination result management device 100, and a medical field communication terminal 102 are communicably connected via a network 18.

[0088] The medical field communication terminal 102 is used by a user such as a medical field staff member or a medical examination result creator. The medical field communication terminal 102 is a communication terminal such as a PC, a smartphone, a tablet terminal, a mobile phone, or an intercom. By connecting the medical field communication terminal 102 to the medical examination result management device 100, the user can use the medical examination result management service provided by the medical examination result management device 100.

[0089] Voice data during a medical examination is recorded in the voice recognition system 2. The voice data may be recorded by the medical examination result management device 100 or by the medical field communication terminal 102. The voice recognition system 2 may also acquire and use voice data recorded by a recording device such as an IC recorder.

[0090] The medical examination result management device 100 provides a medical examination result management service to a user who uses the medical field communication terminal 102. For example, the medical examination result management device 100 accepts registration of medical examination results from the medical field communication terminal 102, and stores and manages the medical examination results.

[0091] The voice recognition device 10 receives from the user the designation of voice data to be recognized, performs voice recognition as described above, and outputs the results of the voice recognition. The designation of voice data to be recognized may be performed by uploading the voice data to the voice recognition device 10, or by designating a storage location for the voice data. The voice recognition device 10 may receive from the user the designation of voice data to be recognized by displaying a web page on the medical field communication terminal 102 for accepting the designation of voice data.

[0092] The configuration of the voice recognition system 2 shown in FIG. 11 is an example. The configuration of the voice recognition system 2 varies depending on the application, purpose, etc. For example, the voice recognition device 10 and the medical examination result management device 100 may be configured as one or more computers. The voice recognition device 10 and the medical examination result management device 100 may be realized as a cloud computing service. Furthermore, the voice recognition device 10 and the medical examination result management device 100 may be integrated into one configuration. The function of the voice recognition device 10 may be provided as one function of the medical examination result management device 100.

[0093] The voice recognition system 2 according to the second embodiment performs voice recognition on voice data containing recorded voices of multiple speakers, such as the voices of medical staff or patients, the voices of care staff or facility users, and can also assist in creating care results by outputting the results of the voice recognition.

[0094] [Third embodiment] In the third embodiment, the speech recognition method according to the first or second embodiment is realized by a speech recognition device 200. Fig. 12 is a functional configuration diagram of an example of the speech recognition device 200 according to the third embodiment.

[0095] The speech recognition device 200 of the third embodiment realizes the functional configuration of FIG. 12 by, for example, the smartphone 400 shown in FIG. 2 or the computer 500 shown in FIG. 3 executing an OS and a program.

[0096] The speech recognition device 200 shown in FIG. 12 includes a reception unit 202, a speech data acquisition unit 204, a speech period detection unit 206, a speaker information assignment unit 208, a speech data division unit 210, a speech recognition unit 212, a recording processing unit 214, a storage unit 216, a result output unit 218, and an edit reception unit 219.

[0097] The accepting unit 202 accepts various operations from the user, such as an operation to start video recording, an operation to start audio recording, and an operation to specify audio data for speech recognition.

[0098] The voice data acquisition unit 204 acquires voice data to be subjected to voice recognition specified by the user, for example, from the storage unit 216. The speech interval detection unit 206 detects a speech interval from the voice data acquired by the voice data acquisition unit 204 by speech interval detection.

[0099] The speaker information assigning unit 208 assigns speaker information identifying each speaker from multiple speakers to the audio data in order to identify the timing of speaker switching within the speech period detected by the speech period detection unit 206. The speaker information assigning unit 208 has a speaker feature extraction unit 220 and a speaker label determination unit 222. The speaker feature extraction unit 220 divides the speech period detected by the speech period detection unit 206 into frames and extracts speaker features for each frame. The speaker label determination unit 222 determines speaker information for each frame based on the extracted speaker features representing the speaker characteristics, and assigns speaker information (speaker label) of the determined speaker.

[0100] The voice data dividing unit 210 divides the voice data to be subjected to voice recognition based on the speech interval detected by the speech interval detection unit 206 and the timing at which speaker information switches within the speech interval. The timing at which speaker information switches within the speech interval can be determined by the speaker label assigned by the speaker label determination unit 222. Even if the speech interval detected by the speech interval detection unit 206 includes voices of multiple speakers, the voice data dividing unit 210 divides the voice data based on the timing at which the speaker switches within the speech interval. The voice data divided by the voice data dividing unit 210 includes the speech content of one speaker.

[0101] The speech recognition unit 212 performs speech recognition on each piece of speech data divided by the speech data division unit 210. The speech recognition unit 212 performs speech recognition on speech data that has been divided by the speech data division unit 210 so as to include the speech of one speaker and has consistent acoustic features, thereby maintaining better speech recognition accuracy even in speech segments that include speech from multiple speakers.

[0102] The recording processing unit 214 records audio data by video or audio recording. The storage unit 216 stores the audio data recorded by the recording processing unit 76. The storage unit 216 also stores the results of speech recognition of the audio data. The result output unit 218 outputs the results of speech recognition and speaker information by, for example, displaying them on a screen. The editing receiving unit 219 receives an operation from the user to edit the output speech recognition results and speaker information.

[0103] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), a conventional circuit module, and other devices designed to perform each of the above-described functions.

[0104] The devices described in the examples merely represent one of multiple computing environments for implementing the embodiments disclosed herein. The present invention is not limited to these embodiments, and the components in these embodiments include those that would be easily conceived by a person skilled in the art, those that are substantially identical, and those that are within the scope of what is called equivalents. Furthermore, various omissions, substitutions, modifications, and combinations of the components can be made without departing from the spirit of the present embodiments.

[0105] <Additional Notes> [Appendix 1] a speech interval detection unit that detects speech intervals in speech data including speeches of a plurality of speakers; a speaker information assigning unit that assigns speaker information identifying each speaker from the plurality of speakers to the audio data in order to identify a timing at which the speaker switches within the detected speech section; a voice data dividing unit that divides the voice data for each piece of speaker information at a timing when the speaker information assigned within the utterance section changes; a voice recognition unit that performs voice recognition on each of the voice data divided for each of the speaker information; A speech recognition device having: [Appendix 2] The speaker information providing unit a speaker feature extraction unit that extracts speaker features representing characteristics of the speaker from the speech section for each predetermined unit; a speaker label determination unit that determines speaker information based on the extracted speaker features and assigns the determined speaker information to the speech data; have 2. The speech recognition device according to claim 1. [Appendix 3] The speaker label determination unit determines whether each speaker is speaking at each time and assigns the result of the determination as the speaker information. 3. The speech recognition device of claim 2. [Appendix 4] The speaker label determination unit divides the set of extracted speaker features into a plurality of groups, associates the speaker information with each group, determines the speaker information from the speaker features, and assigns the determined speaker information to the speech data. 3. The speech recognition device according to claim 2. [Appendix 5] The speaker label determination unit determines the number of groups according to a specification of the number of speakers included in the speech data received from a user, and assigns the speaker information to the speech data based on the determined number of groups. 5. The speech recognition device according to claim 4. [Appendix 6] The speaker label determination unit determines the number of groups according to a range of the number of speakers included in the speech data received from a user, and assigns the speaker information to the speech data based on the determined number of groups. 5. The speech recognition device according to claim 4. [Appendix 7] The speaker label determination unit enables or disables the assignment of the speaker information based on a valid or invalid designation received from a user. 7. A speech recognition device according to any one of appendices 2 to 6. [Appendix 8] a result output unit that outputs the speech recognition result and the speaker information; an edit receiving unit that receives an operation to edit the speech recognition result and the speaker information from a user; 8. A speech recognition device according to any one of claims 1 to 7, comprising: [Appendix 9] the speaker feature extraction unit divides the speech section into predetermined units such that each unit overlaps the preceding and succeeding speech sections in time, and extracts speaker features for each predetermined unit; The speaker label determination unit determines the speaker information for each predetermined unit based on the extracted speaker features, and assigns the determined speaker information to the speech data. 7. A speech recognition device according to any one of appendices 2 to 6. [Appendix 10] The speaker information providing unit performs speaker diarization processing. 10. A speech recognition device according to any one of appendices 1 to 9. [Appendix 11] A speech recognition system including a communication terminal and a speech recognition device, The communication terminal a receiving unit that receives, from a user, designation of speech data including speeches of a plurality of speakers for speech recognition; and The speech recognition device a speech period detection unit that detects a speech period of the audio data; a speaker information assigning unit that assigns speaker information identifying each speaker from the plurality of speakers to the audio data in order to identify a timing at which the speaker switches within the detected speech section; a voice data dividing unit that divides the voice data for each piece of speaker information at a timing when the speaker information assigned within the utterance section changes; a voice recognition unit that performs voice recognition on each of the voice data divided for each of the speaker information; A speech recognition system having: [Appendix 12] A speech recognition method performed by a speech recognition device, a speech interval detection step of detecting speech intervals of speech data including speeches of a plurality of speakers; a speaker information assigning step of assigning speaker information identifying each speaker from the plurality of speakers to the audio data in order to identify a timing at which a speaker switches within the detected speech section; a voice data dividing step of dividing the voice data for each piece of speaker information at a timing when the speaker information assigned within the utterance section changes; a speech recognition step of performing speech recognition on each of the speech data divided for each of the speaker information; A speech recognition method comprising: [Appendix 13] For voice recognition devices, a speech segment detection step of detecting speech segments in speech data including speeches from multiple speakers; a speaker information assigning step of assigning speaker information identifying the speaker to the audio data in order to identify a timing at which the speaker switches within the detected speech section; a voice data division step of dividing the voice data for each piece of speaker information at a timing when the speaker information assigned within the speech section changes; a speech recognition step of performing speech recognition on each of the speech data divided for each of the speaker information; A program that executes the following. [Explanation of symbols]

[0106] 1. Voice Recognition System 10 Voice recognition device 14 Conference communication terminals 32 Audio data acquisition unit 34 Speech Activity Detection Unit 36 Speaker information assignment unit 38 Audio data division unit 40 Voice Recognition Unit [Prior art documents] [Patent documents]

[0107] [Patent Document 1] Japanese Patent Publication No. 2023-004116

Claims

1. a speech interval detection unit that detects speech intervals in speech data including speeches of a plurality of speakers; a speaker information assigning unit that assigns speaker information identifying each speaker from the plurality of speakers to the audio data in order to identify a timing at which the speaker switches within the detected speech section; a voice data dividing unit that divides the voice data for each piece of speaker information at a timing when the speaker information assigned within the utterance section changes; a voice recognition unit that performs voice recognition on each of the voice data divided for each of the speaker information; A speech recognition device having:

2. The speaker information providing unit a speaker feature extraction unit that extracts speaker features representing characteristics of the speaker from the speech section for each predetermined unit; a speaker label determination unit that determines speaker information based on the extracted speaker features and assigns the determined speaker information to the speech data; have 2. The speech recognition device according to claim 1.

3. The speaker label determination unit determines whether each speaker is speaking at each time and assigns the result of the determination as the speaker information.

3. The speech recognition device according to claim 2.

4. The speaker label determination unit divides the set of extracted speaker features into a plurality of groups, associates the speaker information with each group, determines the speaker information from the speaker features, and assigns the determined speaker information to the speech data.

3. The speech recognition device according to claim 2.

5. The speaker label determination unit determines the number of groups according to a specification of the number of speakers included in the speech data received from a user, and assigns the speaker information to the speech data based on the determined number of groups.

5. The speech recognition device according to claim 4.

6. The speaker label determination unit determines the number of groups according to a range of the number of speakers included in the speech data received from a user, and assigns the speaker information to the speech data based on the determined number of groups.

5. The speech recognition device according to claim 4.

7. The speaker label determination unit enables or disables the assignment of the speaker information based on a valid or invalid designation received from a user.

3. The speech recognition device according to claim 2.

8. a result output unit that outputs the speech recognition result and the speaker information; an edit receiving unit that receives an operation to edit the speech recognition result and the speaker information from a user; 2. The speech recognition device according to claim 1, comprising:

9. the speaker feature extraction unit divides the speech section into predetermined units such that each unit overlaps the preceding and succeeding speech sections in time, and extracts speaker features for each predetermined unit; The speaker label determination unit determines the speaker information for each predetermined unit based on the extracted speaker features, and assigns the determined speaker information to the speech data.

3. The speech recognition device according to claim 2.

10. The speaker information providing unit performs speaker diarization processing.

3. The speech recognition device according to claim 1.

11. A speech recognition system including a communication terminal and a speech recognition device, The communication terminal a receiving unit that receives, from a user, designation of speech data including speeches of a plurality of speakers for speech recognition; and The speech recognition device a speech period detection unit that detects a speech period of the audio data; a speaker information assigning unit that assigns speaker information identifying each speaker from the plurality of speakers to the audio data in order to identify a timing at which the speaker switches within the detected speech section; a voice data dividing unit that divides the voice data for each piece of speaker information at a timing when the speaker information assigned within the utterance section changes; a voice recognition unit that performs voice recognition on each of the voice data divided for each of the speaker information; A speech recognition system having:

12. A speech recognition method performed by a speech recognition device, a speech interval detection step of detecting speech intervals of speech data including speeches of a plurality of speakers; a speaker information assigning step of assigning speaker information identifying each speaker from the plurality of speakers to the audio data in order to identify a timing at which a speaker switches within the detected speech section; a voice data dividing step of dividing the voice data for each piece of speaker information at a timing when the speaker information assigned within the utterance section changes; a speech recognition step of performing speech recognition on each of the speech data divided for each of the speaker information; A speech recognition method comprising:

13. For voice recognition devices, a speech segment detection step of detecting speech segments in speech data including speeches from multiple speakers; a speaker information assigning step of assigning speaker information identifying the speaker to the audio data in order to identify a timing at which the speaker switches within the detected speech section; a voice data division step of dividing the voice data for each piece of speaker information at a timing when the speaker information assigned within the speech section changes; a speech recognition step of performing speech recognition on each of the speech data divided for each of the speaker information; A program that executes the following.

Images