Registration method and device, speaker recognition method and device

Abstract

A registration method and device, and a speaker recognition method and device are disclosed. The registration method includes: receiving a speech signal of a speaker; synthesizing the received speech signal with a noise signal to generate a synthesized signal; generating a feature vector based on the synthesized signal; and constructing a registration database (DB) corresponding to the speaker based on the generated feature vector.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2019-0026714 filed on March 8, 2019, in the Korean Intellectual Property Office, and Korean Patent Application No. 10-2019-0051557 filed on May 2, 2019, in the Korean Intellectual Property Office, the disclosures of which are incorporated herein in their entirety by reference for all purposes. Technical Field

[0002] The following description relates to methods and apparatus with speaker recognition. Background Art

[0003] Speaker recognition is used to authenticate or identify a speaker based on his or her voice or speech, and can be applied in various situations and application areas (e.g., recognition in gatherings, meetings, and conversations or sessions). Speaker recognition systems can be applied to vehicles, buildings, and bank accounts for access control for security purposes. Summary of the invention

[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0005] In one general aspect, a method with registration includes: receiving a speech signal of a speaker; synthesizing the received speech signal with a noise signal to generate a synthesized signal; generating a feature vector based on the synthesized signal; and constructing a registration database (DB) corresponding to the speaker based on the generated feature vector.

[0006] The method may further include generating a second feature vector based on the received speech signal, wherein the step of constructing the registration DB includes constructing the registration DB based on any one or both of the feature vector and the second feature vector.

[0007] The step of constructing the registration DB may include: clustering a plurality of feature vectors including any one or both of the feature vector and the second feature vector into a plurality of groups; and extracting at least one representative feature vector corresponding to at least one of the plurality of groups.

[0008] The second feature vector may be based on the received speech signal without synthesis.

[0009] The step of generating the feature vector may include: performing a domain transformation on the synthesized signal; and extracting the feature vector from a result of the domain transformation.

[0010] The synthesized signal may include time domain sound information, and the result of the domain transformation may include frequency domain image information. The step of generating the feature vector may include extracting the feature vector from the frequency domain image information using a convolutional neural network (CNN).

[0011] The step of synthesizing may comprise adding a noise signal to the received speech signal. The noise signal may comprise an additive noise signal.

[0012] The additive noise signal may be determined based on any one or any combination of any two or more of the type of noise, the timing of the noise, and a signal-to-noise ratio (SNR) indicating an energy ratio of the noise to the speech signal.

[0013] The step of synthesizing may include performing a convolution operation on the received speech signal and the noise signal. The noise signal may include a channel noise signal.

[0014] The channel noise signal may be determined based on at least one of a type of noise and an SNR indicating an energy ratio of the noise to the speech signal.

[0015] The step of generating the feature vector may include: dividing the synthesized signal into a plurality of parts; and extracting at least one partial feature vector corresponding to at least one of the plurality of parts.

[0016] The method may further include extracting a start point and an end point of sound information included in any one or both of the speech signal and the synthesized signal, wherein the start point and the end point are applied to any one or both of the synthesizing step and the generating step.

[0017] The step of receiving a speech signal may include: receiving a speech signal of a speaker through a plurality of channels. The step of synthesizing and the step of generating may be performed independently for each of the plurality of channels. The step of constructing a registration DB may include: constructing a registration DB based on feature vectors generated through the plurality of channels.

[0018] The method may further include: after receiving the speech signals, estimating noise based on a difference between the received speech signals; and subtracting the noise from each of the received speech signals.

[0019] The step of receiving the speech signal may include: receiving the speech signal of the speaker through a plurality of channels. The method may also include: determining the speech signal for synthesis by pre-processing the received speech signal.

[0020] The determining of the voice signal may include selecting a voice signal from among the received voice signals based on whether a recognition error due to hardware specifications is included in the received voice signals.

[0021] The determining of the voice signal may include selecting one voice signal from the received voice signals for each of the plurality of parts of the received voice signal based on whether a recognition error due to hardware specifications is included in each part of the received voice signal.

[0022] The method may further include identifying a speaker by comparing the input speech signal with the generated feature vector.

[0023] In another general aspect, a non-transitory computer-readable storage medium stores instructions that, when executed by a processor, cause the processor to perform the above method.

[0024] In another general aspect, a speaker recognition method includes: receiving a speech signal of a speaker; generating at least one input feature vector based on the received speech signal; comparing the at least one input feature vector with at least one registration feature vector of a registered user stored in a registration database (DB); and identifying the speaker based on a result of the comparison, wherein the at least one registration feature vector is generated based on a synthesized signal of a speech signal and a noise signal used to register the registered user.

[0025] The registered users may include a plurality of registered users. The step of identifying the speaker may include: recognizing the speaker from among the plurality of registered users.

[0026] The step of generating the at least one input feature vector may include: performing a domain transformation on the received speech signal; and extracting the input feature vector from a result of the domain transformation.

[0027] The step of performing the domain transformation may include: performing a Fast Fourier Transform (FFT) on the received speech signal; and performing padding on a result of the FFT based on the result of the FFT.

[0028] The step of generating the at least one input feature vector may include performing a sliding window operation on the received speech signal so that a window size of the input feature vector is equal to a window size of the registration feature vector.

[0029] The number of the at least one input feature vector may be determined based on a security level or a response time required by an application.

[0030] The noise signal may include any one or both of an additive noise signal and a channel noise signal.

[0031] The step of receiving a speech signal may include: receiving a speech signal of a speaker through a plurality of channels. The step of generating and the step of comparing may be performed independently for each of the plurality of channels. The registration feature vector may include a registration feature vector generated through the plurality of channels.

[0032] The speaker recognition method may further include: after receiving the voice signals, estimating noise based on a difference between the received voice signals; and subtracting the noise from each of the received voice signals.

[0033] The step of receiving the speech signal may include: receiving the speech signal of the speaker through a plurality of channels. The speaker recognition method may further include: determining the speech signal for synthesis by pre-processing the received speech signal.

[0034] The determining of the voice signal may include selecting a voice signal from among the received voice signals based on whether a recognition error due to hardware specifications is included in the received voice signals.

[0035] The determining of the voice signal may include selecting one voice signal from the received voice signals for each of the plurality of parts of the received voice signal based on whether a recognition error due to hardware specifications is included in each part of the received voice signal.

[0036] The method may further include, before receiving the speech signal, generating the at least one registration feature vector based on the synthesized signal, and storing the at least one registration feature vector in a registration DB.

[0037] In another general aspect, a non-transitory computer-readable storage medium stores instructions that, when executed by a processor, cause the processor to perform the above-described speaker recognition method.

[0038] In another general aspect, a device with registration includes: an acoustic sensor configured to receive a speech signal of a speaker; and a processor configured to synthesize the received speech signal with a noise signal to generate a synthesized signal, generate a feature vector based on the synthesized signal, and construct a registration database (DB) corresponding to the speaker based on the generated feature vector.

[0039] The processor may be further configured to generate a second feature vector based on the received voice signal, and construct a registration DB based on any one or both of the feature vector and the second feature vector.

[0040] The processor may be further configured to cluster a plurality of feature vectors including any one or both of the feature vector and the second feature vector into a plurality of groups, and extract at least one representative feature vector corresponding to at least one of the plurality of groups.

[0041] The second feature vector may be based on the received speech signal without synthesis.

[0042] The processor may be further configured to perform a domain transform on the synthesized signal and extract a feature vector from a result of the domain transform.

[0043] The processor may be further configured to add a noise signal to the received speech signal. The noise signal may include an additive noise signal.

[0044] The processor may also be configured to perform a convolution operation on the received speech signal and the noise signal. The noise signal may include a channel noise signal.

[0045] The processor may be further configured to segment the synthesized signal into a plurality of parts and extract at least one partial feature vector corresponding to at least one of the plurality of parts.

[0046] The processor may be further configured to identify a speaker by comparing the input speech signal to the generated feature vector.

[0047] In another general aspect, a device with speaker recognition includes: an acoustic sensor configured to receive a speech signal of a speaker; and a processor configured to generate at least one input feature vector based on the received speech signal, compare the at least one input feature vector with at least one registration feature vector of a registered user stored in a registration database (DB), and identify the speaker based on a result of the comparison, wherein the at least one registration feature vector is generated based on a synthesized signal of a speech signal and a noise signal for registering the registered user.

[0048] The processor may be further configured to perform a domain transformation on the received speech signal and extract an input feature vector from a result of the domain transformation.

[0049] The processor may also be configured to: perform a sliding window operation on the received speech signal so that a window size of the input feature vector is equal to a window size of the registration feature vector.

[0050] The processor may be further configured to: perform a Fast Fourier Transform (FFT) on the received speech signal; and perform padding on a result of the FFT based on the result of the FFT.

[0051] The processor may be further configured to: before receiving the speech signal, generate the at least one registration feature vector based on the synthesized signal, and store the at least one registration feature vector in the registration DB.

[0052] Other features and aspects will be apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 An example of a speaker identification method is shown.

[0054] Figure 2 An example of a registration system for speaker identification is shown.

[0055] Figure 3 A flow chart illustrating an example of a registration method for speaker identification.

[0056] Figure 4 An example of a method of generating a feature vector by synthesizing a speech signal and additive noise is shown.

[0057] Figure 5 An example of a method of generating a feature vector by synthesizing a speech signal and channel noise is shown.

[0058] Figure 6 An example of a method of generating a feature vector by segmentation is shown.

[0059] Figure 7 An example of a method of constructing a registration database (DB) is shown.

[0060] Figure 8 An example of a speaker verification method is shown.

[0061] Fig. 9 A flow chart illustrating an example of a speaker identification method.

[0062] Fig.10 An example of a speaker identification method is shown.

[0063] Fig.11 An example of a registration device for speaker recognition is shown.

[0064] Fig.12 An example of a speaker verification method using multiple channels is shown.

[0065] Fig.13 An example of a speaker verification method through channel selection is shown.

[0066] Fig.14 An example of a method that is robust to utterance changes due to the position and orientation of the registration device is shown.

[0067] Throughout the drawings and detailed description, the same drawing reference numerals represent the same elements, features, and structures. The drawings may not be to scale, and the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0068] The following specific embodiments are provided to help the reader gain a comprehensive understanding of the methods, devices and / or systems described herein. However, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear after understanding the disclosure of the present application. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order.

[0069] The features described herein can be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein that will be clear after understanding the disclosure of the present application.

[0070] Here, it should be noted that the use of the term "may" with respect to examples or embodiments (e.g., with respect to what an example or embodiment may include or implement) means that there is at least one example or embodiment that includes or implements such features, while all examples and embodiments are not limited thereto.

[0071] Throughout the specification, when a component is described as being “connected to” or “coupled to” another component, the component may be directly “connected to” or “coupled to” the other component, or one or more other components may be present in between. Conversely, when an element is described as being “directly connected to” or “directly coupled to” another element, there may be no other elements in between. Likewise, similar expressions (e.g., “between” versus “immediately between” and “adjacent to” versus “immediately adjacent to”) should also be interpreted in the same manner.

[0072] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0073] Although terms such as "first", "second", and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions should not be limited by these terms. Instead, these terms are only used to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Therefore, without departing from the teachings of the examples described herein, the first member, first component, first region, first layer, or first portion referred to in the examples may also be referred to as the second member, second component, second region, second layer, or second portion.

[0074] The terms used herein are only used to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the described features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0075] The speaker recognition method and apparatus can be applied to vehicles, buildings, and bank accounts for access control for security purposes. According to the following description, a system and method for verifying or authenticating a speaker from the speaker's voice can be robust to noise and can utilize limited speaker's voice information.

[0076] Figure 1 An example of a speaker recognition method is shown. Figure 1 The speaker recognition method includes a registration process and a test process. The speaker recognition method can be performed by a speaker recognition device, and the speaker recognition device can be implemented by one or more hardware modules or a combination of hardware modules and software modules.

[0077] In order to perform the speaker recognition method, in a registration process, the speaker recognition device may generate a feature vector from an input sound or voice of a speaker, and store the generated feature vector in a registration database (DB). In a test process, the speaker recognition device may compare the feature vector generated from the input voice with the feature vector stored in the registration DB, and determine whether the speaker is recognized. Hereinafter, for ease of description, an example in which the registration process and the test process are performed by the same speaker recognition device will be described. However, the example may also be applied to an example in which the registration process and the test process are performed by different devices.

[0078] Reference Figure 1 , the voice signal 110 may be a voice signal of a speaker corresponding to a preset sentence as a target for verification. For example, the voice signal 110 is a voice signal of a speaker corresponding to a sentence to be recognized (eg, "hi, bixby" as shown in the figure).

[0079] In the registration process, the speaker recognition device receives a voice signal 110 corresponding to "hi, bixby" as a voice uttered by a speaker, generates a feature vector, and stores the generated feature vector in the registration DB. In the test process, when the voice signal 110 corresponding to the voice "hi, bixby" is received, the speaker recognition device generates a feature vector corresponding to the voice signal 110, compares the generated feature vector with the feature vector stored in the registration DB, and determines whether the speaker is recognized.

[0080] The registration process may be performed in a quiet environment with relatively small noise, while the test process may be performed in an environment with relatively large noise. Therefore, in the registration process, a feature vector may be generated based on a speech signal (e.g., speech signal 120) in an environment with relatively small noise. However, in the test process, noise may be added to the speaker's speech signal based on the surrounding environment, and a feature vector may be generated based on the speech signal to which noise is added (e.g., speech signal 130 and speech signal 140). Noise may include additive noise, channel noise, or a combination thereof. The speech signal to which noise is added may include, for example, a speech signal 130 to which additive noise is added and a speech signal 140 to which channel noise is added. In such an example, a mismatch between the feature vector generated in the registration process and the feature vector generated in the test process may occur, and thus the performance level of recognition may be reduced. In addition, in the case where the number of utterances of the speaker's speech signal corresponding to the sentence to be recognized in the registration process is relatively small, errors in the feature vector may easily occur, and thus recognition may not be easily performed.

[0081] In order to help prevent such a degradation in performance that may occur due to a mismatch between an environment in which a registration process is performed (hereinafter, "registration environment") and an environment in which a test process is performed (hereinafter, "test environment"), a feature vector may be registered based on a synthesized signal obtained by synthesizing a speaker's speech signal with a preset noise signal in the registration process. Thus, when verification is performed in an actual environment that is different from the registration environment and is exposed to various types of noise, the performance level of verification may be improved.

[0082] Speaker recognition can be classified into speaker verification and speaker identification. Speaker verification is to verify whether the speaker is a registered user, and speaker identification is to identify the speaker from among registered users.

[0083] Speaker recognition can be implemented in various embodiments in a wide range of fields, for example, embodiments implementing management of personal electronic documents, and embodiments implementing banking and e-commerce performed through example embodiment telephone or Internet. For speaker recognition, methods using sound or voice are generally used because there is a low risk of theft or counterfeiting compared to physical mechanisms such as cards, seals, signatures, and identity (ID) cards. In addition, speaker recognition using sound or voice can be easily performed from a long distance by an acoustic sensor such as a microphone, which is relatively low-priced compared to other methods using other biometric means such as fingerprints and retinas (which require relatively high-priced devices or equipment).

[0084] In the following, reference will be made to Figures 2 to 7 A noise-robust registration method for speaker recognition is described. Figure 8 and Fig. 9 Describes the speaker verification method and will refer to Fig.10 The speaker identification method is described. In addition, reference will be made to Fig.11 Describes a registered device with speaker recognition.

[0085] Figure 2 An example of a registration system for speaker recognition is shown. Figure 2 , a registration device for speaker recognition receives a speech signal 210 of a speaker. The registration device described herein means a device configured to construct a registration DB corresponding to a speaker for speaker recognition, and can be implemented by, for example, a hardware module or a module including both hardware and software (e.g., a processor configured to execute instructions). The registration device can be set or used in various computing devices and / or systems (e.g., smart phones, tablet computers, laptop computers, desktop computers, televisions (TVs), wearable devices, security systems, smart home systems, etc.).

[0086] The speech signal 210 may be a signal that is not processed after being received by the acoustic sensor of the registration device, or a signal from which a component corresponding to noise is removed after being received by the acoustic sensor. Even for an unprocessed signal, the registration process can generally be performed in a quiet environment with less noise, so the unprocessed signal is not significantly different from the signal from which the noise is removed.

[0087] like Figure 2 As shown in , in order to help prevent performance degradation that may occur due to a mismatch between the registration environment and the test environment, the registration device may synthesize the voice signal 210 with the additive noise signal 220, the channel noise signal 230, or a combination of the additive noise signal 220 and the channel noise signal 230. The additive noise may be generated from a source different from the source of the voice signal and may have no correlation with the voice signal. Additive noise can be added to the voice signal by addition. Additive noise may include, for example, door slams, horns, and ambient noise. In addition, channel noise may be a type of noise detected in a conversion or transformation process. Channel noise may include, for example, a room impulse response. However, the types of noise described in the foregoing description are provided only as examples of additive noise and channel noise, and thus additive noise and channel noise may include various other types of noise. In the following, reference will be made to each of the following: Figure 4 and Figure 5 The method of synthesizing the speech signal 210 with the additive noise signal 220 and the channel noise signal 230 is described in more detail.

[0088] Refer again Figure 2In one example, the registration device aligns the speech signal in stage 240. By aligning the speech signal, the registration device can determine the target portion to be processed, and extract the starting point and the ending point of the sound information included in the speech signal to improve the efficiency of registration and recognition. For example, the registration device can measure the strength of the speech signal to extract the starting point and the ending point of the sound information included in the speech signal, and set the portion between the extracted starting point and the extracted ending point as the target portion to be processed. In this example, the speech signal to be aligned can be a synthesized signal in which the additive noise signal 220, the channel noise signal 230, or a combination thereof is synthesized.

[0089] In another example, the registration device may extract the starting point and the ending point of the sound information by first aligning the speech signal 210, and then synthesize the speech signal 210 with the additive noise signal 220, the channel noise signal 230, or a combination of the additive noise signal 220 and the channel noise signal 230. Alternatively, the speech signal 210 may first be aligned in another device (e.g., a digital signal processor) and then input to the registration device. Subsequently, the additive noise signal 220, the channel noise signal 230, or a combination thereof may be synthesized with the speech signal 210.

[0090] In one example, in stage 250, the registration device divides the synthesis signal of the speech signal and the noise signal into a plurality of parts. Figure 6 The method of segmenting the composite signal is described in more detail.

[0091] Reference Figure 2 , the feature vector generator 260 of the registration device generates a feature vector based on a synthesized signal of the speech signal and the noise signal. The feature vector generator 260 recognizes the synthesized signal and outputs a feature vector corresponding to the recognized synthesized signal. The feature vector may include information that distinguishes each recognition element, for example, time-based frequency information having a compressed component required for recognition in the speech signal.

[0092] Therefore, by generating and registering a feature vector based on a synthesized signal obtained by synthesizing a speaker's speech signal and various types of noise in a registration process, performance degradation that may occur due to a mismatch between a registration environment and a test environment can be prevented.

[0093] Figure 3 A flow chart showing an example of a registration method for speaker recognition. Figure 3 In operation 310, a registration device for speaker recognition receives a voice signal of a speaker. The registration device may receive a preset number of voice signals. For example, the registration device repeatedly receives a voice signal of a speaker corresponding to a sentence to be recognized (eg, "hi, bixby") N times.

[0094] In operation 320, the registration device synthesizes the received speech signal with a noise signal. The noise signal may be a signal that is preset to be similar to noise that may occur in the test process, and includes, for example, an additive noise signal, a channel noise signal, or a combination thereof.

[0095] In operation 330, the registration device generates a feature vector based on a synthesized signal of the speech signal and the noise signal. Various methods may be performed to extract the feature vector from the synthesized signal. For example, the feature vector is generated by generating an image pattern from the synthesized signal via a fast Fourier transform (FFT) and inputting the generated image pattern into an artificial neural network that generates the feature vector. Figures 4 to 6 Such a feature vector generation method is described in more detail. In addition, the registration device generates a second feature vector based on the speech signal that is not synthesized with the noise signal.

[0096] In operation 340, the registration device constructs a registration DB corresponding to the speaker based on the generated feature vector. The registration device may construct the registration DB based on at least one of the generated feature vector and the generated second feature vector. Speaker recognition may be performed in the test process based on the registration DB constructed through operations 310 to 340.

[0097] Figure 4 An example of a method for generating a feature vector by synthesizing a speech signal and additive noise is shown. Figure 4 , the registered device receives the voice signal 410. Figures 1 to 3 The description provided is applicable to Figure 4 method, and therefore a more detailed description and repeated description will be omitted here.

[0098] The additive noise may occur from another source different from the source from which the speech signal 410 occurs, and thus there is no correlation between the additive noise and the speech signal 410. In one example, a synthetic signal is generated by adding the additive noise signal 420 to the speech signal 410. In this example, when the speech signal 410 is a plurality of speech signals, the additive noise signal 420 will be added to each speech signal, and thus a plurality of synthetic signals are generated.

[0099] The additive noise signal 420 can be determined in various ways. For example, the additive noise signal 420 is determined based on at least one of the type of noise, the timing of the noise, and the energy ratio of the noise to the speech signal (i.e., the signal-to-noise ratio (SNR)). For another example, the additive noise signal 420 is determined based on the repetition and / or combination of the noise signal determined based on the type of noise, the timing of the noise, and the SNR.

[0100] The type of noise represents the type of source of the noise including, for example, nearby gibberish, door slamming, and horn sounds. Even if the noise is generated from the same source, the type of noise may be different from each other based on the length of the noise. The timing of the noise represents the starting point and / or ending point of the synthesis of the noise and the speech signal. The SNR represents the relative volume difference between the speech signal and the noise.

[0101] As shown in the figure, the registration device generates a feature vector 450 by inputting a synthesized signal of a speech signal 410 and an additive noise signal 420 into a feature vector generator. The feature vector generator transforms the domain of the synthesized signal and extracts the feature vector 450 from the result of the transformation. The synthesized signal is generated by adding the additive noise signal 420 to the speech signal 410, and thus may include time domain sound information. The feature vector generator transforms the time domain synthesized signal into a frequency domain form including image information, and generates a feature vector 450 based on the transformation.

[0102] In one example, the feature vector generator includes a preprocessor 430 and a neural network 440. The preprocessor 430 transforms the time domain synthetic signal into a frequency domain form including image information. For example, the preprocessor 430 uses FFT to transform the domain of the synthetic signal into the frequency domain to obtain spectrum size information, and extracts Figure 1 The image information is shown in the frequency domain (e.g., as shown by the representations of signals 120, 130, and 140). Figure 1 In the representation of the signals 120 , 130 , and 140 in , the x-axis indicates the time domain, the y-axis indicates the frequency domain, and the brightness level may be proportional to the spectrum size.

[0103] The feature vector 450 is extracted by inputting the result of the domain transformation including the image information to the neural network 440. The neural network 440 can be implemented in various architectures and includes, for example, a convolutional neural network (CNN) model. The CNN model includes one or more convolutional layers, one or more pooling layers, and one or more fully connected layers, and a feature vector can be generated from the image information to be input thereto by the operation performed by each layer. The image information may indicate a pixel value of a pixel included in the image, for example, a brightness value.

[0104] Although a CNN model is described above, the neural network 440 may include other types of models including a recurrent neural network (RNN) model, a deep belief network model, a fully connected network model, a bidirectional neural network model, a restricted Boltzmann machine model, or may include different neural network parts or overlapping neural network parts having full connections, convolutional connections, recurrent connections and / or bidirectional connections, respectively.

[0105] The preprocessor 430 and the neural network 440 are provided as examples, and the feature vector generator may also be based on other models configured to generate feature vectors in addition to the aforementioned architectures of the preprocessor 430 and the neural network 440. For example, the feature vector generator may also be based on a linear predictive coding (LPC) method modeled by the human vocal tract, a filter bank method modeled by the human auditory organ (e.g., Mel-scaled frequency cepstral coefficients (MFCC)), or a combination thereof (e.g., a method of transforming a speech signal based on MFCC and inputting the transformed result to a neural network).

[0106] Figure 5 An example of a method for generating a feature vector by synthesizing a speech signal and channel noise is shown. Figure 5 , the channel noise may be noise generated based on the inherent characteristics of the location of the input speech signal 510 that is dissimilar to additive noise generated from a source different from the source of the speech signal 510. The registration device generates a synthetic signal by performing a convolution operation on the speech signal 510 and the channel noise signal 520.

[0107] The channel noise signal 520 may be determined based on at least one of the type of noise and the SNR. For example, the channel noise signal 520 may be determined by repetition and / or combination of noise signals determined based on the type of noise and the SNR.

[0108] The type of noise may differ based on the type of location where reverberation is generated (e.g., a large room, a small room, and a concert hall). In addition, the type of noise may differ based on the characteristics of the microphone. SNR represents the relative volume difference between the speech signal and the noise signal. Figure 4 The description of the feature vector generator also applies here, and therefore a more detailed description and repeated description will be omitted here.

[0109] Figure 6 An example of a method of generating a feature vector by segmentation is shown. Figure 6 , the registered device receives a voice signal 610 .

[0110] When a speech signal of a speaker corresponding to a sentence to be recognized is emitted relatively few times in the registration process, the performance of a recognizer configured to recognize the starting point of an utterance may become more sensitive, and thus an error in a representative feature vector may easily occur, and recognition may not be easily performed. In addition, when a method using only a portion of the speech signal 610 (e.g., a center cropping method that selects a center portion of a speech signal) is used to generate a feature vector, a deviation may increase when selecting an utterance point.

[0111] In one example, the registration device divides the speech signal 610 into a plurality of parts in stage 620 so as to register and use the limited speech in various ways. The division in stage 620 may be performed in a time series, and the window sizes of the plurality of parts may be the same. In addition, there may be overlapping parts between adjacent parts among the plurality of parts.

[0112] The feature vector generator 630 generates a feature vector 640 based on the plurality of parts. The feature vector generator 630 extracts at least one partial feature vector corresponding to at least one of the plurality of parts. Here, the partial feature vector means a feature vector generated based on the part segmented by the segmentation in stage 620. For example, when the registration device segments one speech signal into M parts in stage 620, M partial feature vectors are generated. For another example, when the registration device receives N speech signals 610 and segments each of the received N speech signals 610 into M parts in stage 620, N×M partial feature vectors are generated.

[0113] Reference above Figures 4 to 6 Describe a method for generating a feature vector. The method may include referring to Figure 4 The example method of generating a feature vector by synthesizing a speech signal and additive noise, as shown in Figure 5 The example method of generating a feature vector by synthesizing a speech signal and channel noise, as shown in Figure 6 The exemplary method for generating a feature vector by segmentation, and various other methods by combining the exemplary methods described above.

[0114] For example, the registration device synthesizes N speech signals with P additive noise signals, and performs segmentation to have M parts, thereby generating N×P×M feature vectors. For another example, the registration device synthesizes N speech signals with P1 additive noise signals and P2 channel noise signals, and performs segmentation to have M parts, thereby generating N×P1×P2×M feature vectors.

[0115] The registration device may store the generated feature vector in the registration DB to construct a registration DB corresponding to a speaker to be identified in speaker recognition. In a test process, speaker recognition may be performed based on the constructed registration DB.

[0116] Figure 7 An example of a method for constructing a registration DB is shown. Figure 7 ,The registration device constructs a registration DB based on the feature vector.

[0117] Although the registration device constructs the registration DB by storing all generated feature vectors in the registration DB, since comparison needs to be performed on each of all feature vectors in the test process, an outlier may greatly affect the construction, and the performance of processing data may be reduced.

[0118] In one example, the registration device clusters a plurality of feature vectors into a plurality of groups, extracts at least one representative feature vector corresponding to at least one of the groups, and registers the extracted representative feature vector to construct a registration DB.

[0119] The registration device clusters the feature vectors into a plurality of groups based on the type of noise and the segmented parts. Optionally, the registration device clusters the feature vectors into a plurality of groups based on the similarity of each feature vector. For example, the registration device clusters the feature vectors based on a K-means algorithm. The K-means algorithm represents an algorithm for grouping given data into k clusters, and the K-means algorithm is operable to minimize the variance of the distance difference between clusters. However, clustering is not limited to the aforementioned example method, and various methods may be used for clustering according to the design.

[0120] The registration device extracts at least one representative feature vector corresponding to at least one of the groups obtained by clustering. Here, the representative feature vector may be a representative value of the feature vector included in each group. For example, the representative feature vector is selected from the feature vector included in each group, or is determined by a statistical operation on the feature vector included in each group. The representative feature vector may be the average value of the feature values ​​included in each group, or the central value or centroid of the feature vector. Using the representative feature vector, the influence of the outlier can be reduced, and thus robust feature values ​​can be extracted. For example, the registration device clusters N×M feature vectors into multiple groups (each group includes K feature vectors), extracts the average value of each group as a representative value, and thus extracts a total of P representative feature vectors. However, the use of representative values ​​to extract representative feature vectors is not limited to the aforementioned example method. The number of feature vectors included in each group may vary from group to group, and various methods may be used to extract representative feature vectors according to the design.

[0121] Figure 8 An example of a speaker verification method is shown. Hereinafter, the voice signal 810 to be registered, the registration feature vector 835, and the representative registration feature vector 840 are used to explain the registration process, and indicate the same meanings of the voice signal, feature vector, and representative feature vector described previously, respectively. Therefore, the description of the voice signal 810 to be registered, the registration feature vector 835, and the representative registration feature vector 840 is substantially the same as the description of the voice signal, feature vector, and representative feature vector, and therefore the above reference is made to Figures 2 to 7 The description provided is also applicable to the following references Figure 8Describes the registration process.

[0122] In the registration process, the registration device receives a speech signal 810 of a speaker, and synthesizes the speech signal 810 with additive noise 815 or channel noise 820. After performing segmentation on the synthesized signal obtained by the synthesis in stage 825, the registration device generates a registration feature vector 835 by inputting the segmented result to a feature vector generator 830, and extracts a representative registration feature vector 840 to construct a registration DB. In order to store all the registration feature vectors 835 in the registration DB, extracting the representative registration feature vector 840 may be omitted.

[0123] The recognition device receives an input speech signal 845 of a speaker. The input speech signal 845 may indicate a speech signal obtained from the speaker's speech for verification in a test process. The test process may be performed in an environment with a lot of noise, and thus the input speech signal 845 may naturally include noise.

[0124] The recognition device may be a device configured to generate a feature vector from the input speech of the speaker and compare the generated feature vector with the feature vector stored in the registration DB and determine whether the speaker is recognized (including verification). The recognition device may be implemented, for example, by a hardware module or a combination of a hardware module and a software module. The recognition device may be set as a variety of computing devices and / or systems (e.g., smart phones, tablet computers, laptop computers, desktop computers, TVs, wearable devices, security systems, smart home systems, etc.), or be set in a variety of computing devices and / or systems (e.g., smart phones, tablet computers, laptop computers, desktop computers, TVs, wearable devices, security systems, smart home systems, etc.).

[0125] In one example, the identification device may include a registration device. Here, the identification device configured to perform the test process may also be referred to as an identification device that does not have a registration operation or relies only on pre-registered information, and the identification device including the registration device may also be referred to as an identification device that can generate and thereafter rely on such registration information. For ease of explanation only, the identification device and the registration device may be described separately here to distinguish the functions of identification from the functions of registration. Therefore, when the actual product is implemented, all functions may be configured to be processed in a single device (e.g., a broadly defined identification device). For example, the acoustic sensor of the registration device configured to receive the voice signal 810 may be the acoustic sensor of the identification device configured to receive the input voice signal 845. In addition, the feature vector generator 830 of the registration device and the feature vector generator 855 of the identification device may be the same feature vector generator. In the following, for ease of description, the identification device may indicate a reference to an identification device that, for example, includes or does not include a registration operation. Similarly, although the discussion of the registration device and method may describe the registration operation, the referenced registration device and method should be understood to also include identification device components and operations (i.e., the reference to the registration device or method is not limited to such registration components or operations).

[0126] The recognition device segments and extracts the input speech signal 845 of the speaker to be recognized so that the window size of the input feature vector is equal to the window size of the registration feature vector. For example, in stage 850, the recognition device performs center cropping on the input speech signal 845 of the speaker, or performs a sliding window operation on the input speech signal 845. This is because: when the window size of the input feature vector is the same as the window size of the registration feature vector, it may be possible to compare the vectors. Although for ease of description, the sliding window operation will be mainly described below, various methods may also be applied to segment and extract the input speech signal 845.

[0127] The recognition device inputs the result obtained by performing the sliding window operation in stage 850 to the feature vector generator 855 to generate an input feature vector 860. As described above, the feature vector generator 855 can generate the input feature vector 860 using the same neural network as that used for the feature vector generator 830.

[0128] In stage 865, the recognition device compares at least one input feature vector 860 with at least one registration feature vector 835 or a representative registration feature vector 840 of a registered user stored in the registration DB constructed in the registration process. The registered user described herein may be a speaker corresponding to the registration feature vector or the representative registration feature vector stored in the registration DB.

[0129] In stage 870, the recognition device verifies the speaker based on the result of the comparison obtained in stage 865. In stage 870, the recognition device verifies the speaker based on the similarity score between the input feature vector 860 and the registered feature vector 835 or the representative registered feature vector 840. For example, when the representative value of the similarity score is greater than or equal to a preset threshold, the recognition device determines that the verification of the speaker performed in stage 870 is successful, or determines that the speaker is successfully verified. Optionally, the recognition device may determine that the verification of the speaker performed in stage 870 is successful only when the number of the registered feature vectors 835 or the representative registered feature vectors 840, each having a similarity score greater than or equal to the threshold with the input feature vector 860, is greater than or equal to a preset number. However, the conditions for speaker verification in stage 870 are not limited to the conditions in the aforementioned description, and therefore various conditions may be applied to the speaker verification in stage 870.

[0130] For example, the similarity score may be determined in the form of a matching score or a distance score. The matching score may represent the degree of similarity between the input feature vector 860 and the registered feature vector 835 or the representative registered feature vector 840, or a score indicating the similarity between the feature vectors. A low matching score may indicate a low similarity between the feature vectors, or an indication that the feature vectors are not very similar to each other. A high matching score may indicate a high similarity between the feature vectors, or an indication that the feature vectors are highly similar to each other. The distance score may represent a score indicating a feature distance (e.g., Euclidean distance) between the input feature vector 860 and the registered feature vector 835 or the representative registered feature vector 840. A low distance score may indicate that the feature distance between the feature vectors in the feature vector space is short, and a high distance score may indicate that the feature distance is long.

[0131] Fig. 9 A flowchart showing an example of a speaker identification method. Fig. 9 , in operation 910, the recognition device receives a speech signal of a speaker.

[0132] In operation 920, the recognition device generates at least one input feature vector based on the received voice signal. Here, the number of input feature vectors may be determined based on the security level and response time required by the application. For example, the speaker recognition method described herein may be applied to executing a payment application and unlocking a smart phone. Compared to when the speaker recognition method is used for a payment application, when the speaker recognition method is applied to unlocking a smart phone, user convenience may be more significantly considered, and therefore a smaller number of input feature vectors may be sufficient when compared to when the speaker recognition method is used for a payment application.

[0133] The recognition device performs domain transformation on the speech signal of the speaker and extracts an input feature vector from the result of the domain transformation. For example, the recognition device performs FFT on the speech signal of the speaker and performs padding on the result of the FFT based on the result of the FFT.

[0134] The result obtained by performing FFT may not include data in a part thereof. In such a case, the data can be filled or filled by padding. In the case where there is no data in the part as a result of FFT, padding can be performed based on the data in the adjacent part adjacent to the part without data. This is because: the speech signal is a continuous signal, and the data value does not change rapidly during a sufficiently short time period. For example, when the part where data is lost includes a plurality of unit parts, the data in the first unit lost part can be determined to be the same value as the value of the data in the adjacent unit part that is adjacent to the first unit lost part and has no data lost, and the data in the second unit lost part can be determined to be the same value as the value of the data in the adjacent unit part that is adjacent to the filled data and has no data lost.

[0135] In operation 930, the recognition device compares at least one input feature vector with at least one registered feature vector of a registered user stored in the registration DB. In operation 940, the recognition device identifies the speaker based on the result of the comparison. Here, speaker recognition may include speaker verification and speaker identification. Speaker verification is to verify whether the speaker currently speaking is a registered user, and speaker identification is to identify the speaker from among various registered speakers. In the following, reference will be made to Fig.10 Describe the method of speaker identification.

[0136] Fig.10 An example of a speaker identification method is shown. Fig.10 , the speaker identification method can be used to identify the speaker who is currently speaking from among various registered users. Because there may be multiple registered users, the speaker identification method can be performed by performing speaker verification multiple times.

[0137] As a non-limiting example, the Figure 8 The registration process in the speaker identification method is performed in the same manner as the registration process in the speaker verification method described above. However, the speaker verification method may generate a feature vector of a speaker, and Fig.10 As shown in , the speaker identification method can generate a feature vector for each of the multiple speakers. For example, the speaker identification method can use the feature vector generator 1015 to generate and register the feature vectors 1020 of the registration speech signal 1010 of N speakers, and construct the registration DB of the N speakers.

[0138] Can be compared with the above Figure 8The test process in the speaker identification method is performed in the same manner as the test process in the speaker verification method described above. For example, the test process may generate an input feature vector 1040 for the input speech signal 1030 using the feature vector generator 1035. For example, the similarity between the input feature vector 1040 and the registration feature vector 1020 generated for each individual speaker or the representative registration feature vector 1025 generated for each individual speaker is measured in stage 1045 to determine a numerical similarity vector 1050. The numerical similarity vector 1050 may be an example of a similarity score. In stage 1055, the speaker identification method may identify the registered user corresponding to the registration feature vector 1020 or the representative registration feature vector 1025 having the largest numerical similarity vector 1050 for the input feature vector 1040 as a speaker. However, in such a case, the size of the numerical similarity vector may need to be greater than or equal to a threshold value. In the case where the size of all numerical similarity vectors is less than the threshold value, the speaker identification method may output that the speaker does not correspond to the registered user. Optionally, in stage 1055, the speaker identification method may identify as a speaker a registered user having a maximum number of registered feature vectors 1020 or representative registered feature vectors 1025 having a magnitude of a numerical similarity vector 1050 greater than or equal to a threshold. Optionally, in stage 1055, the speaker identification method may identify as a speaker a registered user having a maximum representative value of registered feature vectors 1020 or representative registered feature vectors 1025 whose magnitude of a numerical similarity vector 1050 is greater than or equal to a threshold. However, the conditions for identifying a speaker in stage 1055 are not limited to the aforementioned examples, and thus various conditions may be applied to identifying a speaker in stage 1055.

[0139] Fig.11 An example of a registration device 1100 for speaker recognition is shown. Fig.11 The registration device 1100 includes a processor 1110. The registration device 1100 also includes a memory 1130, a communication interface 1150, and a sensor 1170. The processor 1110, the memory 1130, the communication interface 1150, and the sensor 1170 can communicate with each other through a communication bus 1105.

[0140] The processor 1110 synthesizes the speech signal and the noise signal, generates a feature vector based on a synthesized signal obtained by synthesizing the speech signal and the noise signal, and constructs a registration DB corresponding to the speaker based on the generated feature vector.

[0141] The memory 1130 includes a registration DB including feature vectors. The memory 1130 may be a volatile memory or a nonvolatile memory.

[0142] The sensor 1170 includes, for example, an acoustic sensor configured to receive a speech signal of a speaker. The sensor 1170 may collect a collection of various sound information.

[0143] According to an example, the processor 1110 may generate a second feature vector based on a speech signal (e.g., a speech signal not synthesized with a noise signal), and construct a registration DB based on at least one of the feature vector and the second feature vector. The processor 1110 performs a domain transformation on the synthesized signal, and extracts a feature vector from the result of the domain transformation. The processor 1110 adds an additive noise signal to the speech signal. The processor 1110 performs a convolution operation on the speech signal and the channel noise signal. The processor 1110 divides the synthesized signal into a plurality of parts, and extracts at least one partial feature vector corresponding to at least one of the plurality of parts. The processor 1110 clusters a plurality of feature vectors including at least one of the feature vector and the second feature vector into a plurality of groups, and extracts at least one representative feature vector corresponding to at least one of the plurality of groups.

[0144] In addition, the processor 1110 may execute the above reference Figures 2 to 7 At least one method of the methods described (e.g., any one of the methods or any combination of any two or more methods) or an algorithm corresponding to at least one method. The processor 1110 can execute a program and control the registration device 1100. The code of the program to be executed by the processor 1110 can be stored in the memory 1130. The registration device 1100 can be connected to an external device (e.g., a personal computer (PC) and a network) through an input and output device (not shown), and can exchange data with the external device. The registration device 1100 can be set to various computing devices and / or systems (e.g., smart phones, tablet computers, laptop computers, desktop computers, TVs, wearable devices, security systems, smart home systems, etc.), or can be set in various computing devices and / or systems (e.g., smart phones, tablet computers, laptop computers, desktop computers, TVs, wearable devices, security systems, smart home systems, etc.).

[0145] According to an example embodiment, the recognition device may include a processor. The recognition device may also include a memory, a communication interface, and a sensor. The processor, the memory, the communication interface, and the sensor may communicate with each other via a communication bus. Thus, as a non-limiting example, the recognition device may have a processor and a communication interface. Fig.11 The configuration of the registration device 1100 shown in FIG. 1 is similar to or the same as the configuration of the registration device 1100 shown in FIG.

[0146] The processor may generate at least one input feature vector based on the voice signal, compare the input feature vector with at least one registered feature vector of a registered user stored in the registration DB, and identify a speaker based on a result of the comparison.

[0147] The memory may include a registration DB including feature vectors. The registration DB may include at least one registration feature vector generated based on a synthesis signal of a speech signal and a noise signal for registering a user. The memory may be a volatile memory or a non-volatile memory.

[0148] The sensor may include an acoustic sensor configured to receive a speaker's voice signal and collect a collection of various sound information.

[0149] According to one example, the processor may perform a domain transformation on the speaker's speech signal and extract an input feature vector from a result of the domain transformation. The processor may perform a sliding window operation on the speaker's speech signal so that the window size of the input feature vector is equal to the window size of the registration feature vector. The processor may perform an FFT on the speaker's speech signal and perform padding on a result of the FFT.

[0150] The identification device can be set to various computing devices and / or systems (e.g., smart phones, tablet computers, laptop computers, desktop computers, TVs, wearable devices, security systems, smart home systems, etc.), or can be set in various computing devices and / or systems (e.g., smart phones, tablet computers, laptop computers, desktop computers, TVs, wearable devices, security systems, smart home systems, etc.).

[0151] Fig.12 An example of a speaker verification method using multiple channels is shown. Fig.12 , the registration process and the test process can be performed through multiple channels.

[0152] The registration device may include a plurality of acoustic sensors, each of which includes a channel. The channel used herein indicates a channel through which information or a signal is to be transmitted, each channel may be independent of each other, and thus the signal of each channel may be independent without interference. Hereinafter, for ease of description, an example of an environment including two channels will be mainly described, but the example may also be applied to other environments including three or more channels.

[0153] Reference Fig.12 In the registration process, the acoustic sensor of the registration device receives the first voice signal 1210 to be registered and the second voice signal 1211 to be registered, respectively. The first voice signal 1210 and the second voice signal 1211 may be signals generated simultaneously from the same sound source. For example, the first voice signal 1210 may be a signal input through a microphone located at the lower end of the registration device, and the second voice signal 1211 may be a signal input through a microphone located at the upper end of the registration device.

[0154] The operations to be performed in the registration process using the multiple channels can be performed independently through the multiple channels, and the above reference Figure 8 The detailed operations described above for the registration process using a single channel are applicable to the registration process using multiple channels. Fig.12 As shown in , the registration device generates a registration feature vector of the first speech signal by synthesizing the first speech signal 1210 received through the microphone located at the lower end of the registration device with the additive noise 1215 and the channel noise 1220, performing segmentation on the synthesized signal in stage 1225, and then inputting the result of the segmentation performed in stage 1225 to the feature vector generator 1230. In the same manner, the registration device independently generates a registration feature vector of the second speech signal 1211 received through the microphone located at the upper end of the registration device. That is, the registration device generates a registration feature vector of the second speech signal 1211 by synthesizing the second speech signal 1211 received through the microphone located at the upper end of the registration device with the additive noise 1216 and the channel noise 1221, performing segmentation on the synthesized signal in stage 1226, and then inputting the result of the segmentation performed in stage 1226 to the second feature vector generator 1231.

[0155] The registration device extracts a representative registration feature vector 1240 based on the registration feature vector generated in each channel, and constructs a registration DB based on the extracted result. Since the registration feature vector is generated for each channel, more registration feature vectors can be generated. Here, the registration feature vector used here may indicate a feature vector to be registered, and the representative registration feature vector used here may indicate a feature vector to be registered that may represent a registration feature vector.

[0156] For example, the registration process is performed in the same manner, and the test process can be performed independently for each channel. For example, as shown in the figure, the registration device generates an input feature vector independently for each channel by receiving a first input voice signal 1245 via a microphone located at the lower end of the registration device and receiving a second input voice signal 1246 via a microphone located at the upper end of the registration device. In such an example, since an input feature vector is generated for each channel, more input feature vectors can be generated.

[0157] For example, the registration device receives a first input speech signal 1245, and in stage 1250 performs a sliding window operation on the first input speech signal 1245 to generate a first input feature vector 1260 using a feature vector generator 1255. In addition, for example, the registration device receives a second input speech signal 1246, and in stage 1251 performs a sliding window operation on the second input speech signal 1246 to generate a second input feature vector 1261 using a feature vector generator 1256.

[0158] In stage 1265, the enrollment device compares either or both of first input feature vector 1260 and second input feature vector 1261 with enrollment feature vector 1235 or representative enrollment feature vector 1240. In stage 1270, the enrollment device verifies the speaker based on the results of the comparison performed in stage 1265.

[0159] Fig.13 An example of a speaker verification method through channel selection is shown.

[0160] Reference Fig.13 , the registration device includes a plurality of acoustic sensors, each of which includes a channel. Hereinafter, although an example of an environment including two channels will be mainly described for ease of description, the example may also be applied to other environments including three or more channels. Figures 1 to 12 The description provided is applicable to Fig.13 Examples are given below, and for increased clarity and conciseness, repeated descriptions and detailed descriptions will be omitted.

[0161] The acoustic sensor of the registration device receives a first speech signal 1310 of the speaker and a second speech signal 1311 of the speaker respectively. The first speech signal 1310 and the second speech signal 1311 correspond to the registration speech signals. In stage 1320, the registration device preprocesses the received speech signals 1310 and 1311.

[0162] In one example, the registration device estimates noise based on the difference between the speech signals 1310 and 1311, and obtains a noise-removed signal by subtracting the noise from each of the speech signals 1310 and 1311. For example, a common portion in the speech signals 1310 and 1311 may have a high probability of corresponding to the voice of the speaker, and the difference between the speech signals 1310 and 1311 may have a high probability of corresponding to noise. Therefore, noise may be estimated based on the difference between the speech signals 1310 and 1311, and a noise-removed signal may be obtained by subtracting the noise from each of the speech signals 1310 and 1311 of the speaker.

[0163] In another example, the registration device selects a speech signal from the speech signals of the speaker received through multiple channels. When a speech signal of an allowable input size or larger is input to an acoustic sensor of a registration device that allows limited input, recognition errors (e.g., clipping) may occur due to the hardware specifications of the registration device. For example, such clipping may occur when a speech signal is input from a position very close to the acoustic sensor or when a speech signal is input through very loud speech. In a speaker recognition method using a single channel, recognition errors may occur due to such clipping. However, the registration device may select a speech signal from the speech signals of the speaker received through multiple channels for each portion, thereby reducing such errors.

[0164] For example, when a voice signal is input from a position close to the acoustic sensor located at the lower end of the registration device, the voice signal 1310 received from the acoustic sensor located at the lower end of the registration device may be clipped, and the voice signal 1311 received from the acoustic sensor located at the upper end of the registration device may be input normally. In this example, the voice signal 1311 received by the acoustic sensor located at the upper end of the registration device, which does not undergo clipping in the stage 1320 of performing preprocessing, may be selected. Therefore, the clipping problem may be solved.

[0165] In addition, the registration device may select a voice signal from among the voice signals of the speaker received through multiple channels for each part of the signal. For example, the registration device may select voice signal 1310 in the first part and voice signal 1311 in the second part.

[0166] For by Fig.13 The subsequent operations indicated by reference numerals 1330 to 1380 in Figure 2 A description of the operation of the registration system for speaker identification indicated by reference numerals 220 to 270 in FIG.

[0167] Fig.14 An example of a method that is robust to utterance changes due to the position and orientation of the registration device 10 is shown.

[0168] When the registration device receives a speech signal through a single channel, a recognition error may occur because of a change in speech due to the reverberation characteristics of the registration device. For example, in the registration process 1400, the acoustic sensor 14 of the registration device 10 is located in an open space, so there is no reverberation due to the table 20. In the test process 1450, the acoustic sensor 14 is located on the table 20, and therefore there is reverberation. In this example, a recognition error may occur because of a change in speech due to the reverberation characteristics of the registration device 10.

[0169] However, in the case where the registration device 10 adds the registration feature vector using multiple channels, the registration device 10 may improve the speaker recognition rate by applying the change of speech based on the direction of the registration device 10. For example, in the registration process 1400, the acoustic sensor 14 arranged at the lower end of the registration device is located in an open space and the acoustic sensor 16 arranged at the upper end of the registration device is located on the table 20, so the registration device 10 may perform registration in the same environment as the environment in which the test process 1450 is performed. Therefore, when the registration device 10 uses multiple channels, the change of speech based on the reverberation characteristics of the registration device 10 may be considered and applied.

[0170] In addition, although the registration device 10 uses a plurality of channels, in an environment where the plurality of channels are processed in parallel (eg, a multi-threaded environment), a speaker recognition rate may be increased without an additional time delay in speaker recognition.

[0171] Perform the operations described in this application Figure 2 , Figures 4 to 6 , Figure 8 as well as Figures 10 to 13The preprocessors 430 and 530, the neural networks 440 and 540, the feature vector generators 260, 630, 830, 1015, 1035, 1230, 1231, 1255, 1256 and 1370, the processor 1110 and the memory 1130 in the present invention are implemented by hardware components configured to perform the operations performed by the hardware components described in the present application. Examples of hardware components that can be used to perform the operations described in the present application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in the present application. In other examples, one or more of the hardware components that perform the operations described in the present application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer can be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond and execute instructions in a limited manner to achieve the desired result). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or computer can execute instructions or software (such as operating systems (OS) and one or more software applications running on OS) for performing the operations described in this application. The hardware components can also access, manipulate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or a processor or computer can include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may implement a single hardware component or two or more hardware components. The hardware components may have any one or more of different processing configurations, examples of which include: a single processor, an independent processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, a multiple instruction single data (MISD) multiprocessing, and a multiple instruction multiple data (MIMD) multiprocessing.

[0172] Figures 1 to 10 and Figure 12 to Figure 14 The method for performing the operations described in the present application shown in the embodiment is performed by computing hardware (e.g., by one or more processors or computers), which is implemented as instructions or software as described above to perform the operations performed by the method described in the present application. For example, a single operation or two or more operations may be performed by a single processor or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may perform a single operation or two or more operations.

[0173] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform methods as described above may be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as a machine or special-purpose computer to perform operations performed by the hardware components and methods as described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) directly executed by one or more processors or computers. In another example, the instructions or software include high-level code executed by one or more processors or computers using an interpreter. Instructions or software may be written in any programming language based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification, wherein the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification disclose algorithms for performing operations performed by the hardware components and methods as described above.

[0174] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, and any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access memory (RAM), flash memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state disk, and any other device, wherein any other device is configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers so that one or more processors and computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed across a networked computer system so that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0175] Although the present disclosure includes specific examples, it will be clear after understanding the disclosure of the present application that various changes in form and detail can be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are considered to be descriptive only, not for the purpose of limitation. The description of the features or aspects in each example will be considered to be applicable to similar features or aspects in other examples. If the described techniques are performed in a different order, and / or if the components in the described systems, architectures, devices, or circuits are combined in different ways and / or replaced or supplemented by other components or their equivalents, appropriate results can be achieved. Therefore, the scope of the disclosure is not limited by specific embodiments, but by the claims and their equivalents, and all changes within the scope of the claims and their equivalents will be interpreted as included in the disclosure.

Claims

1. A registration method, comprising: receiving a speech signal of a speaker through a plurality of channels; estimating noise based on a difference between received speech signals; subtracting the noise from each of the received speech signals; synthesizing the received speech signal and the noise signal to generate a synthesized signal; generating a feature vector based on the synthesized signal; and Based on the generated feature vector, a registration database corresponding to the speaker is constructed. The steps of synthesizing and generating a feature vector are performed for each speech signal in the received speech signal from which the noise is subtracted, and the step of constructing a registration database includes constructing a registration database based on the feature vectors generated through the multiple channels.

2. The registration method according to claim 1, further comprising: generating a second feature vector based on the received speech signal, The step of constructing a registration database includes: constructing a registration database based on any one or both of the feature vector and the second feature vector.

3. The registration method according to claim 2, wherein: The steps to construct a registration database include: clustering a plurality of feature vectors including any one or both of the feature vector and the second feature vector into a plurality of groups; and At least one representative feature vector corresponding to at least one of the plurality of groups is extracted.

4. The registration method according to claim 2, wherein: The second feature vector is based on the received speech signal without synthesis.

5. The registration method according to claim 1, wherein: The steps to generate a feature vector include: performing a domain transform on the composite signal; and Extract feature vectors from the result of the domain transformation.

6. The registration method according to claim 5, wherein: The synthesized signal includes time domain sound information, and the result of the domain transformation includes frequency domain image information, The step of generating a feature vector includes: extracting a feature vector from frequency domain image information using a convolutional neural network.

7. The registration method according to claim 1, wherein: The step of synthesizing includes adding a noise signal to the received speech signal, the noise signal including an additive noise signal.

8. The registration method according to claim 7, wherein: The additive noise signal is determined based on any one or any combination of any two or more of a type of noise, a timing of the noise, and a signal-to-noise ratio indicating an energy ratio of the noise to the speech signal.

9. The registration method according to claim 1, wherein: The step of synthesizing includes performing a convolution operation on the received speech signal and a noise signal, wherein the noise signal includes a channel noise signal.

10. The registration method according to claim 9, wherein: The channel noise signal is determined based on at least one of a type of noise and a signal-to-noise ratio indicating an energy ratio of the noise to the speech signal.

11. The registration method according to claim 1, wherein: The steps to generate a feature vector include: splitting the composite signal into a plurality of parts; and At least one partial feature vector corresponding to at least one of the plurality of parts is extracted.

12. The registration method according to claim 1, further comprising: extracting the starting point and the ending point of the sound information included in any one or both of the speech signal and the synthesized signal, The starting point and the ending point are applied to any one or both of the steps of synthesizing and generating the feature vector.

13. The registration method according to claim 1, wherein: The registration method further includes: determining a speech signal for synthesis by preprocessing the received speech signal.

14. The registration method according to claim 13, wherein: The step of determining the voice signal includes selecting a voice signal from the received voice signals based on whether a recognition error due to hardware specifications is included in the received voice signals.

15. The registration method according to claim 13, wherein: The step of determining the voice signal includes selecting a voice signal from the received voice signals for each of a plurality of parts of the received voice signal based on whether a recognition error due to hardware specifications is included in each part of the received voice signal.

16. The registration method according to claim 1, further comprising: The speaker is identified by comparing the input speech signal with the generated feature vector. 17 . A non-transitory computer-readable storage medium storing instructions, wherein when the instructions are executed by a processor, the processor is caused to perform the registration method of claim 1 .

18. A method for speaker identification, comprising: receiving a speech signal of a speaker through a plurality of channels; estimating noise based on a difference between received speech signals; subtracting the noise from each of the received speech signals; generating at least one input feature vector based on the received speech signal; comparing the at least one input feature vector to at least one registered feature vector for a registered user stored in a registration database; and Based on the comparison results, the speaker is identified. wherein the at least one registration feature vector is generated based on a synthesis signal of a voice signal and a noise signal used to register a registered user, The step of generating the at least one input feature vector and the step of comparing are performed for each speech signal in the received speech signal from which the noise is subtracted, and the at least one registered feature vector comprises: a registered feature vector generated through the multiple channels.

19. The speaker recognition method according to claim 18, wherein: The registered users include a plurality of registered users, and the step of identifying the speaker includes recognizing the speaker from among the plurality of registered users.

20. The speaker recognition method according to claim 18, wherein: The step of generating the at least one input feature vector comprises: performing a domain transform on the received speech signal; and The input feature vector is extracted from the result of the domain transformation.

21. The speaker recognition method according to claim 20, wherein: The steps to perform a domain transformation include: performing a fast Fourier transform on the received speech signal; and The padding is performed on the result of the fast Fourier transform based on the result of the fast Fourier transform.

22. The speaker recognition method according to claim 18, wherein: The step of generating the at least one input feature vector comprises: A sliding window operation is performed on the received speech signal so that the window size of the input feature vector is equal to the window size of the registration feature vector.

23. The speaker recognition method according to claim 18, wherein: The number of the at least one input feature vector is determined based on a security level or a response time required by an application.

24. The speaker recognition method according to claim 18, wherein: The noise signal includes any one or both of an additive noise signal and a channel noise signal.

25. The speaker recognition method according to claim 18, wherein: The speaker recognition method further includes: determining a speech signal for synthesis by preprocessing the received speech signal.

26. The speaker recognition method according to claim 25, wherein: The step of determining the voice signal includes selecting a voice signal from the received voice signals based on whether a recognition error due to hardware specifications is included in the received voice signals.

27. The speaker recognition method according to claim 25, wherein: The step of determining the voice signal includes selecting a voice signal from the received voice signals for each of a plurality of parts of the received voice signal based on whether a recognition error due to hardware specifications is included in each part of the received voice signal.

28. The speaker recognition method according to claim 18, further comprising: Before receiving the speech signal, the at least one enrollment feature vector is generated based on the synthesized signal and the at least one enrollment feature vector is stored in an enrollment database.

29. A non-transitory computer-readable storage medium storing instructions, wherein when the instructions are executed by a processor, the processor is caused to perform the speaker recognition method of claim 18.

30. A registration device, comprising: an acoustic sensor configured to receive a speech signal of a speaker through a plurality of channels; and a processor configured to estimate noise based on a difference between received speech signals, subtract the noise from each of the received speech signals, synthesize the received speech signals with the noise signal to generate a synthesized signal, generate a feature vector based on the synthesized signal, and construct a registration database corresponding to the speaker based on the generated feature vector, The processor is configured to: perform a synthesis step and a feature vector generation step for each of the received speech signals from which the noise is subtracted, and construct a registration database based on the feature vectors generated through the multiple channels.

31. A registration device according to claim 30, wherein: The processor is further configured to generate a second feature vector based on the received speech signal, and construct a registration database based on either or both of the feature vector and the second feature vector.

32. A registration device according to claim 31, wherein: The processor is further configured to cluster a plurality of feature vectors including any one or both of the feature vector and the second feature vector into a plurality of groups, and extract at least one representative feature vector corresponding to at least one of the plurality of groups.

33. The registration device according to claim 30, wherein: The second feature vector is based on the received speech signal without synthesis.

34. The registration device according to claim 30, wherein: The processor is further configured to perform a domain transform on the synthesized signal and extract a feature vector from a result of the domain transform.

35. The registration device according to claim 30, wherein: The processor is further configured to add a noise signal to the received speech signal, wherein the noise signal comprises an additive noise signal.

36. The registration device according to claim 30, wherein: The processor is further configured to perform a convolution operation on the received speech signal and the noise signal, wherein the noise signal includes a channel noise signal.

37. The registration device according to claim 30, wherein: The processor is further configured to segment the synthesized signal into a plurality of parts and extract at least one partial feature vector corresponding to at least one of the plurality of parts.

38. The registration device according to claim 30, wherein: The processor is also configured to identify a speaker by comparing the input speech signal to the generated feature vector.

39. A device for speaker recognition, comprising: an acoustic sensor configured to receive a speech signal of a speaker through a plurality of channels; and a processor configured to estimate noise based on a difference between received speech signals, subtract the noise from each of the received speech signals, generate at least one input feature vector based on the received speech signals, compare the at least one input feature vector with at least one registered feature vector of a registered user stored in a registration database, and identify a speaker based on a result of the comparison, wherein the at least one registration feature vector is generated based on a synthesis signal of a voice signal and a noise signal used to register a registered user, The step of generating the at least one input feature vector and the step of comparing are performed for each speech signal in the received speech signals from which the noise is subtracted, and the at least one registered feature vector comprises: a registered feature vector generated through the multiple channels.

40. The apparatus of claim 39, wherein: The processor is further configured to perform a domain transformation on the received speech signal and extract an input feature vector from a result of the domain transformation.

41. The apparatus of claim 39, wherein: The processor is further configured to perform a sliding window operation on the received speech signal so that a window size of the input feature vector is equal to a window size of the registration feature vector.

42. The apparatus of claim 39, wherein: The processor is further configured to: performing a fast Fourier transform on the received speech signal; and The padding is performed on the result of the fast Fourier transform based on the result of the fast Fourier transform.

43. The apparatus of claim 39, wherein: The processor is further configured to: before receiving the speech signal, generate the at least one enrollment feature vector based on the synthesized signal, and store the at least one enrollment feature vector in an enrollment database.

Images