Method and device for processing human voice data of speech
By extracting feature vectors from voice data and generating feature maps, the problem of storage and registration restrictions in the prior art is solved, and the effect of providing personalized services and relationship prediction without registration is achieved.
Patent Information
- Application Number
- CN201980076729.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-11-22
- Filing Date
- 2019-11-22
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2039-11-22
AI Technical Summary
The prior art requires storing user's vocal data and registering identification information before using the service in a multi-user environment, which limits the personalization and convenience of the service and makes it difficult to predict the relationship and status between users.
By extracting feature vectors from the received voice data, generating a speaker feature map, and classifying and relation prediction of the speaker based on the positional relationship on the feature map, personalized services are achieved without registering user identification information.
It improves user convenience, can provide personalized services to each speaker without registering user identification information, and accurately predicts the relationships and status between users.
Smart Images

Figure CN113168837B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to a method and apparatus for processing vocal data of a speech received from a user. Background Art
[0002] With the development of multimedia technology and network technology, various services can be provided to users by using devices. Recently, with the development of artificial intelligence (AI) technology, users can input human voice to the device and perform the operation of the device based on the input of the human voice.
[0003] In the case where multiple users use a multi-user device, in order to provide personalized services based on the human voice input of each user, it is necessary to identify the user from the voice. According to the related art, a technology is used, whereby after the user's voice is input and stored in the device and the identification information of the user corresponding to the stored voice is registered, user identification is performed based on the matching probability of the comparison between the voice input from the user and the pre-registered voice. However, this technology requires the process of storing the user's human voice and registering the identification information before using the service. In addition, according to the related art, there is a limitation in predicting the relationship between multiple users or the current status of the user based on the input voice.
[0004] The above information is presented as background information only to assist with an understanding of the present disclosure. No determination has been made, and no assertion is made, as to whether any of the above might be applicable as prior art with respect to the present disclosure. Summary of the invention
[0005] Aspects of the present disclosure are to at least solve the above-mentioned problems and / or disadvantages, and to provide at least the advantages described below. Therefore, one aspect of the present disclosure is to provide a method and apparatus for processing vocal data of a speech received from a speaker. Specifically, the present disclosure will provide a method and apparatus for extracting feature vectors from vocal data of a speech, generating a speaker feature map based on the extracted feature vectors, and classifying speakers based on the positional relationship of the feature vectors on the generated speaker feature map.
[0006] Another aspect of the present disclosure is to provide a method and apparatus for predicting a relationship between users and status information of the users based on a positional relationship of feature vectors on a speaker feature graph.
[0007] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments of the disclosure.
[0008] Beneficial Effects
[0009] According to the embodiment of the present disclosure, multiple speakers are classified by using only input voices without additionally registering user identification information. Therefore, according to the embodiment of the present disclosure, personalized services are provided for each speaker when the user is not registered, thereby improving the convenience of use. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which:
[0011] Figure 1 is a diagram illustrating an example of a method of generating a speaker characteristic graph by processing vocal data of a speech input from a speaker and classifying a plurality of speakers on the speaker characteristic graph, performed by an electronic device according to an embodiment of the present disclosure;
[0012] Figure 2A is a block diagram of components of an electronic device according to an embodiment of the present disclosure;
[0013] Figure 2B is a block diagram of components of an electronic device and a server according to an embodiment of the present disclosure;
[0014] Figure 3A is a flowchart of a method for processing vocal data of a speech input from a speaker, performed by an electronic device according to an embodiment of the present disclosure;
[0015] Figure 3B is a flowchart of an operating method of a system including an electronic device and a server according to an embodiment of the present disclosure;
[0016] Figure 3C is a flowchart of an operating method of a system including an electronic device and a server according to an embodiment of the present disclosure;
[0017] Figure 3D is a flowchart of an operating method of a system including an electronic device and a server according to an embodiment of the present disclosure;
[0018] Figure 4A is a diagram for describing a method for generating a speaker feature map by processing vocal data input from a speaker, which is performed by an electronic device according to an embodiment of the present disclosure;
[0019] Figure 4B is a diagram for describing a method of generating a speaker feature map by processing vocal data input from a speaker, which is performed by an electronic device according to an embodiment of the present disclosure;
[0020] Figure 4C is a diagram for describing a method of generating a speaker feature map by processing vocal data input from a speaker, which is performed by an electronic device according to an embodiment of the present disclosure;
[0021] Figure 4D is a flowchart of a method for generating a speaker characteristic map by processing vocal data of speech input from a plurality of speakers, performed by an electronic device according to an embodiment of the present disclosure;
[0022] Figure 5 is a diagram illustrating a method of converting a human voice feature vector into a speaker feature vector by using an artificial intelligence (AI) technology, performed by an electronic device according to an embodiment of the present disclosure;
[0023] Fig. 6A is a diagram illustrating a method of registering user identification information corresponding to clusters formed on a speaker feature graph, performed by an electronic device according to an embodiment of the present disclosure;
[0024] Figure 6B is a diagram illustrating a method of registering user identification information corresponding to clusters formed on a speaker feature graph, performed by an electronic device according to an embodiment of the present disclosure;
[0025] Figure 6C is a diagram illustrating a method of registering user identification information corresponding to clusters formed on a speaker feature graph, performed by an electronic device according to an embodiment of the present disclosure;
[0026] Fig. 7A is a diagram illustrating a method of generating a distinguishing sentence for identifying a speaker on a speaker characteristic graph based on a frequency feature value of each phoneme of speech and outputting the generated distinguishing sentence according to an embodiment of the present disclosure;
[0027] Figure 7B is a diagram illustrating a method of generating a distinguishing sentence for identifying a speaker on a speaker characteristic graph based on a frequency feature value of each phoneme of speech and outputting the generated distinguishing sentence according to an embodiment of the present disclosure;
[0028] Figure 7C is a flowchart of a method of generating a distinguishing sentence for identifying a speaker on a speaker characteristic graph and outputting a message requesting a voice of the distinguishing sentence, performed by an electronic device according to an embodiment of the present disclosure;
[0029] Figure 8 is a block diagram of components of a virtual assistant module included in an electronic device according to an embodiment of the present disclosure;
[0030] Fig. 9 is a diagram illustrating a method for predicting a relationship between a plurality of speakers based on a positional relationship between a plurality of clusters on a speaker feature graph, performed by an electronic device according to an embodiment of the present disclosure;
[0031] Fig. 10Ais a diagram illustrating a method performed by an electronic device according to an embodiment of the present disclosure of predicting a relationship between a plurality of speakers based on a positional relationship between a plurality of clusters on a speaker feature graph and providing a personalized service based on the predicted relationship;
[0032] Fig. 10B is a diagram illustrating a method performed by an electronic device according to an embodiment of the present disclosure of predicting a relationship between a plurality of speakers based on a positional relationship between a plurality of clusters on a speaker feature graph and providing a personalized service based on the predicted relationship;
[0033] Fig.11 is a flowchart of a method for predicting a relationship between multiple speakers based on a positional relationship between multiple clusters on a speaker feature graph and providing a personalized service based on the predicted relationship, performed by an electronic device according to an embodiment of the present disclosure;
[0034] Fig. 12A is a diagram illustrating a method for predicting a user state based on a positional relationship between a plurality of speaker feature vectors on a speaker feature graph, performed by an electronic device according to an embodiment of the present disclosure;
[0035] Fig. 12B is a diagram illustrating a method for predicting a user state based on a positional relationship between a plurality of speaker feature vectors on a speaker feature graph, performed by an electronic device according to an embodiment of the present disclosure;
[0036] Fig.13 is a flowchart of a method for predicting a user state based on a positional relationship between a plurality of speaker feature vectors on a speaker feature graph, performed by an electronic device according to an embodiment of the present disclosure; and
[0037] Fig.14 is a block diagram of components of an electronic device according to an embodiment of the present disclosure.
[0038] Throughout the drawings, like reference numerals will be understood to refer to like parts, components and structures. DETAILED DESCRIPTION
[0039] According to one aspect of the present disclosure, a method for processing vocal data of speech is provided. The method includes: extracting a speaker feature vector from the vocal data of speech received from a speaker, generating a speaker feature graph by locating the extracted speaker feature vector at a specific position on a multidimensional vector space, forming a plurality of clusters indicating features of speech of a plurality of speakers by grouping at least one speaker feature vector located on the speaker feature graph, and classifying the plurality of speakers according to the formed plurality of clusters.
[0040] The step of generating the speaker feature map may include: training the DNN model by applying a vocal feature vector extracted from vocal data as input to the DNN model and using feature values indexed for the speaker as output, and extracting the speaker feature vector by using a result of the training.
[0041] The method may further include: receiving speech for inputting recognition information from a user, extracting a speaker feature vector from vocal data of the received speech, mapping the extracted speaker feature vector to a cluster among the multiple clusters on the speaker feature map, and registering the recognition information of the user to the mapped cluster among the multiple clusters.
[0042] The method may further include, before receiving identification information from the user, requesting the user to speak a distinguishing sentence for registering the identification information of the user.
[0043] The distinguishing sentences may include phonemes, wherein a difference in cumulative frequency feature values between a plurality of speakers corresponding to a plurality of pre-registered clusters for the phonemes is equal to or greater than a predetermined critical value.
[0044] The method may further include analyzing a cumulative degree of a frequency feature value of each phoneme with respect to a speech received from a speaker, and generating a distinguishing sentence including phonemes having a relatively low cumulative degree or a zero cumulative degree based on a result of the analysis.
[0045] The forming of the plurality of clusters may include forming the plurality of clusters by grouping the at least one speaker feature vector located within a predetermined distance from each other on the speaker feature map.
[0046] The step of forming the plurality of clusters may include training the DNN model by applying a speaker feature vector as an input value to the DNN model and using a similarity index between speakers as an output value, and forming the plurality of clusters by grouping at least one similar speaker feature vector based on the training.
[0047] The method may further include predicting a relationship between the plurality of speakers corresponding to the plurality of clusters based on distance and directionality information between the plurality of clusters on the speaker feature map.
[0048] The method may further include identifying a speaker of the received voice based on the predicted relationship between the speakers, and providing a personalized service synchronized with the identified speaker.
[0049] The method may further include predicting a state of a speaker based on a relative position relationship between a plurality of speaker feature vectors in one cluster of the plurality of clusters.
[0050] The step of predicting the state of the speaker may include: identifying directionality by comparing the position of a first feature vector of the speech received from the user with the position of a second feature vector in the cluster to which the first feature vector is mapped on the speaker feature map, and predicting the health state of the speaker based on the identified directionality.
[0051] The method may further include outputting a message corresponding to the predicted health status of the user from pre-stored messages.
[0052] According to another aspect of the present disclosure, an electronic device for processing human voice data of speech is provided. The electronic device includes: a speech input device that receives speech from a speaker; a controller that is configured to extract a speaker feature vector from the human voice data of the speech received by the speech input device, generate a speaker feature map by locating the extracted speaker feature vector at a specific position on a multidimensional vector space, form a plurality of clusters indicating features of human voices of a plurality of speakers by grouping at least one speaker feature vector located on the speaker feature map, and classify the plurality of speakers according to the formed plurality of clusters; and a memory that stores the speaker feature map.
[0053] The controller may also be configured to train the DNN model by applying a vocal feature vector extracted from vocal data as input to the DNN model and using feature values of an index about a speaker as output, and extract a speaker feature vector by using a result of the training.
[0054] The voice inputter can receive voice for inputting identification information from a user, and the controller can also be configured to: extract a speaker feature vector from the vocal data of the received voice, map the extracted speaker feature vector to a cluster among the multiple clusters on the speaker feature map, and register the user's identification information to the mapped cluster among the multiple clusters.
[0055] The electronic device may further include a speaker to output a voice message, and the controller may be further configured to control the speaker to output the voice message for requesting the user to speak a distinguishing sentence for registering identification information of the user.
[0056] The controller may be further configured to generate a distinction sentence including phonemes for which a difference in cumulative frequency feature values between a plurality of speakers corresponding to a plurality of pre-registered clusters is equal to or greater than a predetermined critical value.
[0057] The controller may be further configured to analyze a cumulative degree of a frequency feature value of each phoneme for the speech received from the speaker, and generate a distinguishing sentence including phonemes having a relatively low cumulative degree or a zero cumulative degree based on a result of the analysis.
[0058] The controller may be further configured to form the plurality of clusters by grouping the at least one speaker feature vector located within a predetermined distance from each other on the speaker feature map.
[0059] The controller may also be configured to train the DNN model by applying a speaker feature vector as an input value to the DNN model and using a similarity index between speakers as an output value, and to form the plurality of clusters by grouping at least one similar speaker feature vector based on the training.
[0060] The controller may be further configured to predict a relationship between the plurality of speakers corresponding to the plurality of clusters based on distance and directionality information between the plurality of clusters on the speaker feature map.
[0061] The electronic device may further include a display, and the controller may be further configured to identify a speaker of the received voice based on the predicted relationship between the speakers, and display a personalized service synchronized with the identified speaker on the display.
[0062] The controller may also be configured to identify directionality by comparing the position of a first feature vector of speech received by the speech inputter with the position of a second feature vector in the cluster to which the first feature vector is mapped on the speaker feature map, and predict the user's health status based on the identified directionality.
[0063] The electronic device may further include a speaker to output a voice message, and the controller may be further configured to control the speaker to output a message corresponding to the predicted health status of the user from among pre-stored messages.
[0064] According to another aspect of the present disclosure, a computer program product including a non-transitory computer-readable recording medium is provided. The computer program product includes instructions for controlling an electronic device to perform the following operations: receiving speech from a speaker, extracting a speaker feature vector from the human voice data of the received speech, generating a speaker feature map by locating the extracted speaker feature vector at a specific position on a multidimensional vector space, forming a plurality of clusters indicating features of human voices of a plurality of speakers by grouping at least one speaker feature vector located on the speaker feature map, and classifying the plurality of speakers according to the formed plurality of clusters.
[0065] Other aspects, advantages, and salient features of the present disclosure will become apparent to those skilled in the art from the following detailed description, which, taken in conjunction with the accompanying drawings, discloses various embodiments of the present disclosure.
[0066] This application is based on and claims the benefit of priority from Korean Patent Application No. 10-2018-0145644 filed on November 22, 2018 in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference.
[0067] The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of the various embodiments of the present disclosure as defined by the claims and their equivalents. It includes various specific details to assist in understanding, but these details are considered to be exemplary only. Therefore, it will be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, descriptions of well-known functions and structures may be omitted for clarity and conciseness.
[0068] The terms and words used in the following description and claims are not limited to the bibliographical meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustrative purposes only and not for the purpose of limiting the present disclosure as defined by the appended claims and their equivalents.
[0069] It should be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component surface" includes reference to one or more of such surfaces.
[0070] Throughout the disclosure, the expression "at least one of a, b, or c" means only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.
[0071] Throughout the specification, when a component "includes" an element, it will be understood that the component additionally includes other elements rather than excluding other elements, as long as there is no specific contrary description. In addition, the terms described in the specification (such as "unit", "module", etc.) represent a unit that processes at least one function or operation, which can be implemented as hardware or software or a combination thereof.
[0072] Hereinafter, the embodiments of the present disclosure will now be described more fully with reference to the accompanying drawings so that those skilled in the art can perform the present disclosure without difficulty. However, the present disclosure can be implemented in many different forms and should not be construed as being limited to the embodiments of the present disclosure set forth herein.
[0073] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.
[0074] Figure 1is a diagram illustrating a method of generating a speaker characteristic map by processing vocal data of a speech received from a speaker and classifying a plurality of speakers on the speaker characteristic map, performed by an electronic device according to an embodiment of the present disclosure.
[0075] Reference Figure 1 , the electronic device 10 may receive a voice from the speaker 1. The electronic device 10 may recognize a human voice input from the speaker 1, and receive a "voice" corresponding to the input human voice, which does not include a non-human voice part. The language spoken by the speaker 1 may be referred to as "spoken language", and the spoken language may correspond to a predetermined instruction. However, the present invention is not limited thereto. The spoken language may correspond to a natural language, wherein the natural language is a language commonly used by the speaker 1.
[0076] The electronic device 10 may extract the vocal data 20 from the speech. In this case, the electronic device 10 may detect an actual vocal portion by detecting the start and end points of the vocal input from the speaker 1, and may extract the vocal data 20 from the detected actual vocal portion.
[0077] The vocal feature may be indicated as a vocal feature vector. In detail, the electronic device 10 may extract the vocal feature vector of the input vocal data 20 by applying the feature vector extraction technology to the vocal data 20. According to an embodiment of the present disclosure, the electronic device 10 may extract the vocal feature value of the input vocal data 20 by using any one of the feature vector extraction technologies including cepstrum, linear prediction coefficient (LPC), mel-frequency cepstrum coefficient (MFCC) and filter bank energy. The described feature vector extraction technology is only an example, and the feature vector extraction technology used in the present disclosure is not limited to the described example.
[0078] The extracted vocal feature vector may be an n-dimensional vector, where n is equal to or greater than 2. In addition, the extracted vocal feature vector may include at least two vector columns.
[0079] The speaker feature may be indicated as a speaker feature vector. In detail, the electronic device 10 may extract a speaker feature vector by applying a deep neural network (DNN) model to the vocal feature vector extracted from the vocal data 20. The electronic device 10 may train a DNN model by applying the vocal feature vector as an input value to the DNN model and applying the speaker feature value as an output value to the DNN model to obtain a speaker feature vector. Here, the speaker index may represent a value obtained by converting speaker recognition information into a vector value using at least one of one-hot encoding, word2vec, and GloVe.
[0080] The DNN model may include at least one of a convolutional neural network (CNN), a recurrent neural network (RNN), or a generative adversarial network (GAN). However, the DNN model is not limited to the above examples. The DNN model used by the electronic device 10 according to the present disclosure may include all types of DNN models currently known.
[0081] According to an embodiment of the present disclosure, the vocal feature vector input to the DNN model may be an n-dimensional vector, where n is equal to or greater than 2. In addition, the vocal feature vector input to the DNN model may include at least two vector columns. The speaker feature vector extracted by training the DNN model may be an n-dimensional vector, where n is equal to or greater than 2.
[0082] According to another embodiment of the present disclosure, a speaker feature vector may be indicated as an i-vector (which is a feature vector). In detail, when the distribution of sound parameters extracted from vocal data is modeled by using a Gaussian mixture model (GMM), information obtained by subtracting a global background model (UBM) from a GMM supervector generated by connecting average Gaussian values may be indicated as the product of an i-vector and a total variable transformation matrix. Here, the UBM may be a value obtained by using a large amount of data indicating vocal features of a plurality of various speakers, and the total variable transformation matrix may be a value determined based on a speaker recognition model (e.g., GMM, etc.), and therefore, the electronic device 10 may obtain an i-vector by obtaining a vocal feature vector from the vocal data of the speaker.
[0083] The electronic device 10 may generate a speaker feature map 30 that displays speaker feature vectors at specific locations in an n-dimensional virtual space. N-dimensional speaker feature vectors 31, 32, and 33 extracted from vocal data 20 input by a plurality of speakers may be located at specific locations of the speaker feature map 30, wherein the specific locations correspond to feature vector values. Figure 1 3 shows that the speaker feature graph 30 and the speaker feature vectors 31, 32, and 33 are all three-dimensional. However, this is for the convenience of explanation. The speaker feature graph 30 and the speaker feature vectors 31, 32, and 33 according to the present disclosure are not limited thereto and may be n-dimensional, where n is equal to or greater than 2.
[0084] The electronic device 10 may form a plurality of clusters 41, 42, and 43 based on the relative positional relationship between the speaker feature vectors 31, 32, and 33 located on the generated speaker feature graph 30. Figure 1As shown, speaker feature vectors 31, 32, and 33 may be located at positions of the speaker feature graph 30, respectively, wherein the positions correspond to their speaker feature vector values, respectively, and speaker feature vectors located within a predetermined distance may form a spherical group. According to an embodiment of the present disclosure, the electronic device 10 may form a plurality of clusters 41, 42, and 43 by grouping a plurality of speaker feature vectors located within a predetermined distance. That is, the electronic device 10 may form each of clusters 41, 42, and 43 by combining speaker feature vectors clustered within a predetermined distance on the speaker feature graph 30 into a group. Here, "clusters 41, 42, and 43" may be groups of speaker feature vectors having the same vocal features, and are concepts derived from the following aspects: speaker feature vectors having similar features of vocal data may be located relatively adjacent to each other on the speaker feature graph 30, and speaker feature vectors having dissimilar features of vocal data may be located relatively far away from each other on the speaker feature graph.
[0085] The electronic device 10 may classify a plurality of speakers according to the plurality of clusters 41, 42, and 43 formed on the speaker feature graph 30. For example, the electronic device 10 may classify a voice having a speaker feature vector 31 mapped in the first cluster 41 as a first user, a voice having a speaker feature vector 32 mapped in the second cluster 42 as a second user, and a voice having a speaker feature vector 33 mapped in the third cluster 43 as a third user.
[0086] According to an embodiment of the present disclosure, the electronic device 10 may cluster the speaker feature vectors on the speaker feature map 30 according to each speaker by applying a DNN model. For example, the electronic device 10 may classify the speaker feature vectors on the speaker feature map according to each speaker by applying the speaker feature vector as an input value to the DNN and applying the similarity index between speakers as an output value to the DNN model to train the DNN model. Here, for example, when the speakers are the same person, the similarity index between speakers may be indicated as 1, and when the speakers are not the same person, the similarity index between speakers may be indicated as a value converted to 0. The DNN model used and trained by the electronic device 10 may include at least one of a CNN, an RNN, or a GAN, but is not limited to the described examples.
[0087] According to an embodiment of the present disclosure, the electronic device 10 may predict the relationship between the plurality of speakers respectively corresponding to the plurality of clusters 41, 42, and 43 based on the distance and directionality information of the plurality of clusters 41, 42, and 43 formed on the speaker feature graph 30. According to another embodiment of the present disclosure, the electronic device 10 may predict the relationship between the plurality of speakers by using a DNN model. Fig. 9and FIG. 10A to FIG. 10B According to an embodiment of the present disclosure, the electronic device 10 can predict the state of the speaker based on the positional relationship of multiple speaker feature values in any one of the multiple clusters 41, 42 and 43 on the speaker feature graph 30. Fig. 12A , Fig. 12B and Fig.13 Describe this aspect in detail.
[0088] When multiple users use the electronic device 10, it is necessary to identify the user from the voice in order to provide personalized services based on the user's voice input. According to the related art, a technology is used whereby after the electronic device 10 pre-inputs and stores the voices of multiple users and registers user identification information corresponding to the stored voices, the electronic device 10 performs user identification based on a matching probability by comparing the voice input from the user with the pre-registered voice. However, this technology requires the process of storing the user's voice and registering the identification information before using the service.
[0089] The electronic device 10 according to an embodiment of the present disclosure can classify multiple speakers by using only input voices without additionally registering user identification information. Therefore, when a user is not registered, the electronic device 10 according to the present disclosure can provide personalized services for each speaker, thereby improving the convenience of use.
[0090] Figure 2A is a block diagram of components of an electronic device according to an embodiment of the present disclosure. The electronic device may be a fixed type terminal or a mobile terminal implemented as a computer device. The electronic device may include, for example, at least one of a smart phone, a cellular phone, a navigation device, a computer, a notebook computer, a digital broadcast terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), or a tablet personal computer (PC), but is not limited thereto. The electronic device may communicate with other electronic devices and / or servers via a network using a wireless or wired communication method.
[0091] Reference Figure 2A , the electronic device 100 may include a voice input device 110, a controller 120, a memory 130, a speaker 140, and a display 150. Figure 2A All components shown are essential components of the electronic device 100. Figure 2A More components than shown or by including Figure 2A Fewer components are shown to implement the electronic device 100 .
[0092] For example, Fig.14As shown, the electronic device 1000 according to one or more embodiments may include a user inputter 1100 , a sensor 1400 , a communicator 1500 , and an audio / video (A / V) inputter 1600 .
[0093] The voice inputter 110 may receive a voice including a human voice from a speaker. The voice inputter 110 may recognize the input human voice and output a result of recognizing the human voice. The result of recognition may include a language spoken by the speaker. "Spoken language" as a language spoken by the speaker may correspond to a predetermined instruction. However, the present invention is not limited thereto. Spoken language may correspond to a natural language, which is a language commonly used by the speaker.
[0094] The voice inputter 110 may include a voice recognition engine. The voice recognition engine may recognize a voice spoken by a speaker by applying a voice recognition algorithm to the input voice, and may generate a recognized result. Here, the input voice may be converted into a form suitable for voice recognition, and the voice inputter 110 may detect an actual voice part included in the input voice by detecting a start point and an end point from the voice signal.
[0095] The controller 120 may include a processor 122, a data training module 123, and a memory 124. The controller 120 may be implemented as a computer-readable recording medium including software, hardware, or a combination of software and hardware.
[0096] The processor 122 may be configured to process instructions of a computer program by performing calculations, logic, input and output calculations, and signal processing. The instructions of the computer program may be stored in the memory 124 and may be provided to the processor 122 by the memory 124. In the embodiments described below, the functions and / or operations performed by the controller 120 may be implemented by the processor 122 by executing instructions received according to a computer program code stored in a recording medium such as the memory 124.
[0097] The processor 122 may include, for example, at least one of a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), or a field programmable gate array (FPGA), but is not limited thereto. According to an embodiment of the present disclosure, when the electronic device 100 is a mobile device such as a smart phone, a tablet PC, etc., the processor 122 may be an application processor (AP) configured to execute an application.
[0098] The data training module 123 may include a hardware device configured to train vector data such as a vocal feature vector or a speaker feature vector by using a DNN model. The data training module 123 may include a hardware device having computing and training capabilities to train a large amount of data by executing software that performs learning via a DNN model.
[0099] Memory 124 may include, for example, any of random access memory (RAM), static random access memory (SRAM), read only memory (ROM), electrically erasable programmable read-only memory (EEPROM), or programmable read-only memory (PROM), but is not limited to the described examples.
[0100] The controller 120 may extract a speaker feature vector from the vocal data of the speech input by the speech inputter 110, and generate a speaker feature map by positioning the extracted speaker feature vector at a specific position on the multidimensional vector space. According to an embodiment of the present disclosure, the speech inputter 110 may extract a vocal feature vector of the input vocal data by applying any one of the feature vector extraction techniques of cepstrum, LPC, MFCC, and filter bank energy to the vocal data detected from the speaker's speech by using an endpoint detection (EPD) method. Here, the extracted vocal feature vector may be extracted based on the size of the vocal signal value accumulated in the frequency range of each phoneme, and may be an n-dimensional vector, where n is equal to or greater than 2.
[0101] The data training module 123 of the controller 120 may extract a speaker feature vector by applying a DNN model to a vocal feature vector extracted from vocal data of a speech input by a speaker. The data training module 123 may train the DNN model by applying the vocal feature vector as an input value to the DNN model and using the speaker feature value as an output value to obtain a speaker feature vector. Here, the DNN model may include at least one of CNN, RNN, or GAN. However, DNN is not limited to the described example. The data training module 123 may extract a speaker feature vector from a vocal feature vector by using all types of neural network model methods currently known. The controller 120 may accumulate the extracted n-dimensional speaker feature vectors, and may generate a speaker feature map by locating the accumulated speaker feature vectors at specific positions on an n-dimensional vector space, respectively. The speaker feature map may indicate the speech input from multiple speakers as vectors in a multidimensional vector space (i.e., an n-dimensional vector space, where n is equal to or greater than 2). Since the speaker feature vectors are located adjacent to each other on the speaker feature map, the human voices of the speakers may be similar.
[0102] The controller 120 may form a cluster by grouping at least one speaker feature vector located within a predetermined distance on the speaker feature graph. Here, a "cluster" may be a group of speaker feature vectors having the same vocal features, and may be a group unit indicating the vocal features of each speaker. According to an embodiment of the present disclosure, the controller 120 may classify a plurality of speakers according to the formed plurality of clusters.
[0103] According to an embodiment of the present disclosure, the data training module 123 may cluster the speaker feature vectors on the speaker feature map based on each speaker by applying the DNN model. For example, the data training module 123 may classify the speaker feature vectors on the speaker feature map according to each speaker by applying the speaker feature vector as an input value to the DNN model and applying the speaker's index as an output value to the DNN model. Here, the speaker's index may represent a value obtained by converting speaker identification information into a vector value using at least one of one-hot encoding, word2vec, and GloVe. The data training module 123 may analyze whether the speaker feature vectors are similar to each other by using the DNN model, and may cluster similar speaker feature vectors. For example, the data training module 123 may cluster the speaker feature vectors by using a DNN model having an output value of 1 when speaker 1 and speaker 2 are the same person and an output value of 0 when speaker 1 and speaker 2 are different people.
[0104] According to an embodiment of the present disclosure, the data training module 123 can obtain information about the relationship between speakers by using a DNN model. In detail, the data training module 123 can predict the relationship between speakers by applying the speaker feature vector as an input value to the DNN model and applying the information about the relationship between speakers as a label to the DNN model.
[0105] According to an embodiment of the present disclosure, the voice inputter 110 may receive a voice for inputting recognition information from a user, and the controller 120 may extract a speaker feature vector according to the voice received from the user, and map the extracted speaker feature vector to one of the multiple clusters on the speaker feature map. Here, the mapping method used by the controller 120 may be to compare the speaker feature vector extracted from the received voice with the speaker feature vector on the speaker feature map, and may correspond to a direct comparison or statistical method. Here, direct comparison refers to a method of setting a unit to be recognized (such as a word, phoneme, etc.) as a speaker feature vector and measuring the similarity between the input human voice and the speaker feature vector. Direct comparison may include vector quantization. A statistical method refers to a method of setting a unit to be recognized as a state sequence and using the relationship between the state sequences. Statistical methods may include dynamic time warping (DTW), hidden Markov models (HMM), methods using neural circuit networks, and the like.
[0106] The controller 120 may map the speaker feature vector extracted from the voice input by the user to the cluster, and may register and store the user identification information to the mapped cluster. FIG. 6A to FIG. 6C A detailed method of registering user identification information performed by the controller 120 is described in detail.
[0107] The controller 120 may generate a distinguishing sentence for registering user identification information, and may control the speaker 140 to output a voice message requesting the user to speak the generated distinguishing sentence. According to an embodiment of the present disclosure, the controller 120 may generate a distinguishing sentence including phonemes, wherein, for the phonemes, the difference in frequency signal values between a plurality of speakers included in a plurality of clusters pre-registered on a speaker feature map is equal to or greater than a predetermined critical value. According to an embodiment of the present disclosure, the controller 120 may analyze the cumulative degree of the frequency signal value of each phoneme for a voice input from a speaker, and may generate a distinguishing sentence including phonemes having a relatively low cumulative degree or a zero cumulative degree. Reference will be made to the embodiments of the present disclosure. 7A to 7C A detailed method of generating a distinction sentence performed by the controller 120 is described in detail.
[0108] The controller 120 may predict the relationship between the plurality of speakers corresponding to the plurality of clusters based on the distances and the degree of directionality between the plurality of clusters on the speaker feature graph. According to an embodiment of the present disclosure, the controller 120 may identify the speaker of the input speech based on the predicted relationship between the speakers, and provide a personalized service synchronized with the identified speaker. According to an embodiment of the present disclosure, the data training module 123 may identify the speaker from the speaker feature vector by using a DNN model, and predict the relationship between the speakers. Figures 9 to 11 Describe this aspect in detail.
[0109] The controller 120 may extract a speaker feature vector of the user's voice received by the voice inputter 110, map the extracted speaker feature vector to a specific cluster on the speaker feature map, and predict the user's health status based on the positional relationship between other speaker feature vectors in the mapped cluster and the speaker feature vector of the input voice. According to an embodiment of the present disclosure, the controller 120 may extract a first feature vector from the user's voice received by the voice inputter 110, map the first feature vector to a specific cluster on the speaker feature map, identify directionality by comparing the position of a second feature vector in the mapped cluster with the position of the first feature vector, and predict the user's health status based on the identified directionality. Fig. 12A , Fig. 12B and Fig.13 The detailed method of this aspect is described in detail.
[0110] The memory 130 may store the speaker feature vector and the speaker feature graph extracted by the controller 120. The memory 130 may include, for example, at least one type of storage medium among a flash memory type storage medium, a hard disk type storage medium, a multimedia card micro storage medium, a card type memory (e.g., a secure digital (SD) or extreme digital (XD) memory), a magnetic memory, a magnetic disk, and an optical disk, but is not limited to the described examples.
[0111] According to an embodiment of the present disclosure, the memory 130 may not be a component included in the electronic device 100, and may be implemented as an external network server or a cloud server. In this case, the speaker feature vector and the speaker feature map extracted by the controller 120 may be stored in the external network server or the cloud server, and the controller 120 may access the feature vector and the speaker feature map via a communication network. Figure 2B Describe this aspect in detail.
[0112] The speaker 140 may output voice data related to functions performed by the electronic device 100. These functions may include, for example, a message receiving sound, a notification sound, etc. The speaker 140 may output a voice message stored in the memory 130 or a voice message generated by the controller 120. According to an embodiment of the present disclosure, the speaker 140 may output a voice message requesting the user to speak a distinguishing sentence for registering user identification information. According to an embodiment of the present disclosure, the speaker 140 may output a voice message corresponding to the health status of the user predicted by the controller 120.
[0113] The display 150 may display a personalized service synchronized with a speaker identified by the controller 120. The display 150 may include a physical device, wherein the physical device includes, for example, at least one of a cathode ray tube (CRT) display, a liquid crystal display (LCD) display, a plasma display panel (PDP) display, an organic light emitting diode (OLED) display, a field emission display (FED) display, a light emitting diode (LED) display, a vacuum fluorescent display (VFD) display, a digital light processing (DLP) display, a flat panel display, a 3D display, or a transparent display, but is not limited thereto. According to an embodiment of the present disclosure, the display 150 may be formed as a touch screen including a touch interface. When the display 150 is formed as a touch screen, the display 150 may be integrated with a touch panel and may receive a touch input from a user.
[0114] Figure 2B is a block diagram of components of an electronic device and a server according to an embodiment of the present disclosure.
[0115] Reference Figure 2B , the electronic device 100 may include a voice inputter 110 , a controller 120 , a speaker 140 , a display 150 , and a communicator 160 . Figure 2B The voice input device 110, the controller 120, the speaker 140 and the display 150 of the components shown are respectively Figure 2A The components shown are the same, and therefore, no repeated description will be given. Figure 2A The electronic device 100 shown is different. Figure 2B The electronic device 100 shown may not include the memory 130. However, it is not limited thereto, and Figure 2B The electronic device 100 shown may also include a memory 130 (see Figure 2A ).
[0116] Reference Figure 2B , the electronic device 100 may transmit and receive data to and from the server 200 via the communicator 160. The communicator 160 may be connected to the server 200 via a wired or wireless communication method, and may perform data communication with the server 200. The communicator 160 may perform data communication with the server 200 by using at least one of a data communication method such as a wired local area network (LAN), a wireless LAN, Wi-Fi, Bluetooth, Zigbee, Wi-Fi Direct (WFD), an infrared data association (IrDA), Bluetooth low energy (BLE), near field communication (NFC), wireless broadband Internet (Wibro), world interoperability for microwave access (WiMAX), shared wireless access protocol (SWAP), wireless Gigabit Alliance (WiGiG), and RF communication.
[0117] According to an embodiment of the present disclosure, the electronic device 100 may obtain vocal data from a voice input from a speaker via the voice inputter 110, and may transmit the obtained vocal data to the server 200. The server 200 may generate a speaker feature map via the vocal data received from the electronic device 100. Figure 3B Describe this aspect in detail.
[0118] The server 200 may be a human voice recognition server or a speaker recognition server. Figure 2B One server 200 is shown, but this is an example for convenience of explanation. According to an embodiment of the present disclosure, the server 200 may include one or more servers 200.
[0119] According to an embodiment of the present disclosure, the electronic device 100 may obtain vocal data according to a voice input from a speaker, extract a multi-dimensional vocal feature vector from the obtained vocal data, and transmit the extracted vocal feature vector to the server 200 via the communicator 160. Here, the vocal data may be, for example, an audio file of a voice. The server 200 may generate a speaker feature map for each speaker through training related to classification of the vocal feature vector received from the electronic device 100. Figure 3C Describe this aspect in detail.
[0120] According to an embodiment of the present disclosure, the electronic device 100 may obtain vocal data according to a voice input from a speaker, extract a multidimensional vocal feature vector from the vocal data, and train a DNN model by taking the extracted vocal feature vector as input and taking the speaker feature value as output to extract the speaker feature vector. The electronic device 100 may generate a speaker feature graph indicating the extracted speaker feature vector on a multidimensional vector space, and transmit the generated speaker feature graph to the server 200 via the communicator 160. Figure 3D Describe this aspect in detail.
[0121] Figure 3A is a flowchart of a method of processing vocal data of a speech received from a speaker, performed by an electronic device according to an embodiment of the present disclosure.
[0122] Reference Figure 3A In operation S310, the electronic device 100 may extract a speaker feature vector from vocal data of a speech received from a speaker, and generate a speaker feature graph indicating the extracted speaker feature vector on a multi-dimensional space.
[0123] According to an embodiment of the present disclosure, the electronic device 100 can extract the vocal feature value of the input vocal data by using any one of the feature vector extraction techniques among the cepstrum, LPC, MFCC and filter bank energy for the vocal data of the speech. The extracted vocal feature vector can be an n-dimensional vector, where n is equal to or greater than 2. In addition, the extracted vocal feature vector may include at least two vector columns.
[0124] The speaker feature may be indicated as a speaker feature vector. In detail, the electronic device 100 may extract the speaker feature vector by applying the DNN model to the vocal feature vector. The electronic device 100 may train the DNN model by applying the vocal feature vector as an input value to the DNN model and applying the speaker feature value as an output value to the DNN model to obtain the speaker feature vector. Here, the DNN model may include at least one of CNN, RNN, or GAN. However, the DNN model is not limited to the described example.
[0125] According to an embodiment of the present disclosure, the vocal feature vector input to the DNN model may be an n-dimensional vector, where n is equal to or greater than 2. In addition, the vocal feature vector input to the DNN model may be at least two vector columns. The speaker feature vector extracted by training the DNN model may be an n-dimensional vector, where n is equal to or greater than 2.
[0126] According to another embodiment of the present disclosure, the electronic device 100 may indicate the speaker feature vector as an i-vector (which is a feature vector). The electronic device 100 may obtain the i-vector by obtaining the vocal feature vector from the vocal data of the speaker.
[0127] The electronic device 100 may generate a speaker feature map by positioning the extracted speaker feature vectors at positions in an n-dimensional virtual space, wherein the positions respectively correspond to vector values of the speaker feature vectors.
[0128] In operation S320, the electronic device 100 may form a plurality of clusters by grouping at least one speaker feature vector located on the speaker feature graph.
[0129] According to an embodiment of the present disclosure, the electronic device 100 may form a cluster by grouping a set of speaker feature vectors clustered within a predetermined distance on a speaker feature graph into a virtual group. Clustering is a concept derived from the following aspects: speaker feature vectors with similar features of vocal data may be positioned relatively adjacent to each other on a speaker feature graph, and speaker feature vectors with non-similar features of vocal data may be positioned relatively far away from each other on a speaker feature graph. The cluster may indicate the vocal features of each speaker.
[0130] According to an embodiment of the present disclosure, the electronic device 100 can cluster speaker feature vectors on a speaker feature map based on each speaker by applying a DNN model. For example, the electronic device 100 can classify speaker feature vectors on a speaker feature map according to each speaker by applying the speaker feature vector as an input value to the DNN model and applying the speaker index as an output value to the DNN model. Here, the speaker index may represent a value obtained by converting speaker identification information into a vector value using at least one encoding of one-hot encoding, word2vec, and GloVe.
[0131] In operation S330, the electronic device 100 may classify the plurality of speakers according to the plurality of clusters. For example, the electronic device 100 may classify a voice having a speaker feature vector mapped in a first cluster among the plurality of clusters as a first user, a voice having a speaker feature vector mapped in a second cluster as a second user, and a voice having a speaker feature vector mapped in a third cluster as a third user.
[0132] FIG. 3B to FIG. 3D is a flowchart of an operating method of a system including the electronic device 100 and the server 200 according to various embodiments of the present disclosure.
[0133] Reference FIG. 3B to FIG. 3D , the server 200 may be a human voice recognition server or a speaker recognition server. FIG. 3B to FIG. 3D One server 200 is shown, but this is an example for convenience of explanation. According to an embodiment of the present disclosure, the server 200 may include at least one server 200 .
[0134] Reference Figure 3B , the electronic device 100 can communicate with the user through the communicator 160 (refer to Figure 2B ) sends data to and receives data from the server 200.
[0135] In operation S312, the electronic device 100 may obtain vocal data from a voice input by a speaker. According to an embodiment of the present disclosure, the electronic device 100 may receive a voice from a speaker via a microphone or a speaker and obtain vocal data from the received voice.
[0136] In operation S313 , the electronic device 100 may transmit the vocal data to the server 200 .
[0137] In operation S322, the server 200 may extract a multi-dimensional vocal feature vector based on the vocal data received from the electronic device 100. According to an embodiment of the present disclosure, the server 200 may extract the vocal feature value of the input vocal data by using any one of the feature vector extraction techniques of cepstrum, LPC, MFCC, and filter bank energy for the vocal data. The vocal feature vector extracted by the server 200 may be extracted based on the size of the vocal signal value accumulated within the frequency range of each phoneme, and may be an n-dimensional vector, where n is equal to or greater than 2.
[0138] In operation S324, the server 200 may generate a speaker feature map by classifying the vocal feature vector according to each speaker. According to an embodiment of the present disclosure, the server 200 may extract the speaker feature vector by applying the DNN model to the vocal feature vector. The server 200 may train the DNN model by applying the vocal feature vector as an input value to the DNN model and applying the speaker's feature value as an output value to the DNN model to obtain the speaker feature vector. Here, the DNN model may include at least one of CNN, RNN, or GAN. However, the DNN model is not limited to the described example.
[0139] The server 200 may accumulate the extracted n-dimensional speaker feature vectors, and may generate a speaker feature graph by respectively locating the accumulated speaker feature vectors at specific positions on the n-dimensional vector space.
[0140] In operation S326 , the server 200 may transmit the speaker characteristic map to the electronic device 100 .
[0141] In operation S328, the electronic device 100 may generate a plurality of clusters by grouping the speaker feature vectors on the speaker feature graph. Figure 3A The operation S320 is the same as that of the operation S320, and thus, a repeated description will not be given.
[0142] In operation S330 , the electronic device 100 may classify the plurality of speakers according to the plurality of clusters.
[0143] Reference Figure 3C , the electronic device 100 can communicate with the user through the communicator 160 (refer to Figure 2B ) sends data to and receives data from the server 200.
[0144] In operation S312, the electronic device 100 may obtain vocal data from a voice input by a speaker.
[0145] In operation S314, the electronic device 100 may extract a multidimensional vocal feature vector from the vocal data. According to an embodiment of the present disclosure, the electronic device 100 may extract a vocal feature vector of the input vocal data by applying any one of the feature vector extraction techniques of cepstrum, LPC, MFCC, and filter bank energy to the vocal data included in the speech received from the speaker. The vocal feature vector extracted by the electronic device 100 may be extracted based on the size of the voice signal value accumulated within the frequency range of each phoneme, and may be an n-dimensional vector, where n is equal to or greater than 2.
[0146] In operation S318 , the electronic device 100 may transmit the vocal feature vector to the server 200 .
[0147] In operation S324, the server 200 may generate a speaker feature map by classifying the vocal feature vector according to each speaker. In operation S326, the server 200 may transmit the speaker feature map to the electronic device 100. Operations S324 and S326 are respectively related to Figure 3B Operations S324 and S326 are the same, and therefore, a repeated description will not be given.
[0148] In operation S328 , the electronic device 100 may generate a plurality of clusters by grouping speaker feature vectors on the speaker feature graph.
[0149] In operation S330 , the electronic device 100 may classify the plurality of speakers according to the plurality of clusters.
[0150] exist Figure 3B and Figure 3C , the speaker characteristic graph is generated by the server 200. However, the speaker characteristic graph may be generated by an edge device including the electronic device 100. The edge device may include, for example, a device including at least one of a smart phone, a cellular phone, an AI speaker, a speaker, a navigation device, a computer, a laptop computer, a digital broadcast terminal, a PDA or a tablet PC, a home appliance having computing capabilities, or a device configured to manage other devices (such as a router), but the edge device is not limited thereto.
[0151] Reference Figure 3D , the electronic device 100 may generate a speaker characteristic graph and transmit the generated speaker characteristic graph to the server 200 .
[0152] In operation S312, the electronic device 100 may obtain vocal data from the voice received from the speaker. In operation S314, the electronic device 100 may extract a multi-dimensional vocal feature vector from the vocal data. In operation S315, the electronic device 100 may generate a speaker feature map by classifying the vocal feature vector according to each speaker. In operation S316, the electronic device 100 may transmit the generated speaker feature map to the server 200.
[0153] In operation S340, the server 200 may generate a plurality of clusters by grouping speaker feature vectors on a speaker feature map received from the electronic device 100. According to an embodiment of the present disclosure, the server 200 may form clusters based on distances of speaker feature vectors on the speaker feature map, but is not limited thereto. According to an embodiment of the present disclosure, the server 200 may cluster speaker feature vectors on a speaker feature map based on each speaker by applying a DNN model. For example, the data training module 123 may train a DNN model by applying the speaker feature vector as an input value to the DNN model and applying the speaker's index as an output value to the DNN model, and classify the speaker feature vectors on the speaker feature map according to each speaker.
[0154] In operation S350, the server 200 may classify the plurality of speakers according to the plurality of clusters. Here, the server 200 may be a speaker recognition server.
[0155] FIG. 4A to FIG. 4C is a diagram for describing a method of generating a speaker characteristic map by processing vocal data input from a speaker, which is performed by the electronic device 100 according to various embodiments of the present disclosure.
[0156] Figure 4A is a diagram illustrating a frequency signal value 400A of each phoneme extracted by processing vocal data of a speech input by speaker A and a frequency signal value 400B of each phoneme extracted by processing vocal data of a speech input by speaker B according to an embodiment of the present disclosure.
[0157] Reference Figure 4A , the numbers are signal values accumulated based on the frequency range of each phoneme and may not be exact values and may indicate relative magnitude. Figure 4A In the example, the total frequency is divided into three ranges, namely high, medium and low. The number of frequency ranges and the length of the frequency ranges are examples, and the number of frequency ranges is not limited to three.
[0158] According to an embodiment of the present disclosure, when the electronic device 100 receives voices from speaker A and speaker B respectively, the electronic device 100 can detect vocal data from the input voice, and can extract a vocal feature vector of the input vocal data by applying any one of the feature vector extraction techniques of cepstrum, LPC, MFCC and filter group energy to the detected vocal data.
[0159] As an example of a method of extracting a vocal feature vector, the electronic device 100 may use a method of calculating the cumulative amount of signal values of a frequency range for each word or each phoneme in the speech (i.e., the cumulative amount of energy values of the speech). Referring to the frequency signal value 400A, in the case of the speech of speaker A, for the phoneme / b / , the cumulative energy in the high frequency range is measured to be relatively high at 80, and the cumulative energy in the middle frequency range is measured to be approximately 20. Similarly, in the speech of speaker A, for the phoneme / d / , the cumulative energy in the high frequency range is measured to be approximately 60, and the cumulative energy in the middle frequency range is measured to be approximately 50. In the speech of speaker A, for the phoneme / g / , the cumulative energy in the high frequency range is measured to be approximately 30, and the cumulative energy in the middle frequency range is measured to be approximately 20.
[0160] Referring to the frequency signal value 400B, in the case of the speech of speaker B, for the phoneme / b / , the cumulative energy in the low frequency range is measured as relatively high 60, and the cumulative energy in the middle frequency range is measured as about 20. In the speech of speaker B, for the phoneme / g / , the cumulative energy in the high frequency range is hardly calculated, that is, the cumulative energy in the high frequency range is calculated as 0, and the cumulative energy in the low frequency range is calculated as about 40.
[0161] In the case of speaker A and speaker B, referring to the frequency signal values 400A and 400B in the frequency range for each phoneme, even for the same phoneme, the speakers are different in the frequency range in which energy is relatively more accumulated than in other frequency ranges. For example, in the case of the phoneme / b / , speaker A has high cumulative energy in the high frequency range, while speaker B has relatively high cumulative energy in the low frequency range. The electronic device 100 may generate a vocal feature vector indicating the characteristics of the voice of each speaker based on the energy value calculated in the frequency range of each phoneme. Here, the vocal feature vector may be formed as a multidimensional vector column.
[0162] Figure 4B An embodiment of extracting features X, Y, and Z based on a signal value calculated in a frequency range of each phoneme of speaker A and speaker B according to an embodiment of the present disclosure is shown.
[0163] Reference Figure 4B, the electronic device 100 can extract features X, Y, and Z by using the difference between the frequency signal values 400A and 400B within the frequency range of each phoneme, wherein the frequency signal values 400A and 400B are calculated in the speech of speaker A and speaker B. Figure 4A , in the case of phoneme / b / , speaker A has a cumulative energy value of 0 in the low frequency range, while speaker B has a cumulative energy value of 60 in the low frequency range. That is, this corresponds to a case where the difference between the signal value of speaker A and the signal value of speaker B is equal to or greater than a predetermined value. Therefore, the electronic device 100 can extract feature X by using the cumulative energy value in the low frequency range for phoneme / b / . Similarly, in the case of phoneme / d / , speaker A has a cumulative energy value of 60 in the high frequency range, while speaker B has a cumulative energy value of 50 in the high frequency range. Therefore, the electronic device 100 can extract feature Y by using the cumulative energy value in the high frequency range for phoneme / d / . By using the same method, the electronic device 100 can extract feature Z by using the cumulative energy value in the high frequency range for phoneme / g / .
[0164] Figure 4C 4 is a diagram illustrating a method of generating a speaker feature graph 410 by using features extracted from speaker A and speaker B and locating the speaker feature vectors 421 and 422 on a multi-dimensional space according to an embodiment of the present disclosure, performed by the electronic device 100. Figure 5 A detailed method of generating the speaker feature vectors 421 and 422 by using the vocal feature vector is described in detail.
[0165] Reference Figure 4C , the electronic device 100 may generate a speaker feature vector 421 having values (0, 6, 3) by using features X, Y, and Z extracted for speaker A. Similarly, the electronic device 100 may generate a speaker feature vector 422 having values (6, 5, 0) by using features X, Y, and Z extracted for speaker B.
[0166] The electronic device 100 may generate the speaker feature graph 410 by positioning the speaker feature vector 421 of speaker A having values (0, 6, 3) and the speaker feature vector 422 of speaker B having values (6, 5, 0) on a three-dimensional space. Figure 4C 4 and 5. Speaker feature vectors 421 and 422 and speaker feature graph 410 are three-dimensionally illustrated in FIG. However, this is for convenience of explanation, and speaker feature vectors 421 and 422 and speaker feature graph 410 are not limited thereto.
[0167] Figure 4Dis a flowchart of a method for generating a speaker characteristic map by processing vocal data of speech input by a plurality of speakers, which is performed by an electronic device according to an embodiment of the present disclosure.
[0168] Reference Figure 4D In operation S410, the electronic device may receive voices from a plurality of speakers. According to an embodiment of the present disclosure, the electronic device 100 may recognize a human voice spoken by a speaker by applying a human voice recognition algorithm to the input human voice using a human voice recognition engine, and generate a recognition result. The electronic device 100 may detect an actual human voice portion included in the input human voice by detecting a start point and an end point of a human voice signal.
[0169] In operation S420, the electronic device 100 may extract the frequency signal value of each phoneme from the input speech and accumulate the signal value. According to an embodiment of the present disclosure, the electronic device 100 may apply any feature vector extraction technology among cepstrum, LPC, MFCC and filter bank energy to the human voice data detected from the input speech. The electronic device 100 may accumulate the relative energy value calculated for each phoneme in each frequency range for the human voice data.
[0170] In operation S430, the electronic device 100 may extract distinguishing features between speakers based on the accumulated signal values. According to an embodiment of the present disclosure, when the difference between the accumulated signal values (i.e., accumulated energy values) of the same phoneme and the same frequency range of the speech of multiple speakers is equal to or greater than a predetermined critical value, the electronic device 100 may extract a feature value. The feature values extracted from the phoneme and the frequency range may be multiple, and the multiple feature values may be independent of each other.
[0171] In operation S440 , the electronic device 100 may convert the extracted feature value into an n-dimensional speaker feature vector, where n is equal to or greater than 2.
[0172] In operation S450, the electronic device 100 may generate a speaker feature map by positioning the converted n-dimensional speaker feature vectors at positions in a multi-dimensional space, wherein the positions respectively correspond to values of the speaker feature vectors.
[0173] Figure 5 is a diagram illustrating a method of converting a human voice feature vector 510 into a speaker feature vector by using an AI technology, performed by the electronic device 100 according to an embodiment of the present disclosure. Figure 5 The illustrated method may be executed not only by the electronic device 100 but also by the server 200 .
[0174] Reference Figure 5, the electronic device 100 may extract a speaker feature vector by applying the DNN model 500N to the vocal feature vector 510. In detail, the electronic device 100 may train the DNN model 500N to obtain a speaker feature vector by applying the vocal feature vector 510 as an input value to the DNN model 500N and applying the speaker index 520 as an output value to the DNN model 500N. Here, the speaker index 520 may be a feature value about the speaker, and may be a vector obtained by converting an index about each of the first to Nth speakers into a vector column using one-hot encoding. For example, when n is 3, the speaker index 520 may be formed as a vector column, such as [1,0,0], [0,1,0], and [0,0,1]. However, this is for ease of explanation, and the speaker index 520 is not limited to the above-mentioned vector column.
[0175] The DNN model 500N may include at least one of CNN, RNN, or GAN. However, the DNN model is not limited to the described examples. The DNN model 500N used by the electronic device 100 according to the present disclosure may include all types of currently known neural network models.
[0176] FIG. 6A to FIG. 6C is a diagram illustrating a method of registering user identification information corresponding to a cluster formed on a speaker characteristic graph, performed by the electronic device 100 according to various embodiments of the present disclosure. FIG. 6A to FIG. 6C All are diagrams showing speaker feature vectors located on speaker feature graphs 600A, 600B, and 600C, and although speaker feature graphs 600A, 600B, and 600C are shown two-dimensionally, this is for convenience of explanation, and the speaker feature graphs of the present disclosure are not limited thereto.
[0177] Reference Fig. 6A In the speaker feature graph 600A shown, a plurality of first feature vectors 610a to 610n clustered within a predetermined distance may form a first cluster 610, and a plurality of second feature vectors 620a to 620n clustered within a predetermined distance may form a second cluster 620. Fig. 6A In the illustrated embodiment, a plurality of first feature vectors 610a to 610n located in the first cluster 610 may be registered as the voice of speaker A, and the identification information of speaker A may be registered. Similarly, a plurality of second feature vectors 620a to 620n located in the second cluster 620 may be registered as the voice of speaker B, and the identification information of speaker B may be registered. Here, the identification information may refer to information for identifying the speaker, and the identification information includes at least one of the speaker's user ID, password, address, phone number, age, gender, or Samsung account information.
[0178] A plurality of third feature vectors 630a to 630n may be clustered within a predetermined distance on the speaker feature graph 600A, but may not form additional clusters. Similarly, a plurality of fourth feature vectors 640a to 640n may be clustered within a predetermined distance on the speaker feature graph 600A, but may not form additional clusters.
[0179] Reference Figure 6B As shown in the speaker characteristic graph 600B, the electronic device 100 may form a third cluster 630 by grouping a plurality of third characteristic vectors 630a to 630n clustered within a predetermined distance.
[0180] Reference Figure 6C , the electronic device 100 may receive a speech inputting recognition information from a speaker C classified into the third cluster 630 , and may register the recognition information of the speaker C to correspond to the third cluster 630 .
[0181] According to an embodiment of the present disclosure, the electronic device 100 may output a message requesting the speaker C to input recognition information, wherein the speaker C has spoken the voice corresponding to the plurality of third feature vectors 630a to 630n included in the third cluster 630. According to an embodiment of the present disclosure, the message requesting recognition information may be output via the speaker 140 (see Figure 2A and Figure 2B ) or displayed on the display 150 (see Figure 2A and Figure 2B ) on the user interface (UI).
[0182] The electronic device 100 may receive a speech in which the recognition information of the speaker C is input from the speaker C, and extract a speaker feature vector from the vocal data of the received speech. The electronic device 100 may map the speaker feature vector extracted from the speaker C to the third cluster 630, and register the recognition information input from the speaker C in the third cluster 630. Through the above process, the recognition information of the speaker C may be stored in the third cluster 630, and the third cluster 630 may be stored in the memory 130 (see FIG. 1 ) as a cluster of registered speakers (similar to the first cluster 610 and the second cluster 620). Figure 2A and 2B )middle.
[0183] Reference FIG. 6A to FIG. 6C, the electronic device 100 can extract speaker feature vectors based on the human voice data of speech input from multiple speakers (speakers A, B, and C), can form clusters by grouping speaker feature vectors clustered within a predetermined distance, can receive speaker identification information corresponding to the cluster, and can register the identification information in the cluster. Therefore, the electronic device 100 can provide personalized services synchronized according to the identification information of the registered user (e.g., user ID, phone number, address, age, gender, etc.), and therefore, the electronic device 100 can improve user convenience.
[0184] Fig. 7A and Figure 7B is a diagram illustrating a method of generating a distinguishing sentence for identifying a speaker on a speaker feature map based on a frequency feature value of each phoneme of speech and outputting the generated distinguishing sentence according to various embodiments of the present disclosure. The electronic device 100 may receive speech from a speaker, generate a distinguishing sentence based on the frequency feature value of each phoneme of speech, and output the distinguishing sentence. However, the device configured to generate and output the distinguishing sentence is not limited to the electronic device 100. According to an embodiment of the present disclosure, the server may analyze the frequency feature value of each phoneme of the human voice received from the electronic device 100 and generate a distinguishing sentence. In this case, the server may directly store the speaker feature map. The server may send the generated distinguishing sentence to the electronic device 100.
[0185] Fig. 7A Tables 700A and 700B illustrate frequency feature values of each phoneme regarding registered speaker A and registered speaker B according to an embodiment of the present disclosure.
[0186] Reference Fig. 7A , the numbers in Table 700A and Table 700B are cumulative signal values based on the frequency range of each phoneme, and the numbers may not be exact values and may indicate relative sizes. Fig. 7A In the example, the total frequency is divided into three ranges, namely high, medium and low. The number of frequency ranges and the length of the frequency ranges are examples, and the number of frequency ranges is not limited to three.
[0187] Referring to Table 700A, in the case of the registered speech of speaker A, for the phoneme / b / , a signal value corresponding to 80 may be calculated in the high frequency range, that is, the cumulative energy value is 80, and a cumulative energy value of 0 may be calculated in the low frequency range. For the phoneme / g / , in the registered speech of speaker A, a cumulative energy value corresponding to 30 may be calculated in the high frequency range, and a cumulative energy value corresponding to 10 may be calculated in the low frequency range.
[0188] Referring to Table 700B, in the case of the registered speech of speaker b, for the phoneme / b / , a signal value corresponding to 30 may be calculated in the high frequency range, that is, the cumulative energy value is 30, a cumulative energy value of 60 may be calculated in the middle frequency range, and a cumulative energy value of 40 may be calculated in the low frequency range. For the phoneme / g / , in the registered speech of speaker B, a cumulative energy value corresponding to 90 may be calculated in the high frequency range, and cumulative energy values corresponding to 60 may be calculated in the middle frequency range and the low frequency range.
[0189] Referring to Tables 700A and 700B, in the case of phonemes / b / and / g / , the cumulative energy values in each frequency range of the voices of the registered speaker A and the registered speaker B have a large difference between the registered speaker A and the registered speaker B, and therefore, it is easy to distinguish between the registered speaker A and the registered speaker B. However, in the case of phonemes / d / and / f / , the cumulative energy values in each frequency range of the voices of the registered speaker A and the registered speaker B have a small difference between the registered speaker A and the registered speaker B.
[0190] According to an embodiment of the present disclosure, the electronic device 100 or the server 200 may generate a distinguishing sentence to map the voice of an unregistered speaker (i.e., a non-registered speaker) to a cluster of a registered speaker on a speaker feature map. The distinguishing sentence may be a sentence for distinguishing which cluster of pre-registered clusters the voice of a user using the electronic device 100 belongs to, or a sentence for distinguishing whether the user's voice belongs to a cluster of a new speaker, and may be a sentence for requesting the user to speak a voice.
[0191] According to an embodiment of the present disclosure, the electronic device 100 or the server 200 may generate a distinguishing sentence such that the distinguishing sentence includes a phoneme for which the difference in frequency characteristics (i.e., the difference in cumulative energy values of frequency ranges between the voices of speakers pre-registered on a speaker feature map) is equal to or greater than a predetermined critical value. Fig. 7A In the illustrated embodiment, for example, a distinguished sentence 704 “go to Greece and buy an ice cream” including the phonemes / g / and / r / may be generated.
[0192] According to an embodiment of the present disclosure, the electronic device 100 may output a message 702 requesting the user to speak the generated distinguishing sentence 704. The electronic device 100 may display the message 702 requesting the user to speak the distinguishing sentence 704 on the display 150. However, the present disclosure is not limited thereto. According to an embodiment of the present disclosure, the electronic device 100 may output a message 702 requesting the user to speak the distinguishing sentence 704 through the speaker 140 (refer to Figure 2A and 2B) outputs a voice message requesting the user to speak a distinguishing sentence 704. The electronic device 100 may directly generate a distinguishing sentence and output the distinguishing sentence. However, it is not limited thereto, and the electronic device 100 may receive a distinguishing sentence generated by the server 200 from the server 200 and output the received distinguishing sentence.
[0193] Figure 7B A table 710 indicating frequency feature values of phonemes in speech of an unregistered speaker according to an embodiment of the present disclosure is shown. Numbers described in the table 710 represent cumulative time of a signal calculated for a frequency range of each phoneme.
[0194] Referring to Table 710, in the case of the speech of the unregistered speaker, for the phonemes / d / and / f / , the phonemes are pronounced and accumulated for 30 seconds and 20 seconds, respectively, and the phoneme / b / is pronounced and accumulated for 1 second, and the phoneme / g / is pronounced and accumulated for 0 seconds. It may indicate that the phonemes / d / and / f / have a higher pronunciation accumulation degree than the phonemes / b / and / g / in the speech of the unregistered voice. In particular, it may indicate that the phoneme / g / is not accumulated at all.
[0195] According to an embodiment of the present disclosure, the electronic device 100 may analyze the degree of accumulation of frequency feature values of each phoneme in speech input from a speaker (especially an unregistered speaker), and may generate a distinguishing sentence 714 including phonemes with a relatively low degree of accumulation or no accumulation at all based on the result of the analysis.
[0196] Reference Figure 7B , the phoneme / b / has a relatively lower degree of accumulation than the phonemes / d / and / f / , and the phoneme / g / is not accumulated. Therefore, the electronic device 100 may generate a distinguishing sentence 714 including the phonemes / b / and / g / . For example, the distinguishing sentence 714 may be "baggage". However, the subject that generates the distinguishing sentence 714 is not limited to the electronic device 100. According to an embodiment of the present disclosure, the server 200 may analyze the frequency feature value of each phoneme in the human voice received from the electronic device 100, and may generate a distinguishing sentence 714 including phonemes with a low degree of accumulation (in Figure 7B / b / in the example shown) and the non-accumulated phonemes (in Figure 7B In the embodiment shown, the server 200 may generate a distinguishing sentence 714 of / g / . In this case, the server 200 may transmit the generated distinguishing sentence 714 to the electronic device 100.
[0197] According to an embodiment of the present disclosure, the electronic device 100 may output a message 712 requesting the user to speak the generated distinguishing sentence 714. The electronic device 100 may display the message 712 requesting the user to speak the distinguishing sentence 714 on the display 150. However, the present invention is not limited thereto. According to an embodiment of the present disclosure, the electronic device 100 may output a message 712 requesting the user to speak the distinguishing sentence 714 through the speaker 140 (refer to Figure 2A and 2B ) outputs a voice message requesting the user to speak the distinguishing sentence 714.
[0198] Figure 7C is a flowchart of a method of generating a distinguishing sentence for identifying a speaker on a speaker characteristic graph based on a frequency feature value of each phoneme of an input voice and outputting a message requesting to speak the distinguishing sentence, performed by the electronic device 100 according to an embodiment of the present disclosure.
[0199] Reference Figure 7C In operation S710, the electronic device 100 may analyze the cumulative degree of the frequency feature value of each phoneme for the voice. According to an embodiment of the present disclosure, the electronic device 100 may receive the voice of an unregistered speaker, obtain the cumulative time information based on the frequency range of each phoneme in the input voice, and analyze the cumulative time information.
[0200] In operation S720, the electronic device 100 may generate a distinction sentence including phonemes having a relatively low degree of accumulation or not accumulated based on the result of the analysis performed in operation S810.
[0201] Operations S710 and S720 may be performed by the electronic device 100 , but are not limited thereto. Operations S710 and S720 may be performed by the server 200 .
[0202] In operation S730, the electronic device 100 may output a message requesting the user to speak a distinguishing sentence. According to an embodiment of the present disclosure, the electronic device 100 may output a voice message requesting the user to speak a distinguishing sentence through a speaker. According to an embodiment of the present disclosure, the electronic device 100 may output a message requesting the user to speak a distinguishing sentence on a display. When operations S710 and S720 are performed by the server 200, the electronic device 100 may receive a distinguishing sentence generated by the server 200 and output the received distinguishing sentence.
[0203] Figure 8 8 is a block diagram of components of a virtual assistant module 800 according to an embodiment of the present disclosure. The virtual assistant module 800 may be included in the electronic device 100, but is not limited thereto. According to an embodiment of the present disclosure, the server 200 may include the virtual assistant module 800.
[0204] The virtual assistant module 800 can generate distinguishing sentences.
[0205] Reference Figure 8 , the virtual assistant module 800 may include hardware modules, wherein the hardware modules include an automatic speech recognition (ASR) module 810, a natural language understanding (NLU) module 820, a dialogue management (DM) module 830, a natural language generation (NLG) module 840 and a text to speech (TTS) module 850.
[0206] The ASR module 810 may recognize a voice input from a user and convert the recognized voice into a computer-readable text. The ASR module 810 may analyze the cumulative degree of the frequency feature value of each phoneme for the input voice, and detect phonemes with a relatively low cumulative degree or no cumulative degree at all based on the result of the analysis. According to an embodiment of the present disclosure, the ASR module 810 may analyze the cumulative degree of the frequency feature value of each phoneme of each speaker.
[0207] When the ASR module 810 is included in the server 200, the electronic device 100 can send the voice received from the user to the server 200 in an audio file format. In this case, the server 200 can analyze the received voice and convert the received voice into a text form. Through this process, the server 200 can analyze the cumulative degree of the frequency feature value of each phoneme.
[0208] The NLU module 820 may perform natural language processing (NLP) in which, for example, words are extracted from the text converted by the ASR module 810 and the extracted words are converted into word vectors to generate a sentence vector.
[0209] The DM module 830 may generate a distinguishing sentence by receiving an input in a text form, and may generate a response related to the voice. The DM module 830 may generate a distinguishing sentence based on a cumulative degree for each phoneme analyzed by the ASR module 810.
[0210] The NLG module 840 may generate distinguished sentences in a natural language form.
[0211] The TTS module 850 may convert the generated distinguished sentences into speech.
[0212] The virtual assistant module 800 may output the distinguished sentences converted into speech by the TTS module 850 .
[0213] Fig. 9 900M is a diagram illustrating a method of predicting a relationship between a plurality of speakers based on a positional relationship between a plurality of clusters on a speaker feature graph 900M, performed by the electronic device 100 according to an embodiment of the present disclosure. Fig. 9 Speaker feature vectors are shown located on speaker feature map 900M. Fig. 9A two-dimensional speaker characteristic graph 900M is shown. However, this is for convenience of explanation. The speaker characteristic graph 900M of the present disclosure is not limited thereto and may be n-dimensional, where n is equal to or greater than 2.
[0214] Reference Fig. 9 The first cluster 911, the second cluster 912, the third cluster 921, and the fourth cluster 922 on the speaker feature graph 900M may be registered user identification information and stored in the memory 130 (refer to Figure 2A and Figure 2B ), and the fifth cluster 931 and the sixth cluster 932 may be clusters in which user identification information is not registered. For example, the first cluster 911 may be a registered cluster storing identification information of mother A, and the second cluster 912 may be a registered cluster storing identification information of son A (son of mother A). A plurality of speaker feature vectors 911a to 911n extracted from the voice input by mother A may be included in the first cluster 911, and a plurality of speaker feature vectors 912a to 912n extracted from the voice input by son A may be included in the second cluster 912.
[0215] The third cluster 921 may be a registered cluster storing identification information of mother B, and the fourth cluster 922 may be a registered cluster storing identification information of son B (son of mother B). A plurality of speaker feature vectors 921a to 921n extracted from the speech input of mother B may be included in the third cluster 921, and a plurality of speaker feature vectors 922a to 922n extracted from the speech input of son B may be included in the fourth cluster 922.
[0216] The first cluster 911 and the second cluster 912 may be separated from each other by a first distance d on the speaker feature map 900M. 1 , and may have a first angle θ relative to the X-axis 1 The third cluster 921 and the fourth cluster 922 may be separated from each other by a second distance d on the speaker feature graph 900M. 2 , and may have a second angle θ relative to the X-axis 2 position relationship.
[0217] The first angle θ 1 and the second angle θ 2 can be substantially the same as each other, and the first distance d 1 and the second distance d 2 According to an embodiment of the present disclosure, the electronic device 100 may analyze a large amount of human voice data by receiving voices from a plurality of speakers, and based on the result of learning, may recognize that in the case of a mother-child relationship, the clusters may have a specific distance d 1 and d2 and a specific angle θ 1 and θ 2 That is, the electronic device 100 can recognize that a specific situation such as a mother-child relationship may have a trend regarding the angle and distance between clusters. According to an embodiment of the present disclosure, the electronic device 100 can analyze a large amount of human voice data and learn the distance and directionality information between different clusters and the relationship matching information of the speakers corresponding to the clusters, and can store the learned matching information.
[0218] The electronic device 100 may predict the relationship between the plurality of speakers corresponding to the plurality of clusters based on the distance and directionality information between the plurality of clusters on the speaker feature graph 900M.
[0219] exist Fig. 9 In the illustrated embodiment, the electronic device 100 may receive speech from speaker 1, extract a fifth feature vector 931k from the vocal data, and map the extracted fifth feature vector 931k to the fifth cluster 931 on the speaker feature map 900M. In addition, the electronic device 100 may receive speech from speaker 2, extract a sixth feature vector 932k from the vocal data, and map the extracted sixth feature vector 932k to the sixth cluster 932 on the speaker feature map 900M. The electronic device 100 may obtain the distance d separating the fifth cluster 931 and the sixth cluster 932 from each other on the speaker feature map 900M. k and an angle θ formed by the fifth cluster 931 and the sixth cluster 932 with respect to the X-axis k Here, within the error range, the distance d k The first distance d 1 and the second distance d 2 The same, and the angle θ k The first angle θ 1 and the second angle θ 2 The electronic device 100 can predict that speaker 1 who generates a speech corresponding to the fifth feature vector 931k in the fifth cluster 931 and speaker 2 who generates a speech corresponding to the sixth feature vector 932k in the sixth cluster 932 have a parent-child relationship based on the result of learning the relationship information between the location information including the distance and directionality information between the clusters and the speaker relationship.
[0220] Despite Fig. 9, it is described that the electronic device 100 predicts the relationship between speakers based on the distance and directionality information between clusters, wherein the distance and directionality information between clusters is based on pre-registered speaker feature vectors and cluster learning, but the embodiments of the present disclosure are not limited thereto. According to an embodiment of the present disclosure, the server 200 may store the speaker feature graph 900M, and may predict the relationship between speakers from the input speech based on the pre-registered speaker feature vectors and the distance and directionality information between clusters.
[0221] Fig. 10A and Fig. 10B is a diagram illustrating a method of predicting a relationship between a plurality of speakers based on a positional relationship between a plurality of clusters on a speaker feature graph and providing a personalized service based on the predicted relationship, performed by an electronic device according to various embodiments of the present disclosure.
[0222] Fig. 10A Speaker feature vectors located on a speaker feature map according to an embodiment of the present disclosure are shown. Fig. 10A A two-dimensional speaker characteristic graph is shown, but this is for ease of explanation. The speaker characteristic graph of the present disclosure is not limited to a two-dimensional speaker characteristic graph, and may be an n-dimensional speaker characteristic graph, where n is equal to or greater than 2.
[0223] Reference Fig. 10A , the first cluster 1010, the second cluster 1020, and the third cluster 1030 on the speaker feature graph 1000M may be registered clusters in which user identification information is registered and stored. According to an embodiment of the present disclosure, identification information of father A, identification information of mother A, and identification information of grandmother A may be stored and registered in the first cluster 1010, the second cluster 1020, and the third cluster 1030, respectively.
[0224] The electronic device 100 can identify the specific positional relationship between the first cluster 1010, the second cluster 1020, and the third cluster 1030. According to an embodiment of the present disclosure, when the electronic device 100 receives voices from multiple speakers, analyzes a large amount of human voice data, and identifies the positional relationship of the clusters having a specific distance and a specific angle based on the analysis, the electronic device 100 can learn which relationships the speakers have and can store the results of the learning. Fig. 10A In the illustrated embodiment, father A and grandmother A have a mother-son relationship, and the third cluster 1030 may be vertically located below the first cluster 1010 , and may have a directivity having a specific angle with respect to the first cluster 1010 .
[0225] The electronic device 100 may extract a speaker feature vector from the newly input user's voice based on the distance and directionality information between clusters learned from the pre-registered speaker feature vectors and clusters, obtain the distance and directionality information between the pre-registered clusters and the newly extracted speaker feature vector, and predict the relationship between the speaker corresponding to the pre-registered cluster and the speaker generating the new voice based on the obtained distance and directionality information. Fig. 10A In the illustrated embodiment, the electronic device 100 may receive a voice from a user, extract a fourth feature vector 1033 from the vocal data, and map the extracted fourth feature vector 1033 to a fourth cluster 1032 on the speaker feature map 1000M. The electronic device 100 may identify the fifth cluster 1012 located at a specific distance and a specific angle relative to the fourth cluster 1032 where the fourth feature vector 1033 is located. The electronic device 100 may identify that the distance and angle at which the fifth cluster 1012 and the fourth cluster 1032 are located relative to each other are the same as the distance and angle between the pre-registered first cluster 1010 and the third cluster 1030. The electronic device 100 may predict, based on the result of the recognition, that a speaker who generates a voice corresponding to the fifth feature vector 1013 included in the fifth cluster 1012 and a speaker who generates a voice corresponding to the fourth feature vector 1033 included in the fourth cluster 1032 have a parent-child relationship. For example, when the identification information of father B is registered in the fifth cluster 1012 , the electronic device 100 may predict that the fourth cluster 1032 corresponds to the cluster of the voice of grandmother B (mother of father B).
[0226] According to an embodiment of the present disclosure, the electronic device 100 can identify the user of the newly input voice based on the predicted relationship between the speakers, and provide a personalized service synchronized with the identified user. Fig. 10B , the electronic device 100 may receive a voice input from Grandma 2, map clusters by extracting a speaker feature vector from the voice of Grandma 2, and obtain distance and directionality information by comparing the positions of the mapped clusters with the positions of pre-registered clusters on the speaker feature map. The electronic device 100 may predict the relationship between Grandma 2 and James Hong 3 as a mother-child relationship based on the obtained angle and directionality information.
[0227] According to an embodiment of the present disclosure, when the electronic device 100 receives a voice message “Please call my son” from the grandmother 2, the electronic device 100 may recognize the grandmother 2 and output the question “Are you the mother of James Hong?” When the grandmother 2 answers “Yes, that’s right”, the electronic device 100 may call James Hong 3 and output the message “Call Mr. Hong” at the same time.
[0228] For example, when the main user of the electronic device 100 is James Hong 3, and the son of James Hong 3 (CheolSu), not James Hong 3, is stored as "son" in the address list of the electronic device 100, the electronic device 100 according to the prior art may not predict the relationship between the speaker and the main user of the electronic device 100, and therefore, the electronic device 100 may make a telephone call to Cheol Su stored in the address list. However, the electronic device 100 according to an embodiment of the present disclosure may recognize that the speaker is Grandma 2 and predict the mother-son relationship between Grandma 2 and James Hong, and therefore, in response to the voice command of "Please call my son", the electronic device 100 may call James Hong3, not Cheol Su. Therefore, the electronic device 100 according to an embodiment of the present disclosure can provide personalized services synchronized with each user, and thus can improve user convenience.
[0229] In addition to the above examples, the electronic device 100 may also provide various personalized services, such as targeted advertising, based on the predicted relationship of the speakers. Fig. 10A and Fig. 10B The embodiments of the present invention may be executed not only by the electronic device 100 but also by the server 200 .
[0230] Fig.11 is a flowchart of a method performed by the electronic device 100 to predict a relationship between a plurality of speakers based on a positional relationship between a plurality of clusters on a speaker feature graph and to provide a personalized service based on the predicted relationship according to an embodiment of the present disclosure.
[0231] Reference Fig.11 , in operation S1110, the electronic device 100 may predict the relationship between the multiple speakers corresponding to the multiple clusters based on the distance and directionality information between the multiple clusters on the speaker feature map. According to an embodiment of the present disclosure, when the multiple speakers have a specific relationship, the electronic device 100 may receive voices from the multiple speakers, analyze and learn a large amount of vocal data, and identify the trend of positional relationships between the clusters having a specific distance and a specific angle. According to an embodiment of the present disclosure, the electronic device 100 may learn and store the distance and directionality information between different clusters and the relationship matching information of the speakers.
[0232] In operation S1120, the electronic device 100 may identify the speaker of the newly input voice based on the predicted relationship between the speakers. According to an embodiment of the present disclosure, the electronic device 100 may receive the voice of a speaker using the electronic device 100 (i.e., the user of the electronic device 100), extract a speaker feature vector from the received voice, locate the speaker feature vector on a speaker feature map, and identify the positional relationship between the speaker feature vector and a cluster of other pre-registered speaker feature vectors. According to an embodiment of the present disclosure, when the cluster to which the speaker feature vector of the newly input voice of the recognized user is mapped is located to be separated by a specific distance from other pre-registered clusters and has a specific angle of directionality relative to the other pre-registered clusters, the electronic device 100 may identify the speaker based on the distance and directionality information.
[0233] In operation S1130, the electronic device 100 may provide a personalized service synchronized with the identified speaker. According to an embodiment of the present disclosure, the electronic device 100 may identify the speaker of the newly received voice and provide a synchronization service based on the relationship between the speaker and another stored speaker. For example, the electronic device 100 may provide a target advertisement corresponding to the identified speaker.
[0234] Fig.11 The illustrated operations S1110 to S1130 may be performed not only by the electronic device 100 but also by the server 200 .
[0235] Fig. 12A and Fig. 12B 1200M is a diagram illustrating a method of predicting a user state based on a positional relationship between a plurality of speaker feature vectors located in a single cluster on a speaker feature graph 1200M, performed by the electronic device 100 according to various embodiments of the present disclosure. Fig. 12A and 12B shows speaker feature vectors located on speaker feature map 1200M, and Fig. 12A and 12B A two-dimensional speaker characteristic graph 1200M is shown, but this is for convenience of explanation. The speaker characteristic graph 1200M of the present disclosure is not limited to a two-dimensional speaker characteristic graph, and may be an n-dimensional speaker characteristic graph, where n is equal to or greater than 2.
[0236] Reference Fig. 12A, the first cluster 1201 and the second cluster 1202 on the speaker feature graph 1200M may both be registered clusters in which user identification information is registered and stored. According to an embodiment of the present disclosure, the first cluster 1201 may be a cluster of speaker A, and the second cluster 1202 may be a cluster of speaker B. The first cluster 1201 may include a first first subcluster 1201-1 and a first second subcluster 1201-2, and the second cluster 1202 may include a second first subcluster 1202-1 and a second second subcluster 1202-2.
[0237] exist Fig. 12A In the illustrated embodiment, all of the plurality of speaker feature vectors included in the first cluster 1201 may be speaker feature vectors extracted from the speech of speaker A. The plurality of speaker feature vectors included in the first first subcluster 1201-1 may be speaker feature vectors extracted from the speech of speaker A when speaker A is healthy, and the plurality of speaker feature vectors included in the first second subcluster 1201-2 may be speaker feature vectors extracted from the speech of speaker A when speaker A has a cold. The first first subcluster 1201-1 and the first second subcluster 1201-2 in the first cluster 1201 may be separated from each other by a specific distance and may form a specific angle θ with respect to the X-axis.
[0238] Likewise, all of the plurality of speaker feature vectors included in the second cluster 1202 are speaker feature vectors extracted from the speech of speaker B, wherein the plurality of speaker feature vectors included in the second first subcluster 1202-1 are speaker feature vectors extracted from the speech of speaker B when speaker B is healthy, and the plurality of speaker feature vectors included in the second second subcluster 1202-2 are speaker feature vectors extracted from the speech of speaker B when speaker B has a cold. The second first subcluster 1202-1 and the second second subcluster 1202-2 in the second cluster 1202 may be separated from each other by a specific distance and may form a specific angle θ with respect to the X-axis. 2 .
[0239] According to an embodiment of the present disclosure, the electronic device 100 may receive speech from a plurality of speakers, analyze and learn a large amount of vocal data, form a plurality of subclusters by grouping a plurality of speaker feature vectors clustered within a predetermined distance in a single cluster based on the result of the learning, and identify a connection trend between the positional relationship between the subclusters and information about the state of the speaker. For example, when a plurality of subclusters in a single cluster are separated from each other by a specific distance and have a directivity of a predetermined angle θ, the electronic device 100 may obtain correlation information that the speaker has a cold or that the speaker is in a poor physical state. The electronic device 100 may store the obtained correlation information between the positional relationship between the subclusters and the state of the speaker.
[0240] Reference Fig. 12B , the electronic device 100 may receive a new speech of speaker C and extract a speaker feature vector 1203-k from the vocal data of the received speech. The electronic device 100 may map the extracted speaker feature vector 1203-k to the third cluster 1203. The electronic device 100 may predict the health status of speaker C based on the positional relationship between the third first subcluster 1203-1 in the third cluster 1203 and the newly extracted speaker feature vector 1203-k.
[0241] exist Fig. 12B In the illustrated embodiment, when speaker C is healthy, the speaker feature vector extracted from the speech may be located in the third first subcluster 1203-1, and when speaker C has a cold, the speaker feature vector 1203-k extracted from the speech may be mapped to a region having an angle k (θ ) relative to the X-axis based on the third first subcluster 1203-1. k ) is a directional position. The electronic device 100 can k ) is compared with the angle θ pre-stored for the health state, and when the angle k(θ k ) is the same as the angle θ, it can be predicted that speaker C currently has a cold.
[0242] That is, the electronic device 100 may extract a speaker feature vector 1203-k from a new input speech of speaker C, compare the position of the extracted speaker feature vector 1203-k with the position of the speaker feature vector in the third first subcluster 1203-1 in the third cluster 1203 on the speaker feature graph 1200M, where the speaker feature vector 1203-k is mapped in the third cluster 1203, identify the directionality, and predict the health status of speaker C based on the identified directionality.
[0243] According to an embodiment of the present disclosure, the electronic device 100 may store the Figure 2A and Figure 2B ) in the message in the electronic device 100. For example, the electronic device 100 may recognize that the speaker C has caught a cold, and may output a comforting message such as "Be careful not to make your cold worse," or provide the speaker C with information about the location of a clinic, pharmacy, etc. related to the cold.
[0244] Fig. 12A and Fig. 12B The embodiments of the present invention may be executed not only by the electronic device 100 but also by the server 200 .
[0245] Fig.13is a flowchart of a method for predicting a user state based on a positional relationship between a plurality of speaker feature vectors in a single cluster on a speaker feature map, performed by the electronic device 100 according to an embodiment of the present disclosure.
[0246] Reference Fig.13 , in operation S1310, the electronic device 100 may identify directionality by comparing the position of a first feature vector of a voice received from a user with the position of a second feature vector of a single cluster on a speaker feature map. According to an embodiment of the present disclosure, the first feature vector may be mapped in a first cluster, and the first cluster may include a plurality of subclusters. The plurality of subclusters may be formed by the voice of one speaker, and may be a subset of speaker feature vectors extracted from a human voice that changes according to a state of the speaker, such as health or age, etc. According to an embodiment of the present disclosure, the electronic device 100 may identify directionality by comparing the position of a first feature vector extracted from a voice received from a user with the positions of a plurality of subclusters in the first cluster.
[0247] In operation S1320, the electronic device 100 may predict the health state of the user based on the identified directionality. The electronic device 100 may store information about the correlation between the directionality between multiple subclusters in a single cluster (i.e., the angle formed between the multiple subclusters) and the speaker state. For example, the electronic device 100 may store correlation information that the speaker has a cold when the directionality formed between the multiple subclusters corresponds to the first angle. The electronic device 100 may predict which health state of the user is indicated by the angle formed by the first feature vector extracted from the voice input in operation S1310 and the subclusters by using the stored correlation information. For example, when the first feature vector forms a first angle with a subcluster that is a subgroup of the speaker feature vector extracted when the user is healthy, the electronic device 100 may predict that the user currently has a cold.
[0248] In operation S1330, the electronic device 100 may output a message corresponding to the predicted health status of the user. According to an embodiment of the present disclosure, when it is predicted that the user currently has a cold, the electronic device 100 may output a message corresponding to the predicted health status of the user through the speaker 140 (see Figure 2A and 2B ) outputs a comforting message such as "Beware of catching a cold", or through the display 150 (see Figure 2A and 2B ) outputs information related to colds about the locations of clinics, pharmacies, etc.
[0249] The electronic device 100 according to an embodiment of the present disclosure may predict the user's current health status via the user's voice and output a relevant message, thereby providing emotional comfort to the user in addition to simple information, so that the user feels the human touch of the electronic device 100 .
[0250] Fig.13 The illustrated operations S1310 to S1330 may be performed not only by the electronic device 100 but also by the server 200 .
[0251] Fig.14 is a block diagram of components of an electronic device 1000 according to an embodiment of the present disclosure. Fig.14 The electronic device 1000 shown may include the Figure 2A , Figure 2B , FIG. 3A to FIG. 3D , FIG. 4A to FIG. 4D , Figure 5 , FIG. 6A to FIG. 6C , 7A to 7C , Figure 8 , Fig. 9 , Fig. 10A , Fig. 10B , Fig.11 , Fig. 12A , Fig. 12B and Fig.13 The electronic device 100 described above has the same components. For example, Fig.14 The components shown in the figure include a controller 1300 and a Figure 2A and 2B The controller 120 is the same as shown, and the sound output device 1220 is the same as Figure 2A and 2B The speaker 140 is the same as shown, and the display 1210 is the same as Figure 2A and 2B The display 150 shown is the same. Therefore, no repeated description will be given.
[0252] Fig.14 The electronic device 1000 shown may execute reference Figure 2A , Figure 2B , FIG. 3A to FIG. 3D , FIG. 4A to FIG. 4D , Figure 5 , FIG. 6A to FIG. 6C , 7A to 7C , Figure 8 , Fig. 9 , Fig. 10A , Fig. 10B , Fig.11 , Fig. 12A , Fig. 12B and Fig.13 Therefore, hereinafter, components of the electronic device 1000 that have not been described so far will be described.
[0253] Reference Fig.14 , the electronic device 1000 may include a user inputter 1100 , an outputter 1200 , a controller 1300 , a sensor 1400 , a communicator 1500 , an A / V inputter 1600 , and a memory 1700 .
[0254] The user input 1100 may refer to a device used by a user to input data to control the electronic device 1000. For example, the user input 1100 may include a keyboard, a dome switch, a touch pad (touch capacitance method, piezoresistive method, infrared detection method, surface ultrasonic conduction method, overall tension measurement method, piezoelectric effect method, etc.), a scroll wheel, a micro switch, etc., but is not limited thereto. The user input 1100 may receive a user input required for generating communication information to be provided to the user.
[0255] The outputter 1200 may output an audio signal, a video signal, or a vibration signal, and may include a display 1210 , a sound outputter 1220 , and a vibration motor 1230 .
[0256] The vibration motor 1230 may output a vibration signal. For example, the vibration motor 1230 may output a vibration signal corresponding to the output of audio data or video data (eg, a call signal reception sound, a message reception sound, etc.).
[0257] The sensor 1400 may sense a state of the electronic device 1000 or a state around the electronic device 1000 and transmit the sensed information to the controller 1300 .
[0258] The sensor 1400 may include at least one of a magnetic sensor 1410, an acceleration sensor 1420, a temperature / humidity sensor 1430, an infrared sensor 1440, a gyro sensor 1450, a position sensor (e.g., a global positioning sensor (GPS)) 1460, an air pressure sensor 1470, a proximity sensor 1480, or a red, green, and blue (RGB) sensor (brightness sensor) 1490, but is not limited thereto. A person of ordinary skill in the art may intuitively infer the function of each sensor based on its name, and thus a detailed description thereof will not be given.
[0259] The communicator 1500 may include components configured to enable communication with other devices. For example, the communicator 1500 may include a short-range wireless communicator 1510 , a mobile communicator 1520 , and a broadcast receiver 1530 .
[0260] The short-range wireless communicator 151 may include a Bluetooth communicator, a BLE communicator, NFC, a wireless LAN (WLAN) (or Wi-Fi) communicator, a Zigbee communicator, an infrared data association (IrDA) communicator, a WFD communicator, an ultra-wideband (UWB) communicator, an Ant+ communicator, etc., but is not limited thereto.
[0261] The mobile communicator 1520 may transceive wireless signals with at least one of a base station, an external terminal or a server via a mobile communication network. Here, the wireless signals may include a voice call signal, a video phone call signal or various types of data based on transceiver of text / multimedia messages.
[0262] The broadcast receiver 1530 may receive a broadcast signal and / or broadcast related information from the outside via a broadcast channel. The broadcast channel may include a satellite channel and a ground wave channel. According to an embodiment of the present disclosure, the electronic device 1000 may not include the broadcast receiver 1530.
[0263] In addition, the communicator 1500 can send information required to generate communication information to be provided to the first user to the second communication electronic device 3000, other devices and servers, and receive information required to generate communication information to be provided to the first user from the second communication electronic device 3000, other devices and servers.
[0264] The A / V input 1600 may be configured to input an audio signal or a video signal, and may include a camera 1610 and a microphone 1620. The camera 1610 may obtain an image frame, such as a still image or a video, via an image sensor in a video phone mode or a capture mode. The image captured by the image sensor may be processed by the controller 1300 or another image processor (not shown).
[0265] The image frame processed by the camera 1610 may be stored in the memory 1700, or may be transmitted to the outside via the communicator 1500. According to an embodiment of the terminal, the camera 1610 may include two cameras.
[0266] The microphone 1620 may receive an external sound signal and process the received external sound signal into electric voice data. For example, the microphone 1620 may receive a sound signal from an external device or a speaker. The microphone 1620 may use various noise removal algorithms to remove noise generated in the process of receiving the external sound signal.
[0267] The memory 1700 may store a program for processing and controlling the operation of the controller 1300 , and may store data input or output to or from the electronic device 1000 .
[0268] Memory 1700 may include at least one type of storage medium among flash memory, hard disk memory, multimedia card micro memory, card type memory (e.g., SD or XD memory), RAM, SRAM, ROM, EEPROM, PROM, magnetic memory, magnetic disk and optical disk.
[0269] The programs stored in the memory 1700 may be classified into a plurality of modules according to their functions. For example, the programs may be classified into a UI module 1710, a touch screen module 1720, a notification module 1730, and the like.
[0270] The UI module 1710 may provide a dedicated UI or a graphical user interface (GUI) or the like synchronized with the electronic device 1000 according to an application. The touch screen module 1720 may sense a touch gesture of a user on the touch screen, and may send information about the touch gesture to the controller 1300. The touch screen module 1720 according to one or more embodiments of the present disclosure may recognize and analyze a touch code. The touch screen module 1720 may be formed as additional hardware including a controller.
[0271] The notification module 1730 may generate a signal for notifying the occurrence of an event of the electronic device 1000. Example events occurring in the electronic device 1000 may include call signal reception, message reception, key signal input, schedule notification, etc. The notification module 1730 may output the notification signal in the form of a video signal via the display 1210, in the form of an audio signal via the sound outputter 1220, and in the form of a vibration signal via the vibration motor 1230.
[0272] The electronic devices 10, 100 and 1000 according to the present disclosure may be implemented as hardware components, software components and / or a combination of hardware components and software components. For example, the electronic devices 10, 100 and 1000 according to the embodiments of the present disclosure may be implemented by using a processor, an arithmetic logic unit (ALU), an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a microcomputer, a microprocessor or one or more general-purpose computers or special-purpose computers (such as devices capable of executing and responding to instructions).
[0273] The software components may include computer programs, codes, instructions, or a combination of one or more thereof, and may configure the processing device to operate as desired or command the processing device individually or collectively.
[0274] The software component may be implemented as a computer program including instructions stored in a computer-readable storage medium. The computer-readable storage medium may include, for example, magnetic storage media (e.g., ROM, RAM, floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM (CD-ROM), digital versatile disk (DVD), etc.). The computer-readable recording medium may be distributed in a computer system connected in a network, and the computer-readable code may be stored and executed in a distributed manner. The medium may be computer-readable, may be stored in a memory, and may be executed by a processor.
[0275] The computer may be a device configured to call instructions stored in a storage medium and perform operations according to embodiments of the present disclosure in response to the called instructions, and may include the electronic devices 10 , 100 , and 1000 according to embodiments of the present disclosure.
[0276] The computer-readable storage medium may be provided as a non-transitory storage medium. Here, the term "non-transitory" merely means that the storage medium does not include a signal and is tangible, and the term does not distinguish between semi-permanent storage and temporary storage of data in the storage medium.
[0277] In addition, the electronic devices 10, 100, and 1000 or the method according to the embodiments of the present disclosure may be included in a computer program product. The computer program product is a product that can be purchased between a seller and a buyer.
[0278] The computer program product may include a software program and a computer-readable storage medium storing the software program. For example, the computer program product may include a computer program developed by a manufacturer of the electronic devices 10, 100, and 1000 or an electronic market (e.g., Google Play TM For electronic distribution, at least a portion of the software program may be stored in a storage medium or temporarily generated. In this case, the storage medium may be a storage medium of a manufacturer's server, an electronic market's server, or a broadcast server that temporarily stores the software program.
[0279] In a system including a server and a terminal (e.g., an ultrasonic diagnostic device), a computer program product may include a storage medium of the server or a storage medium of the terminal. Alternatively, when there is a third device (e.g., a smart phone) connected to the server or the terminal for communication, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a software program sent from the server to the terminal or the third device or from the third device to the terminal.
[0280] In this case, one of the server, the terminal, and the third device may perform the method according to the embodiment of the present disclosure by executing the computer program product. Optionally, at least two of the server, the terminal, and the third device may perform the method according to the embodiment of the present disclosure in a distributed manner by executing the computer program product.
[0281] For example, a server (eg, a cloud server or an AI server) may execute a computer program product stored in the server and control a terminal connected to the server for communication to execute a method according to an embodiment of the present disclosure.
[0282] As another example, the third device may execute the computer program product and control a terminal connected to the third device for communication to perform a method according to an embodiment of the present disclosure.
[0283] When the third device executes the computer program product, the third device may download the computer program product from the server and execute the downloaded computer program product. Alternatively, the third device may execute the computer program product provided in a free loading state and execute the method according to an embodiment of the present disclosure.
[0284] In addition, although the embodiments of the present disclosure have been shown and described above, the present disclosure is not limited to the above-mentioned specific embodiments. Without departing from the scope of the present disclosure as claimed in the claims, a person of ordinary skill in the art may make various modified embodiments, and these modifications should not be understood separately from the technical spirit or prospect of the present disclosure.
[0285] Although the present disclosure has been shown and described with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents. For example, the described techniques may be performed in a different order than the described methods, and / or the components of the described electronic devices, structures, circuits, etc. may be combined or integrated in a different form than the described methods, or may be replaced or substituted by other components or equivalents to achieve appropriate results.
Claims
1. A method for processing human voice data of speech, the method include: extracting a signal value of each phoneme in vocal data of a speech received from a speaker, and calculating a signal energy value by accumulating an amount of the signal value of each phoneme in each of a plurality of frequency ranges; extracting a feature value from the signal energy value accumulated in each of the plurality of frequency ranges of each phoneme when a difference between the accumulated signal energy values of the same phoneme in the same frequency range of the speech of the plurality of speakers is equal to or greater than a predetermined value; Convert the extracted feature values into speaker feature vectors; Generating a speaker feature map by locating a speaker feature vector at a specific position on a multi-dimensional vector space; forming a plurality of clusters indicating characteristics of human voices of the plurality of speakers by grouping at least one speaker feature vector located on the speaker feature map; as well as The plurality of speakers are categorized according to the plurality of clusters.
2. The method according to claim 1, in, The step of forming the plurality of clusters comprises: The speaker feature vectors on the speaker feature map are classified by training a deep neural network (DNN) model, wherein the DNN model is trained by applying the speaker feature vectors as input to the DNN model and using the index of the speaker from among the plurality of speakers as output.
3. The method according to claim 1, further comprising: include: receiving speech inputting recognition information from a user; Extracting a speaker feature vector of the user from the vocal data of the speech; mapping the extracted speaker feature vector to a cluster among the plurality of clusters on the speaker feature map; as well as The identification information of the user is registered in the cluster among the plurality of clusters.
4. The method according to claim 3, further comprising: include: Before receiving the identification information from the user, the user is requested to speak a distinguishing sentence for registering the identification information of the user.
5. The method according to claim 1, in, The step of forming the plurality of clusters includes forming the plurality of clusters by grouping the at least one speaker feature vector located within a predetermined distance from each other on the speaker feature map.
6. The method according to claim 1, further comprising: include: Relationships between the plurality of speakers corresponding to the plurality of clusters are predicted based on distance and directionality information between the plurality of clusters on the speaker feature map.
7. An electronic device for processing human voice data of speech, the electronic device include: A speech input device receives speech from a speaker; The controller is configured as: extracting a signal value of each phoneme in vocal data of a speech received from a speaker, and calculating a signal energy value by accumulating an amount of the signal value of each phoneme in each of a plurality of frequency ranges, extracting a feature value from the signal energy value accumulated in each of the plurality of frequency ranges of each phoneme when a difference between the accumulated signal energy values of the same phoneme in the same frequency range of the voices of the plurality of speakers is equal to or greater than a predetermined value, Convert the extracted feature values into speaker feature vectors, The speaker feature map is generated by locating the speaker feature vector at a specific position in the multidimensional vector space. forming a plurality of clusters indicating characteristics of the human voices of the plurality of speakers by grouping at least one speaker feature vector located on the speaker feature map, and classifying the plurality of speakers according to the formed plurality of clusters; as well as A memory stores the speaker feature map.
8. The electronic device according to claim 7, in, The controller is also configured to classify the speaker feature vectors on the speaker feature map by training a deep neural network (DNN) model, wherein the DNN model is trained by applying the speaker feature vectors as input to the DNN model and using the index of the speaker from the multiple speakers as output.
9. The electronic device according to claim 7, in, The voice input device receives a voice input of recognition information from a user, and The controller is further configured as follows: Extracting a speaker feature vector of the user from the vocal data of the speech, mapping the extracted speaker feature vector to a cluster among the plurality of clusters on the speaker feature map, and The identification information of the user is registered in the cluster among the plurality of clusters.
10. The electronic device according to claim 9, further comprising: include: Speaker, output voice message, The controller is further configured to control the speaker to output a voice message requesting the user to speak a distinguishing sentence for registering the user's identification information.
11. The electronic device according to claim 7, in, The controller is further configured to form the plurality of clusters by grouping the at least one speaker feature vector located within a predetermined distance from each other on the speaker feature map.
12. The electronic device according to claim 7, in, The controller is also configured to identify directionality by comparing the position of a first feature vector of speech received by the speech inputter with the position of a second feature vector in the cluster to which the first feature vector is mapped on the speaker feature map, and predict the user's health status based on the identified directionality.
13. A computer program product, comprising a non-transitory computer-readable recording medium, the non-transitory computer-readable recording medium comprising instructions for: The control electronics receives speech from a speaker; extracting a signal value of each phoneme in vocal data of a speech received from a speaker, and calculating a signal energy value by accumulating an amount of the signal value of each phoneme in each of a plurality of frequency ranges; extracting a feature value from the signal energy value accumulated in each of the plurality of frequency ranges of each phoneme when a difference between the accumulated signal energy values of the same phoneme in the same frequency range of the speech of the plurality of speakers is equal to or greater than a predetermined value; Convert the extracted feature values into speaker feature vectors; Generating a speaker feature map by locating a speaker feature vector at a specific position on a multi-dimensional vector space; forming a plurality of clusters indicating characteristics of human voices of the plurality of speakers by grouping at least one speaker feature vector located on the speaker feature map; as well as The plurality of speakers are categorized according to the plurality of clusters.
