Systems and methods of voiceprint authentication and interpolation

Abstract

To provide a system and method of authenticating a user to interpolate user preference embeddings.SOLUTION: The present system includes: generating, using a neural network trained to generate features based on training data including human voices spoken by a plurality of past speakers inside a vehicle, input features based on a human voice of a current speaker inside the vehicle; and calculating similarities between an input vector of the input features and past vectors in voiceprints of one or more enrolled users. After determining that a similarity between the input vector and at least one past vector in a voiceprint of an identified user is less than a threshold similarity, the system authenticates the current speaker as the identified user, calculates a probabilistic notion based on the similarity, and applies the probabilistic notion to interpolate between downstream user preference embeddings associated with the identified user.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] TECHNICAL FIELD The embodiments described herein relate generally to voice user interface applications, and more specifically to vehicle voiceprint recognition and authentication. [Background technology]

[0002] Modern vehicles use services that require authentication, such as remote start, vehicle lock / unlock, valet, and geofencing. Authentication is an important security feature to identify an entity for services that the vehicle may provide. Once authenticated, the vehicle may grant the user permission to access certain services or data. Thus, a need exists for an accurate and efficient system of authentication to access vehicle systems. Summary of the Invention

[0003] In one embodiment, the method may include generating input features based on a human voice of a current speaker inside the vehicle using a neural network trained to generate features based on training data comprising human voices spoken by multiple past speakers inside the vehicle, and calculating a similarity between the input vector of the input features and past vectors in the voiceprints of one or more enrolled users. After determining that the similarity between the input vector and at least one past vector in the voiceprints of the certified users is less than a threshold similarity, the method includes authenticating the current speaker as the certified user, calculating a probabilistic concept based on the similarity, and applying the probabilistic concept to interpolate between downstream user preference embeddings associated with the certified users.

[0004] In another embodiment, the system includes a controller that generates input features based on the human voice of a current speaker inside the vehicle using a neural network trained to generate features based on training data comprising human voices spoken by multiple past speakers inside the vehicle, and calculates a similarity between the input vector of input features and past vectors in the voiceprints of one or more enrolled users. After determining that the similarity between the input vector and at least one past vector in the voiceprints of the certified users is less than a threshold similarity, the controller may authenticate the current speaker as a certified user, calculate a probabilistic concept based on the similarity, and apply the probabilistic concept to interpolate between downstream user preference embeddings associated with the certified users.

[0005] These and additional features provided by the embodiments described herein will be more fully understood when considered in conjunction with the drawings and the following detailed description. [Brief description of the drawings]

[0006] The embodiments set forth in the drawings are illustrative and exemplary in nature and are not intended to limit the subject matter defined by the claims. The following detailed description of illustrative embodiments can be understood when read in conjunction with the following drawings, in which like structure is designated with like reference numerals and in which:

[0007] [Figure 1] FIG. 1 depicts an exemplary, non-limiting device of a voiceprint authentication system of the present disclosure, according to one or more embodiments described and illustrated herein. [Figure 2A] FIG. 1 depicts an example block diagram of an example implementation of a voice feature module, a similarity module, and an authentication module of the present disclosure, according to one or more embodiments described and illustrated in this specification. [Figure 2B]FIG. 1 depicts an example block diagram of an example implementation of an authentication module for authentication and generation of probabilistic concepts for weighting and interpolation, according to one or more embodiments described and illustrated in this specification. [Diagram 3] FIG. 2 depicts an example block diagram of a neural network trained and applied in the voiceprint authentication system of the present disclosure, according to one or more embodiments described and illustrated herein. [Figure 4] FIG. 2 depicts an exemplary interior view of a vehicle equipped with a voiceprint authentication system of the present disclosure, according to one or more embodiments described and illustrated herein. [Figure 5A] FIG. 2 depicts exemplary original clusters for voiceprints of multiple users in the voiceprint authentication system of the present disclosure, according to one or more embodiments described and illustrated herein. [Figure 5B] FIG. 2 depicts exemplary reduced clusters for voiceprints of multiple users in a voiceprint authentication system of the present disclosure, according to one or more embodiments described and illustrated herein. [Figure 6] FIG. 1 depicts a flow chart describing the operation of the disclosed voiceprint authentication system for voiceprint authentication and interpolation, according to one or more embodiments described and illustrated herein. [Figure 7] FIG. 1 depicts a flow chart describing operations for a neural network of the voiceprint authentication system of the present disclosure, according to one or more embodiments described and illustrated herein. [Figure 8] FIG. 1 depicts a flowchart listing steps of a method for authenticating a speaker and applying the probabilistic concepts of interpolation of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The embodiments disclosed herein are directed to a method and system for voiceprint authentication and interpolation. The system creates a voiceprint for a user to enable authentication of the user from the user's voice. Modern vehicles offer functions and services where authentication of registered users is performed before the user can use the functions and services. However, methods of authentication such as passwords or PIN codes can be easily compromised and inconvenient or impractical to use. Voiceprint authentication overcomes these problems by providing a more secure and convenient method for authenticating users of a vehicle. Voiceprint authentication is based on unique characteristics about the human voice that are difficult to forge, making voiceprint authentication a secure method of authentication. Voiceprint authentication is convenient because users do not need to remember any passwords or PIN codes, but only need to speak into the vehicle's microphone using the user's own voice. Furthermore, voiceprint authentication is flexible for a variety of purposes, such as preventing unauthorized vehicle use, tracking vehicle use, and monitoring multiple fleets of vehicles.

[0009] The voice print authentication disclosed herein may use active voice biometrics or passive voice biometrics. Active approaches require an explicit voice enrollment process that is guided by the user's choice. Active voice biometric approaches require the user to explicitly establish their unique voice print by reciting a predefined script multiple times. Each time the user uses active authentication, the user must say a passphrase. Active authentication systems compare the user's voice to a recorded script. The user is fully aware of the authentication process.

[0010] Passive techniques infer an anonymized user identity based on speech occurring in the vehicle. Passive voice authentication does not require a specific passphrase to be said. A user may engage in normal conversation to trigger passive authentication, which may continually re-identify and re-authenticate the user. The user's voiceprint may be recorded in the system, including recordings at initial enrollment and / or continuous recordings after enrollment. As the user speaks in the vehicle, the system compares the user's speech to the voiceprint to verify the speech, regardless of what the user is saying.

[0011] The embodiments disclosed herein include passive enrollment, which is a background process that occurs again. During the process, speech samples are embedded and stored, and unsupervised clustering is performed at regular intervals. After collecting and processing several voice samples, the voice authentication system creates an embedding of features with high confidence that the embedding belongs to the same speaker. The voice authentication system may further refine the embedding boundaries for existing enrolled users. The voice authentication system may have the ability to create new embeddings that are assigned to anonymous non-users if the detected voiceprint does not belong to any known user. The passively enrolled genuine embeddings may be used to improve the user's experience with their in-car virtual assistant.

[0012] Based on the speech and voiceprint comparison, the system may further generate probabilistic concepts based on their similarity and use the probabilistic concepts in downstream "user preference" embeddings that require authentication. The use of probabilistic concepts in downstream embeddings improves the robustness of user preference modeling, for example by reducing noise levels and capturing variability in the user preference embeddings, improving the accuracy of recommendations.

[0013] Returning to the figures, FIG. 1 depicts an illustrative embodiment of a voiceprint authentication system 100. The voiceprint authentication system 100 may include a voice feature module 322, a similarity module 332, and an authentication module 342. The voiceprint authentication system 100 depicted in FIG. 3 includes a vehicle-mounted computing device that includes a controller 101. The controller 101 may further include various components, such as a memory 302, a processor 304, input / output hardware 305, network interface hardware 306, a data storage component 307, a local interface 303, and a voice sensor 402. The voiceprint authentication system 100 may further include a button 404 (e.g., as shown in FIG. 4) and a touch screen 406 (e.g., as shown in FIG. 4).

[0014] The controller 101 may be any device or combination of components comprising a processor 304 and a memory 302, such as a non-transitory computer-readable memory. The processor 304 may be any device capable of executing a set of machine-readable instructions stored in a non-transitory computer-readable memory. Thus, the processor 304 may be an electrical controller, an integrated circuit, a microchip, a computer, or any other computing device. The processor 304 may include any processing component configured to receive and execute programming instructions (such as from a data storage component 307 and / or a memory component 302). The instructions may be in the form of a set of machine-readable instructions stored in the data storage component 307 and / or the memory component 302. The processor 304 is communicatively connected to other components of the controller 101 by a local interface 303. Thus, the local interface 303 may communicatively connect any number of processors 304 to each other to enable the components connected to the local interface 303 to operate in a distributed computing environment. The local interface 303 may be implemented as a bus or other interface to facilitate communication between the components of the controller 101. In some embodiments, each of the components may operate as a node that may transmit and / or receive data. Although the embodiment depicted in FIG. 1 includes a single processor 304, other embodiments may include more than one processor.

[0015] The memory 302 (e.g., a non-transitory computer-readable memory component) may comprise a RAM, a ROM, a flash memory, a hard drive, or any non-transitory memory device capable of storing machine-readable instructions such that the machine-readable instructions may be accessed and executed by the processor 304. The machine-readable instruction set may comprise any programming language of any generation (e.g., 1GL, 2GL, 3GL, 4GL, or 5GL), such as a machine language that may be directly executed by the processor 304, or logic or algorithms written in assembly language, object-oriented programming (OOP), scripting language, microcode, etc., that may be compiled or assembled into machine-readable instructions and stored in the memory 302. Alternatively, the machine-readable instruction set may be written in a hardware description language (HDL), such as logic implemented via a field programmable gate array (FPGA) configuration or an application-specific integrated circuit (ASIC) or the like. Thus, the functionality described herein may be implemented in any conventional computer programming language, as pre-programmed hardware elements, or as a combination of hardware and software components. For example, memory component 302 may be a machine-readable memory (which may also be referred to as a non-transitory processor-readable memory or medium) that stores instructions that, when executed by processor 304, cause processor 304 to perform methods or control schemes as described herein. Although the embodiment depicted in Figure 1 includes a single non-transitory computer-readable memory 302, other embodiments may include more than one memory module.

[0016] The input / output hardware 305 may include a monitor, keyboard, mouse, printer, camera, microphone, speakers, and / or other devices for receiving, transmitting, and / or presenting data. The network interface hardware 306 may include any wired or wireless networking hardware, such as a modem, LAN port, Wi-Fi card, WiMax card, mobile communication hardware, and / or other hardware for communicating with other networks and / or devices.

[0017] The voice sensor 402 is connected to the local interface 303 and communicatively connected to the processor 304. The voice sensor 402 may be one or more sensors connected to the voiceprint authentication system 100 that determine the volume, pitch, frequency, and / or characteristics of the voice in the vehicle. The voice sensor 402 may include a microphone or an array of microphones that may include mechanisms for filtering background noise, such as engine sounds or beamforming.

[0018] The data storage component 307 stores voiceprints 317 , user preferences 327 , embeddings used 337 , and training data 347 .

[0019] The memory component 302 may include a voice feature module 322, a similarity module 332, and an authentication module 342. The voice feature module 322 may further include a neural network module that includes an encoder and a decoder.

[0020] The audio feature module 322 may be a machine learning function trained and provided via the neural network 122 as described herein. By way of example and not limitation, the neural network 122 may utilize one or more artificial neural networks (ANNs). In an ANN, the connections between the nodes may form a directed acyclic graph (DAG). The ANN may include node inputs, one or more hidden activation layers, and node outputs, and may be utilized with an activation function in the one or more hidden activation layers, such as a linear function, a step function, a logistic (sigmoid) function, a tanh function, a rectified linear unit (ReLu) function, or a combination thereof. The ANN is trained by applying the activation function to a training data set, determining an optimized solution from adjustable weights and biases applied to the nodes in the hidden activation layers, and generating one or more outputs as the optimized solution having a minimized error. In a machine learning application, new inputs (such as the generated one or more outputs) may be provided to the ANN model as training data to continue to improve the accuracy and minimize the error of the ANN model. The one or more ANN models may utilize one-to-one, one-to-many, many-to-one, and / or many-to-many (sequence-to-sequence) sequence modeling. The one or more ANN models may employ a combination of artificial intelligence techniques, such as, but not limited to, deep learning, random forest classifiers, feature extraction from audio, images, clustering algorithms, or combinations thereof. In some embodiments, a convolutional neural network (CNN) may be utilized. For example, a convolutional neural network (CNN) may be used in the field of machine learning as an ANN, which is a class of deep feed-forward ANNs applied, for example, to audio analysis of recordings. A CNN may utilize a shared weight architecture and transformations that are shift or space invariant.

[0021] 2A and 2B depict example block diagrams of example implementations of a voice feature module, a similarity module, and an authentication module that authenticates users and generates probabilistic concepts related to authenticated users for weighting and interpolation.

[0022] As shown in the block diagram of FIG. 2A, the voiceprint authentication system 100 receives the human voice 110 of the current speaker inside the automobile, determines whether the human voice 110 belongs to the registered user to be authenticated, and also determines a probabilistic concept 150 associated with the authentication for downstream embedding interpolation 160 (e.g., as shown in FIG. 2B). More specifically, the voiceprint authentication system 100 receives the human voice 110 and transmits the human voice 110 to a voice feature module 322. The voice feature module 322 includes one or more neural networks 122, which are trained to generate input features 112 based on training data comprising human voices spoken by multiple past speakers inside the automobile. The input features 112 can be converted into an input vector 113 that is received by a similarity module 332. The similarity module 332 compares the input vector 113 with the past vectors 116 in the voiceprint of the enrolled user (e.g., all available vectors belonging to the enrolled user in the voiceprint authentication system 100) to determine whether the input vector 113 is similar to at least one of the past vectors 116 in the voiceprint belonging to the enrolled user. How similar the two vectors are is measured as a similarity. After determining that the similarity between the input vector 113 and the past vectors 116 in the voiceprint belonging to the certified user is less than a threshold similarity, the authentication module 342 authenticates the current speaker of the human voice 110 as a certified user. The authentication module 342 may further calculate a probabilistic concept based on the similarity. The calculated probabilistic concept may be applied to interpolate between downstream user preference embeddings associated with the certified user, for example, between the user preferences 327 and the usage embedding 337 shown in FIG. 2B.

[0023] In an embodiment, the voice feature module 322 uses the neural network 122 to generate a set of input features 112 based on the human voice 110. The input features 112 of the human voice may include, but are not limited to, data on the tone, pitch, volume, rate, and timbre of each speech. The features of the enrolled users may then be pooled into the user's voiceprint 317 (e.g., as shown in FIG. 3 ), which includes all of the past features 115 of the voiceprint since the user first used the voiceprint authentication system 100 for authentication.

[0024] The set of input features 112 may be plotted in a coordinate system as an input vector 113. A vector is a vector point in the coordinate system. For example, the voiceprint authentication system 100 may select a multi-dimensional coordinate system in which each axis represents one variable of the feature, where the variables may include tone, pitch, volume, speed, timbre, or the like. The voiceprint authentication system 100 may plot the vectors each time after receiving the speech of the same enrolled user into a cluster (e.g., a group of vectors), where the cluster represents the voiceprint of that particular enrolled user. Similarly, the voiceprint 317 of an enrolled user may include all the past features 115 of the voiceprint since the user first uses the voiceprint authentication system 100 for authentication. Also, the voiceprint 317 of an enrolled user may include all the past vectors 116 as the enrolled user's cluster since the user first uses the voiceprint authentication system 100 for authentication. When the clusters belonging to different enrolled users are plotted in the coordinate system, the clusters may overlap (e.g., as shown in FIG. 5A). It should be noted that clusters of enrolled users' voiceprints can also be generated through plots of features in the voiceprint 317.

[0025] The similarity measures described herein are based on vectors (i.e., points of embeddings plotted in a coordinate system) or alternatively on features. In an embodiment, the similarity measures may include Euclidean similarity and cosine similarity. Euclidean similarity is the distance between two points of a feature. The higher the similarity, the closer the points are. Euclidean distance is the square root of the sum of squared differences between corresponding elements of two vectors. Cosine similarity is based on the angle between two vectors. Cosine similarity is equal to (1-cos α), where α is the angle between two vectors. For both Euclidean and cosine similarity, a smaller value of similarity indicates that the vectors are more similar. Euclidean similarity may be preferred over cosine similarity when the voiceprint authentication system 100 receives sufficient data of human voices 110 from enrolled users. Cosine similarity may be preferred when the ratio between features is more important than prioritizing the features. For example, if a person has high tone and high pitch features, if the voiceprint authentication system 100 adopts a two-coordinate system with an x-axis representing pitch and a y-axis representing tone, cosine similarity can be found for vectors representing low tone and low pitch. This is useful to admit more voice inputs at an early stage when available voice features are limited and can be used to expand the available data to further verify a particular user and personalize the user's usage preferences. Thus, the system can strategically choose which similarity to use during authentication depending on the availability of data / features in the voiceprint for that particular user. In some embodiments, the similarity module 332 can further adopt other types of similarity such as Jaccard similarity or Dice similarity when calculating the similarity. Jaccard similarity measures the similarity between two sets of features by counting the common features and dividing it by the total number of features in the two sets. Dice similarity, which is similar to Jaccard similarity, weights the count of common features by the frequency of the two sets.

[0026] The authentication module 342 may select the threshold similarity 140 to indicate the minimum threshold at which the human voice received by the voiceprint authentication system 100 is recognized as the voice of an enrolled user. A low value of the threshold similarity 140 suggests a high degree of authentication and privacy protection to exclude unauthorized use by a person. However, during the initial use of the voiceprint authentication system 100, a higher value may be adopted to admit more voice data into the system and increase the pool of voiceprint past features 115 to allow the neural network 122 to be personalized to one or more users. At a later stage, when the system finds that sufficient reliable data is available, a lower threshold similarity 140 may be selected. In some cases, clusters (groups of vectors) of voiceprints may overlap. Because the input vector 113 generated based on the human voice 110 is plotted in the overlapping area of ​​the two clusters, the voiceprint authentication system 100 may find a high degree of confidence that the human voice 110 belongs to two enrolled users. At that point, authentication may result in an inaccurate weighted merging and interpolation 160. Solutions to this inaccuracy are presented further below.

[0027] In an embodiment, when the voiceprint authentication system 100 determines that the human voice does not belong to any registered user, the voiceprint authentication system 100 may create a voiceprint for the non-user. With this function, the voiceprint authentication system 100 may include multiple voiceprints of the non-user. Thus, the similarity module 332 (e.g., as shown in FIG. 2A) may determine the similarity between the input vector and the voiceprint of one or more non-users. After determining that the similarity is less than the threshold similarity, the voiceprint authentication system 100 may merge the input vector 113 into the voiceprint of the non-user. Furthermore, upon determining that the similarity is greater than the threshold similarity, the voiceprint authentication system 100 may create a voiceprint of the non-user based on the input vector 113. The function of creating a voiceprint for the non-user allows the convenience of creating a high-confidence user voiceprint so that the non-user can be later registered as a user.

[0028] The authentication module 342 determines a probabilistic concept 150 based on the similarity 118 calculated by the similarity module 332 between the input vector 113 and the past vectors 116. The probabilistic concept 150 may include a weighting factor inversely proportional to the similarity 118, where the weighting factor has a value between 0 and 1. If the similarity suggests that the input vector 113 is identical to a vector in the past vectors 116, the weighting factor may be equal to 1. If the similarity suggests that the input vector 113 is outside all the past vectors 116, the weighting factor may be equal to 0.

[0029] 2B, an example block diagram of an example implementation of the authentication module for authentication and generation of probabilistic concepts for weighting and interpolation is depicted. After the authentication module 342 generates the probabilistic concepts 150, the probabilistic concepts 150 can be used to weight different features in different user embeddings, with some features being more important than others. For example, the probabilistic concepts can be used to cluster users into different groups, which helps to identify users with similar tastes or predict how users will respond to a new piece of content, so that the vehicle system can predict or recommend content that the user is likely to enjoy.

[0030] In particular, probabilistic concepts 150 can be used to incorporate real-time generated features and vectors into embeddings in the vehicle voiceprint authentication system 100 and other systems. For example, input features 112 can be weighted 152 before being pooled with historical voiceprint features 115 for authorized users, user comments 120 derived from human voice 110 can be weighted 152 before being pooled with user preferences 327, and user interactions 130 derived from human voice 110 can be weighted 152 before being pooled with the embeddings used.

[0031] The authentication module may also grant authentication 142 of the speaker of the human voice 110 after determining a probabilistic concept 150 based on the similarity 118 calculated by the similarity module 332 between the input vector 113 and the past vector 116. Under such authentication, the speaker may be authenticated, for example, using the human voice 110 to command the vehicle to control the vehicle's subsystems to use various functions in the vehicle system. The speaker may command the vehicle to turn on or off the radio or media player, make a phone call, perform a search, and the like.

[0032] In an embodiment, after the voiceprint authentication system 100 determines that the authentication 142 is available for the authorized user based on the human voice 110, the voiceprint authentication system 100 may further determine whether the human voice 110 includes a user comment 120, for example, whether the authorized user provides a positive comment about a newly opened restaurant. After determining the human voice 110 and the user comment 120, the voiceprint authentication system 100 may weight the user comment 120 based on a probabilistic concept 150 and integrate the weighted (152) user comment 120 into the user preferences 327. Similarly, after the voiceprint authentication system 100 determines that the authentication 142 is available for the authorized user based on the human voice 110, the voiceprint authentication system 100 may further determine whether the human voice 110 includes a user interaction 130, for example, whether the authorized user commands the vehicle system to play a particular song or call a friend of the authorized user. After determining the human voice 110 and the user interaction 130, the voiceprint authentication system 100 may weight the user interaction 130 based on a probabilistic concept 150 and integrate the weighted (152) user interaction 130 into the usage embedding 337.

[0033] Furthermore, probabilistic concepts 150 may be used to interpolate between different embeddings, such as downstream user preference embeddings associated with authorized users. A downstream user preference embedding may refer to a user preference embedding that requires voiceprint authentication before providing a service associated with the user preference embedding or integrating new input into the user preference embedding. User preference embeddings may be collected from a variety of sources, such as commands to vehicle systems, website logs, and app usage data of registered users. User preferences may be calculated incrementally based on user comments stated by authorized users during authentication and dynamically integrated with past user preferences. Downstream user preference embeddings include user preferences 327 and usage embeddings as described above. Probabilistic concepts 150 may be used to enable interpolation 160 between downstream high-dimensional user preference embeddings by providing a common representation of authorized users across downstream user preferences using different devices and applications. For example, the interpolated embedding 170 may be generated using the formula α×(user preferences 327)+(1−α)×(usage embedding 337), where α is a weighting factor. The weighting factor, which has a value between 0 and 1, is inversely proportional to the similarity 118, and may represent the relative importance of the two user preference embeddings, i.e., the user preferences 327 and the usage embedding 337. The interpolated embedding 170 may then be used to make predictions regarding the preferences of the user. For example, the vehicle system may predict whether an authenticated user is likely to order a pizza from one of the local pizza places, either by calling for delivery through the vehicle system or by driving directly to pick up the pizza. Furthermore, the vehicle system may use the interpolated embedding 170 as one of the features in a machine learning model.

[0034] Referring to FIG. 3, an exemplary neural network 122 of the voice feature module 322 in the voiceprint authentication system 100 is depicted, which is trained using training data and real-time data. The neural network 122 in the voiceprint authentication system 100 is used to generate features derived from human voices 110 collected in a vehicle system. The neural network 122 may be pre-trained with features based on training data 347 comprising human voices spoken by multiple past speakers inside the vehicle. Once the neural network 122 is pre-trained, it may be used to generate input features 112 derived from the human voices 110. The input features 112 may be weighted using probabilistic concepts 150 and integrated with the associated enrolled user's voiceprint 317, which is then used to continuously train the neural network 122.

[0035] In an embodiment, the neural network 122 may include an incremental learning algorithm that dynamically integrates the input features 112, which are weighted based on the probabilistic concepts 150, with the voiceprint 317 of the certified user. The incremental learning algorithm provides the neural network 122 with the ability to accumulate past features for a previous task and simultaneously incorporate the input features 112 for a current task. For example, the newly updated voiceprint by the neural network 122 is fed back to the neural network 122 to train the neural network 122 based on the data just processed based on the human voice 110. This process is repeated as new human voices 110 become available, which allows the neural network 122 to continuously improve its accuracy. The incremental learning algorithm may enable the neural network 122 to generate features with little or no pre-training data. The incremental learning algorithm may work with four different models in this context: In the evaluation mode, the incremental learning algorithm tracks the predictive performance of the model over the incoming data, such as human speech 110, or the history of the model used for incremental learning. In the detection drift mode, the incremental learning algorithm verifies whether the predicted features show structural changes or distribution drift, e.g., whether the distribution of the predicted features has changed sufficiently. In the active training mode, the incremental learning algorithm may also actively train itself by updating the model based on the incoming data, such as human speech 110. In the generative prediction mode, the incremental learning algorithm may generate features using predicted labels from the latest model. The neural network 122 may robustly switch between these modes depending on the sufficiency of speech data that exists for the incremental model to generate predictions and the sufficiency of training for concise prediction of features.

[0036] Referring to Figure 4, an example interior view of a vehicle with a voiceprint authentication system 100 having hardware to detect speech within the vehicle and initialize a voiceprint registration in the vehicle is depicted. As shown in Figure 4, the voiceprint authentication system 100 may include one or more voice sensors 402 inside the vehicle that detect and process sound waves generated when a person speaks within the vehicle. The voice sensor may be located in the ceiling, dashboard, or center console of the vehicle. The voice sensor may be connected to one or more microphones that pick up the sound waves. The sound waves are then processed by a controller 101 that converts the sound waves into digital signals.

[0037] The voiceprint authentication system 100 may further include a button and a touch screen to initialize voiceprint registration for a new user. One or more enrolled users' voiceprints are enrolled through an initial implementation. The initial implementation may include a physical or audio trigger for enrollment to initialize the enrollment. A user who wants to enroll via an audio trigger for enrollment may need to request the enrolled user to be authenticated by the voiceprint authentication system 100 and approve the enrollment process. A user who wants to enroll via a physical trigger for enrollment may need to physically press the button 404 or touch the touch screen 406 to access the settings menu of the voiceprint authentication system 100 and initialize the voice enrollment process. Once the voiceprint authentication system 100 indicates that the voice enrollment process has begun, the user may provide a voice sample to the system to create an initial voiceprint. For example, the user may create a voiceprint by reading a predefined script. In another example, the user may speak any set of words without following a script. The user's voice wave file may then be input to a neural network that may be pre-trained to create a number of vectors representative of the user's voice characteristics, and the vectors may be stored in the voiceprint authentication system 100. The voiceprint authentication system 100 may further refine the initial voiceprint based on an average of the user's enrollment phrases in a cluster of the initial voiceprint.

[0038] 5A and 5B, an example of reducing the voiceprints of multiple users for more accurately learned features is depicted. As shown in FIG. 5A, the clusters of voiceprints for multiple users may overlap each other. The overlap between different clusters is problematic because the input vector 113 may be considered to be very similar to more than one registered user, and it may be difficult for the voiceprint authentication system 100 to accurately determine which user embedding should be enabled.

[0039] Furthermore, the voiceprint belonging to a user may have a separated vector that is far from the main area of ​​the cluster. If the voiceprint has a separated vector that is far from the main area of ​​the cluster, it may be difficult for the voiceprint authentication system 100 to identify the user. This may be unavoidable when the user has a unique speaking pattern (a truly separated vector) that is not well represented by other vectors in the cluster. However, in some cases, the separated vector exists due to artificial reasons. For example, the user may be speaking in a noisy environment, which may make it difficult to capture all the features of the user's voice, or the user may be speaking quickly, which may also make it difficult to capture all the features of the user's voice. Therefore, it is desirable to remove the separated vector.

[0040] To address the above-mentioned problems, the voiceprint authentication system 100 may include a function to reduce the voiceprint to an extent that the voiceprint authentication system 100 can efficiently and accurately recognize the user. Voiceprint reduction refers to reducing the dimensionality of the voiceprint representation of the user's voice to improve the accuracy of speech recognition. This technique may be achieved through the use of a neural network. For example, the neural network 122 may reduce the voiceprint of a certified user by removing features of the voiceprint that have a confidence that is below a threshold confidence. To enable the neural network 122 to have such a function, the neural network 122 may be trained to recognize different features that make up a voiceprint and the same features of a voiceprint that may belong to a different user. The trained neural network 122 may be used to extract features from an original voiceprint (e.g., as shown in FIG. 5A) to create a new, smaller voiceprint (e.g., as shown in FIG. 5B). The new voiceprint is smaller than the original voiceprint but retains important features. During extraction, a confidence value may be used to determine which features are retained in the new voiceprint. Features with confidence values ​​greater than the threshold are retained, while features with confidence values ​​less than the threshold are discarded. The choice of confidence value depends on the application. For example, voiceprints used for identification require a high confidence value to ensure that new voiceprints are more accurate. Methods for extracting features may include Principal Component Analysis (PCA) or Recursive Feature Reduction (RFE). In particular, as shown in FIG. 5B, a voiceprint may be reduced by removing features of the voiceprint that overlap with another voiceprint. In some embodiments, the voiceprint authentication system 100 may include machine learning capabilities to distinguish truly isolated vectors from other isolated vectors, since there may be a risk of falsely rejecting authentication if truly isolated vectors are removed.

[0041] 6, an exemplary flow chart is depicted describing the operation of the voiceprint authentication system 100 of the present disclosure with respect to voiceprint authentication 142 and interpolation 160. In step 601, the voice feature module 322 uses the neural network 122 to generate input features based on the human voice 110 of a current speaker inside the vehicle. The human voice 110 may be received using the voice sensor 402. The neural network 122 may be pre-trained based on training data 347 including human voices spoken by multiple past speakers inside the vehicle, or may be continuously trained during use of the voiceprint authentication system 100.

[0042] In step 602, the similarity module 332 calculates a similarity between the input vector 113 of the input features 112 and the past vectors 116 in the voiceprint of one or more enrolled users. In step 603, the authentication module 342 determines whether at least one similarity between the input vector 113 and the past vectors 116 in the voiceprint 317 is less than a threshold similarity 140.

[0043] If there is no similarity between the input vector 113 and the past vectors 116 that is less than the threshold similarity (step 603, no), then in step 604, the voiceprint authentication system 100 may create a voiceprint 317 of a non-user based on the input vector 113 of the input features 112. If at least one similarity between the input vector 113 and the past vectors 116 is less than the threshold similarity 140 (step 603, yes), then in step 605, the authentication module 342 determines whether the past vector 116 that is highly similar to the input vector 113 belongs to an enrolled user.

[0044] If the authentication module 342 determines that the past vector does not belong to an enrolled user (step 605: no), then in step 606, the authentication module 342 recognizes the current speaker as a non-user, and the voiceprint authentication system 100 may merge the input vector 113 with the voiceprint of the non-user. If the authentication module 342 determines that the past vector belongs to an enrolled user (step 605: yes), then in step 607, the authentication module 342 recognizes the current speaker as an authorized user and authenticates the current speaker as an authorized user.

[0045] In step 608, the authentication module 342 calculates a probabilistic notion 150 based on the similarity measure 118. In step 609, the voiceprint authentication system 100 applies the probabilistic notion 150 to interpolate between preference embeddings associated with the authenticated users. For example, the voiceprint authentication system 100 may interpolate between the user preferences 327 incorporating weighted user comments 120 and the usage embeddings 337 incorporating weighted user interactions 130.

[0046] 7, an exemplary flow chart describing the operation of the neural network 122 of the voiceprint authentication system 100 is depicted. In step 701, the neural network 122 may be pre-trained based on training data 347 comprising human voices spoken inside the vehicle. The training data 347 may further include background voices in the vehicle, so that the neural network 122 may be trained to remove background noises associated with human voices. The neural network 122 may be further trained to determine whether the received sound waves picked up by the sound sensor 402 are recorded inside the vehicle, or whether the received sound waves are a recording rather than real-time speech.

[0047] At step 702 , the audio sensor 402 receives a human voice and the neural network 122 in the audio feature module 322 generates input features 112 based on the incoming data, such as the human voice 110 .

[0048] In step 703, the authentication module 342 determines whether to authenticate the authorized user. If there is no authentication for the authorized user (step 703: No), the neural network 122 may decline to obtain input features for training and wait to receive a new human voice in step 702 and generate input features based on the new human voice.

[0049] If authentication exists for an authorized user (step 703: yes), then in step 704, the authentication module 342 calculates a probabilistic concept 150 based on the similarity 118 between the input vector 113 and the authorized user's past vectors 116.

[0050] In step 705, the voiceprint recognition system 100 provides the input features 112 and the input vectors 113 to the neural network 122 for training. After training, the voiceprint recognition system 100 may receive another human voice 110 for another round of training, thereby continuously improving the accuracy of the neural network 122.

[0051] Referring to FIG. 8, an exemplary flow chart is depicted listing steps of a method for authenticating a speaker and applying a probabilistic concept for interpolation. In step 801, the method includes training a neural network to generate features based on training data comprising human voices spoken by multiple past speakers inside the vehicle. In step 802, the method includes using the neural network to generate input features based on the human voice of a current speaker inside the vehicle. In step 803, the method includes calculating a similarity between an input vector of the input features and past vectors in the voiceprint of one or more enrolled users. In step 804, the method includes authenticating the current speaker as a certified user after determining that a similarity between the input vector and at least one past vector in the voiceprint of the certified user is less than a threshold similarity. In step 805, the method includes calculating a probabilistic concept based on the similarity. In step 806, the method includes applying the probabilistic concept to interpolate between downstream user preference embeddings associated with the certified users.

[0052] The terms used herein are merely for the purpose of describing particular aspects and are not intended to be limiting. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms, including "at least one," unless the content clearly dictates otherwise. "Or" means "and / or." As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. It will be further understood that as used herein, the terms "comprises" and / or "comprising" or "includes" and / or "including" specify the presence of stated features, regions, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, regions, integers, steps, operations, elements, components, and / or groups thereof. The term "or combinations thereof" means combinations including at least one of the aforementioned elements.

[0053] While particular embodiments have been shown and described herein, it should be understood that various other changes and modifications can be made without departing from the spirit and scope of the claimed subject matter. Moreover, although various aspects of the claimed subject matter have been described herein, the aspects need not be utilized in combination. It is therefore intended that the appended claims cover all such changes and modifications that are within the scope of the claimed subject matter.

Claims

1. generating input features based on the human voice of a current speaker inside the vehicle using a neural network trained to generate features based on training data comprising human voices spoken by a plurality of past speakers inside the vehicle; calculating a similarity between the input vector of input features and past vectors of voiceprints of one or more enrolled users; authenticating the current speaker as the authorized user after determining that a similarity between the input vector and at least one past vector of a voiceprint of an authorized user is less than a threshold similarity; calculating a probabilistic concept based on said similarity measures; applying said probabilistic concepts to interpolate between downstream user preference embeddings associated with said qualified users; A method comprising:

2. The method of claim 1 , wherein the similarity measure is a Euclidean similarity measure or a cosine similarity measure.

3. The method of claim 1 , wherein the probabilistic concept comprises a weighting factor that is inversely proportional to the similarity.

4. the downstream user preference embedding comprises a user preference and a usage embedding; The user preferences include user preferences calculated based on user comments stated by the authorized user during authentication and dynamic integration into past user preferences; The method of claim 1 , wherein the usage embedding comprises a user interaction with the vehicle.

5. after authenticating the authorized user, further determining whether the human voice comprises a user interaction with the vehicle; 5. The method of claim 4, further comprising: after determining that the human speech comprises the user interaction, integrating the user interaction, weighted based on the probabilistic concept, into the usage embedding associated with the qualified user.

6. The method of claim 1 , wherein the neural network comprises an incremental learning algorithm that dynamically integrates the input features, weighted based on the probabilistic concepts, with the voiceprints of the authorized users.

7. The method of claim 1 , further comprising reducing the voiceprint of the qualified user by removing features of the voiceprint that have a confidence that is less than a threshold confidence.

8. The method of claim 7 , wherein the voiceprint of the authorized user is reduced by removing the features of the voiceprint that overlap with the voiceprint of another enrolled user.

9. The method of claim 1 , wherein the input features of human speech comprise tone, pitch, volume, rate, or timbre.

10. 2. The method of claim 1, wherein the voiceprints of one or more registered users are registered through an initial implementation, the initial implementation comprising a physical or audio trigger for registration to initialize the registration and a recording of the human voice to create the voiceprint to be registered.

11. The method comprises: calculating a non-user similarity between the input vector and one or more non-user voiceprint vectors; determining whether the non-user similarity is less than, equal to, or greater than the threshold similarity; after determining that the non-user similarity is less than the threshold similarity, integrating the input vector with the voiceprint of the one or more non-users; generating a voiceprint of the non-user based on the input vector after determining that the non-user similarity exceeds or is equal to the threshold similarity; The method of claim 1 further comprising:

12. generating input features based on the human voice of a current speaker inside the vehicle using a neural network trained to generate features based on training data comprising human voices spoken by a plurality of past speakers inside the vehicle; calculating a similarity between the input vector of the input features and past vectors of the voiceprints of one or more enrolled users; authenticating the current speaker as the authorized user after determining that a similarity between the input vector and at least one past vector of the authorized user's voiceprint is less than a threshold similarity; Calculating a probabilistic concept based on said similarity; applying said probabilistic concepts to interpolate between downstream user preference embeddings associated with said qualified users; A system comprising a controller.

13. The system of claim 12 , wherein the probabilistic concept comprises a weighting factor that is inversely proportional to the degree of similarity.

14. the downstream user preference embedding comprises a user preference and a usage embedding; the user preferences comprise user preferences calculated based on user comments stated by the authorized user during authentication and dynamically integrated with past user preferences; The system of claim 12 , wherein the usage embedding comprises a user interaction with the vehicle.

15. after authenticating the authorized user, further determining whether the human voice comprises a user interaction with the vehicle; 15. The system of claim 14, after determining that the human speech comprises the user interaction, integrates the user interaction, weighted based on the probabilistic concept, into the usage embedding associated with the qualified user.

16. The system of claim 12 , wherein the neural network comprises an incremental learning algorithm that integrates the input features weighted based on the probabilistic concepts with the voiceprint of the authorized user.

17. The system of claim 12 , wherein the input features of human speech comprise tone, pitch, volume, rate, or timbre.

18. The system of claim 12 , further comprising a sound sensor for receiving or recording the human sound.

19. 13. The system of claim 12, wherein the voiceprints of one or more registered users are registered through an initial implementation, the initial implementation comprising a physical or audio trigger for registration to initialize the registration and a recording of the human voice to create the voiceprint to be registered.

20. 20. The system of claim 19, wherein the system further comprises a button or a touch screen, and the initial implementation is physically triggered when the button is pressed or the touch screen is touched.

Images