Method, authentication system, and program

Abstract

To provide a system for achieving interoperability and access to a personal extension knowledge / preference database (PEKD) through interconnected voice verification systems.SOLUTION: A virtual assistant ecosystem comprises devices and a voice verification system (VVS). Users can enroll with the VVS so that the VVS can provide authentication of the users by personal wake phrases. Thereafter users can access their PEKD from unowned devices by speaking their wake phrases.SELECTED DRAWING: None

Description

[Technical Field]

[0001] No reference to related applications. [Background technology]

[0002] background Voice as a human-machine interface has been gaining attention. Various major companies offer voice virtual assistants, such as Google Assistant, Amazon Alexa, Apple Siri, Microsoft Cortana, SoundHound Hound, Alibaba's Timo Wizard, Tencent's Micro, Xiaomi's Xiaoai, Baidu's DualOS, Huawei's Seria, Samsung's Bixby, Naver's Clover, and Mycroft. Many of these assistants are embedded systems (IoT devices). For each, users must create an account. After doing so, they can use the virtual assistant and the devices it can control. These constitute a unique ecosystem controlled by the voice assistant. The company providing the voice assistant uses information about the user while creating the account and the data collected from their use to provide useful features and services. However, each company incorporates the user's personal information into its ecosystem. Users cannot benefit from improved services from another company without recreating a new account and providing their personal information again. This benefits existing providers while blocking innovative, disruptive companies that could offer useful new services without the barrier of forcing users to move between service providers—all to the detriment of consumers. Summary of the Invention

[0003] overview The following specification discloses a verification system that involves storing a wake phrase uniquely associated with a Personal Extension Knowledge / Preference Database (PEKD) and an enrollment signature, accepting a spoken voice, recognizing the unique wake phrase within the spoken voice, extracting the signature from the spoken voice, comparing the extracted signature with the enrollment signature associated with the unique wake phrase to authenticate the user, and allowing the user access to the associated PEKD.

[0004] Such systems may be user-owned devices, public distributed ledgers with service providers allowing third-party access, or other suitable mechanisms. The PEKD may be stored in a database.

[0005] User recognition and authentication is based on phrase-independent automatic speech recognition. One way this can be done is by extracting and computing audio feature vectors using a trained model. The feature vectors may be in a learned embedding space. In such cases The comparison of the received vector with the vector stored in the user profile may be performed by calculating the distance between the vectors. Another approach is to use a phrase spotter trained on the wake phrase. Voice authentication may be phrase-dependent, based on the unique wake phrase, or phrase-independent, based on the utterance of other words.

[0006] This specification also discloses a method of user authentication that includes receiving personal data from a user, receiving a request from a service provider to access the data, requesting authorization from the user, receiving a response from the user, and providing the service provider with access to a copy of the data conditioned on the response.

[0007] Such authentication may also include storing an enrollment biometric signature, comparing it to the biometric signature in the request, and conditioning the response on the similarity between the signatures. The biometric signature may be stored in a machine-learned embedding space. It may be a feature vector, and the similarity may be calculated as the distance between the vectors. Many types of biometrics are possible, but one method is to extract a signature from the speech.

[0008] This specification also discloses a computer-implemented method of database management, the method including receiving an API request for a database operation on a PEKD from a client device, the request initiated by a user of the client device, the method further including determining from the request, for a portion of the user's personal data to be accessed, whether the type of data is to be deleted or retrieved, and from the type of data, determining an appropriate server to store data of that type, sending the operation request to the appropriate server, receiving a response from the server, and sending the response to the device.

[0009] For such methods, the response may be a delete confirmation, a retrieve response containing the PEKD, or other type. Administrative access to the database may involve auth proxy authorization of the request, which may use a mutual authentication protocol. Database operation requests may be sent to a Kafka queue. Additionally, database requests may be distributed across multiple MapReduce clusters. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 illustrates a virtual assistant ecosystem. [Figure 2] FIG. 1 is a block diagram of a virtual assistant ecosystem. [Figure 3] FIG. 1 illustrates a user interaction with a virtual assistant in a hotel room. [Figure 4] FIG. 1 illustrates a device for authenticating a user by voice verification. [Figure 5] FIG. 1 illustrates a method for registering a user in a voice verification system. [Figure 6] FIG. 1 illustrates voice fingerprinting. [Figure 7] FIG. 1 illustrates voice authentication using a voice fingerprint. [Figure 8] FIG. 1 illustrates a voice interaction with a point of sale device. [Figure 9] FIG. 1 illustrates a method for registering a user for double authentication. [Figure 10] FIG. 1 illustrates a method for phrase-independent audio fingerprint verification. [Figure 11] FIG. 10 is a diagram illustrating a method of verification by voice fluctuation detection. [Figure 12] FIG. 1 illustrates a voice interaction with a vending machine. [Figure 13]FIG. 1 illustrates a voice interaction with a building security system. [Figure 14] FIG. 1 illustrates voice verification using edge processing. [Figure 15] FIG. 1 illustrates a virtual assistant platform with authenticated data access. [Figure 16] FIG. 1 illustrates a virtual assistant platform with authenticated access and queuing requirements. [Figure 17A] FIG. 1 illustrates a non-transitory computer-readable medium. [Figure 17B] FIG. 1 illustrates another type of non-transitory computer-readable medium. [Figure 18A] FIG. 1 illustrates a system-on-chip. [Figure 18B] FIG. 1 shows a block diagram of a system-on-chip. [Figure 19A] FIG. 1 illustrates a rack server. [Figure 19B] FIG. 2 is a block diagram of a server. DETAILED DESCRIPTION OF THE INVENTION

[0011] Detailed Description The following discloses a system of process steps, as well as a system of machines and components, for achieving interoperability through interconnected voice verification systems. Some embodiments use a computer executing software instructions stored on a non-transitory computer-readable medium. The following examples illustrate design choices for various aspects of such a system. Generally, the design choices for different aspects are independent and can work together in any combination.

[0012] PEKD

[0013] A Personal Augmented Knowledge / Preference Database (PEKD) is a type of personal database that serves to extend an individual's knowledge, intelligence, preferences, and awareness. PEKDs are stored and managed by personal data service providers. Users can use voice and other inputs to construct and access their PEKD and some of the personal data it stores.

[0014] FIG. 1 illustrates an exemplary system for achieving interoperability through the use of an interconnected system of user PEKDs. A user 11 uses voice to summon, interact with, and control a smart speaker 12, which in turn controls other compatible devices. The smart speaker communicates with a virtual assistant platform 13 over a cloud network, such as the Internet. It may perform speech recognition and natural language understanding, request information or actions, and provide responses to the smart speaker via a web API. The virtual assistant platform stores PEKD information 14 and uses it to provide responses and access appropriate information and actions for the user. In some systems, a single company provides both the virtual assistant platform and the smart speaker. However, in the illustrated system, the smart speaker 12 can also communicate with a separate company that provides the smart speaker 15. The smart speaker vendor may also store PEKD information 16 and use it to provide services or additional functionality to the smart speaker. The smart speaker may also communicate with third-party device and service providers, which may operate independently of the virtual assistant platform and smart speaker vendors. Such providers are typically commercial providers of goods and services. Some such third parties may not store the PEKD information for reasons of ensuring user privacy or because they simply do not need it. Many types of consumer devices may be part of such an open ecosystem 17, including, for example, smartphone, appliance, or automobile vendors. Such a system allows different companies to access at least some of the PEKD information they need, while allowing users free access to various devices and services as they wish.

[0015] In some systems, the PEKD is stored and managed on a distributed ledger such as a blockchain, independent of the voice service provider, virtual assistant platform, or device vendor. Such systems use a variety of technologies, including passwords, passphrases, fingerprints, PEKD database records can be encrypted using user-specific information, such as biometrics like retina or voice fingerprints. Authentication for access to such database records can be multi-factored, as required for data security. The distributed ledger can be private or public. It can support permissions such as read, add, modify, and delete. It can support more complex permissions for all or part of the PEKD data. Such distribution allows any service provider, including innovative disruptors, to use the data. It also avoids data corruption and provides resistance to hacking.

[0016] Some systems may store or share PEKD records in a standard format, relying on one or more data formats, that can be freely exchanged between platforms and devices. Such portability has the benefit of avoiding incompatibility and lock-in to ecosystems such as those controlled today by Amazon, Google, SoundHound, Tencent, etc.

[0017] PEKD data types

[0018] A PEKD database contains records with information about users of voice-based or other computer services. Some such information includes name, age, gender, address or region, and region of birth, which is useful for matching users to different types. Another type of PEKD information is shopping preferences, which is useful for product recommendations and targeted advertising. Such information can be treated as or converted into vectors in the PEKD embedding space. Such vectors are useful because they allow the system to calculate distances between users, cluster users by calculating centroids or using density-based algorithms such as DBSCAN, and map the likely interests of any given user. These techniques enable better targeting by advertisers and sellers and more relevant search results for users.

[0019] Another type of PEKD information is a list of pointers to entities with specific names and semantic meanings. For example, a user's contact list contains pointers to records in other people's databases. These may be private to the user. However, contacts may be identifiably linked to other users by distinct user attributes such as email addresses or phone numbers. Another example of such a list is a list of previously purchased or viewed products or URLs. A list of pointers to other entities with specific names and semantic meanings allows speech recognition to recognize those names when they are spoken, and the list of pointers allows speech recognition to improve accuracy by distinguishing names from standard words with similar pronunciations that speech recognition would otherwise recognize as words with standard pronunciations. Pointers between corresponding users or between users and other types of entities enable connection-based clustering. Improved clustering provides better targeting and more relevant search results.

[0020] Ecosystem

[0021] Companies like Google use their Nest home control and other third-party With the device, Amazon is creating its own closed ecosystem with its Ring home security and other third-party devices, along with the ingestion of user data. These closed ecosystems do not allow disruptive innovators to develop their products / services and allow their partners to develop voice assistants that can control ecosystem products developed by Amazon or Google. This benefits those companies to the detriment of users. Other companies, such as SoundHound, offer ecosystems that support any number of third parties without mandatory data ingestion agreements.

[0022] 2 shows further details of an example ecosystem open to third parties. Platform 20 supports multiple clients 21a and 21b. The platform can operate in a cloud computing environment such as those typically found in a data center or a corporate central office. The clients may be fixed home devices such as smart speakers, portable devices such as phones and wearables, including apps running on such devices, other mobile devices such as automobiles and other transportation systems, purchasing systems such as vending machines, point-of-sale (POS) terminals, or drive-through restaurant ordering systems.

[0023] Client devices may be owned by their users, such as home smart speakers and mobile phones, or they may be public, such as point-of-sale terminals, or privately used by different users at different times, such as smart speakers in hotel rooms. Owners of public or shared devices, such as hotels, and retailers, such as grocery stores and fast-food restaurants, may contract with platform providers. To allow users of public and shared clients to access the offered commercial services, it is helpful to associate the user's identity with a PEKD, if the user has an identity. Association can be done by a unique identifier, such as an email address or credit card, by biometric authentication, such as voice fingerprinting or facial recognition, or by enumeration of an identification phrase, as discussed below. When a new user encounters a public or shared device connected to the platform that the user has never registered, the system may immediately perform a registration process. This may involve steps such as requesting a unique identification, requesting a password, requesting a unique phrase, capturing biometric information, or other procedures described below. However, to avoid keeping time-pressured users waiting and requesting personal information in a public place, the system may create a temporary account for the user. The pseudo account may allow certain non-critical functions, such as requesting room service at a hotel or a menu at a fast food restaurant. The pseudo account may or may not allow more critical functions, such as making higher-value purchases. In some cases, lower-value purchases may be authorized with a pseudo credit. To prevent people from abusing such offerings by making numerous small purchases from voice-enabled point-of-sale clients, the system may receive a client location identifier, such as a serial number associated with a location. The system may alternatively receive geographic information, such as latitude and longitude, from the point-of-sale device.To prevent abuse, the system may store the rough voiceprint and geographic information of users purchasing with virtual credit, and reject multiple requests within a short period of time from nearby locations with generally similar voice characteristics.

[0024] The platform receives voice queries or commands from the client. The system receives and performs speech recognition to transcribe spoken words, then interprets the words using a natural language understanding (NLU) interpreter 23. The interpreter performs NLU by comparing the words with a grammar 24. The grammar corresponds to a particular domain of conversation and defines the intent of the spoken words. The spoken words include different common phrases for such intent. The system uses a data structure stored in memory The intent is expressed as a data structure. Processors can pass data structures between each other. Platforms and clients can also pass data structures between each other. Formats such as Extensible Markup Language (XML) or JavaScript Object Notation (JSON) can be suitable for expressing the intent data structure.

[0025] The platform uses the intent to select data sources 25 appropriate for the domain of the grammar. The platform then selects the desired data sources 25 as needed to respond to the query. Search for information from domain providers, encyclopedias such as Wikipedia Searching for information on Weather Underground and other sources Data sources can be of many types, such as checking the weather forecast from a smartphone or finding restaurant recommendations from a provider like Yelp. In response to a command, the platform uses an intent to invoke the appropriate action capability26, such as sending a short message service (SMS) text message, requesting a ride from a provider like Uber, or turning on a smart light switch. Many types of actions are possible, such as turning on the

[0026] Some dedicated clients 27 may have their own operational capabilities. For example, a car such as a Mercedes may enable the heater or air conditioning. Such devices may have their own grammar 28 for recognizing spoken commands. When the platform receives a spoken command for such a client, it interprets it and sends the appropriate message to the client in the format of an agreed-upon data structure.

[0027] The platform may also have proprietary grammars 29 that are not tied to any particular data source or client. For example, the platform may provide the ability to tell time, perform calculations, or convert between units internally. The platform may even use the results of one partial intent to inform another intent. For example, a user's voice query for the current temperature in degrees Kelvin in Death Valley requires a search of the weather domain, followed by unit conversion of the results.

[0028] A platform that is open to third-party client developers and third-party data sources allows any innovative provider to benefit from others, resulting in a virtuous cycle of increased functionality available to users, improved user experience leading to increased device sales, and increased platform usage leading to more opportunities for data source providers to monetize their data.

[0029] Hotel Scenario

[0030] The wake phrase that invokes the voice assistant gives it a perceived identity. In a closed ecosystem, all devices and services can only be accessed through a single assistant identity, accessible via phrases such as "Okay, Google," "Alexa," or "Xiaoai." By having a single provider-defined identity, users recognize the device as an assistant.

[0031] In an open ecosystem, wake phrases can be unique to a user. Users can summon their Assistant through any device, anywhere, by speaking their personal wake phrase. This universality allows users to summon their Assistant as an omniscient and omnipotent guardian angel. This way, you can be aware that your assistant is around you no matter where you are.

[0032] As shown in Figure 3, in this approach, for example, a hotel guest can say their personal wake phrase, such as, "Hi, Mr. Big. I'm Juliet." The in-room smart speaker can recognize the wake phrase and recognize and verify that Juliet is saying it. At that point, the smart speaker can grant Juliet access to her PEKD information and immediately learn that Juliet's favorite music is country music, that Juliet has an appointment tomorrow at 2:00 PM, and who is in Juliet's contact list. With this information, the smart speaker can enable Juliet to control products in the ecosystem with her voice, regardless of the product manufacturer.

[0033] This means that when a user encounters a device they have not used before, for example, a hotel guest entering a hotel room equipped with a smart speaker, they do not need to determine which provider service (e.g., Amazon's Alexa, Google's Assistant, Xiaomi's Xiaoai, etc.) provides the device and can speak the wake phrase to activate the device, providing a more personalized user experience.

[0034] VVS

[0035] A voice verification system or voice verification server (VVS) is A VVS can provide user authentication via a personal wake phrase. Such a system receives spoken audio, identifies the user's unique wake phrase, and verifies the user's identity regardless of the device requesting authentication. The VVS responds to the device with an authentication message. Figure 4 shows an example of different types of devices using a VVS. Smart speakers 41a, locks 41b, security systems 41c, cars such as rental cars 41d, vending machines 41e, ATMs 41f, point-of-sale systems 41g, phones 41h, computers 41i, appliances 41j, thermostats 41k, IoT devices 41l, home automation devices 41m, robots 41n, drive-through restaurants 41o, and other devices 41p can receive voice queries and commands from a person 42. Voice-activated devices connect to a single, centralized voice verification system 44 via the Internet 43. Having a collaborative system provides a consistent protocol and source of truth for device manufacturers, improving development time and effort. It also provides a consistent authentication experience for users, which increases user satisfaction and demand for voice-enabled devices.

[0036] To be able to be verified, a user must register with the VVS 44. Figure 5 shows one possible method of registration. It begins (50) by determining the user's identity (51). Identity determination may be done by the user entering an email address via a keyboard, or the user may provide identification by other methods, including speaking a government-provided identification number, or by the user scanning a badge with an optical or magnetic scanner. The system then requests a desired wake phrase from the new user (52). The system then receives the user's desired wake phrase (53), such as "Hello, Mr. Big. I'm Juliet." Receiving the wake phrase can be done via a text input method, such as typing, or a spoken input method, such as automatic speech recognition (ASR). The VVS stores a database of the user's personal wake phrases. The VVS searches for the user's desired wake phrase in the database and If the wake phrase is not found, the VVS checks the wake phrase and verifies that it is a unique wake phrase (54). If not, the VVS returns to step 52 to request the desired wake phrase. If the wake phrase is unique, the VVS proceeds to associate the wake phrase with the user's identity and store it in the database (55). This completes wake phrase registration (56).

[0037] By ensuring each wake phrase is unique to an individual, users can summon the assistant from any location and any device. This enables interoperability between devices, voice platforms, and data and service providers, which in turn creates new opportunities for ecosystem participants.

[0038] When a user says their personal wake phrase to a device communicating with the VVS, the device transmits the wake phrase to the VVS. The VVS uses the wake phrase to determine and verify the user's identity and provide access to the user's individual PEKD information. This provides authentication that allows users to use their voice to control voice-controlled products in their vicinity, regardless of who developed the device.

[0039] So, returning to the hotel example, the hotel's smart speaker would connect to the VVS. When a hotel guest enters a room and says, "Hello, Mr. Big. I'm Juliet," the smart speaker would send a wake phrase to the VVS. The VVS receives the wake phrase, "Hello, Mr. Big. I'm Juliet." The VVS uses the wake phrase to determine and verify the user's identity. Upon determining and verifying the user, the VVS would enable connection to the user's PEKD. Hotel guests could use their voice to control voice-controlled products, regardless of who developed the products.

[0040] Wake phrase prefix

[0041] General large vocabulary speech recognition has lower accuracy than phrase spotters trained on specific words or phrases. Furthermore, being able to detect anyone's unique wake phrase requires constant transmission of text or audio from the device to the VVS, which requires significant bandwidth to support a large ecosystem.

[0042] A VVS or a specific device may require a prefix or phrase, such as the word "computer...". Requiring a prefix allows the device to run a simple phrase spotter on the prefix without running a full-vocabulary continuous ASR. Avoiding the need for continuous ASR significantly reduces network bandwidth costs in a VVS. It also allows power-sensitive devices to use much less power, thereby extending battery life and the user experience. It also allows cost-sensitive devices to use lower-cost processors, because a phrase spotter is much less processor-intensive than an always-on ASR.

[0043] ASR Personalization

[0044] The voice-based ecosystem relies on ASR conversion of spoken audio into text or other formats that can be interpreted by a program or natural language understanding model. The more accurate the ASR, the better the user experience and the more products are adopted. ASR generally uses an acoustic model to convert spoken audio into audio information, which is then translated into words or It uses language models to convert tokens into other languages. It is also possible to train end-to-end ASR, which combines acoustic and language modeling into one.

[0045] Another type of PEKD information is a personal acoustic model, which allows the system to more accurately adapt speech recognition to account for a given user's voice, accent, and even the user's typical stationary noise. The personal acoustic model may be represented as an adaptation layer within a neural network acoustic model, as a neural model trained by transfer learning from a more general model, or as a speech-acoustic embedding vector that can be used as input to the acoustic model along with a Mel filter bank or other spectral representation of sampled speech processed by a speech recognition acoustic front end.

[0046] Another type of PEKD information is a personal statistical language model (SLM), which is useful for more accurately adapting speech recognition to account for a given user's different typical uses of words and phrases, as well as a personal set of vocabulary words, such as names of associated people. Similar to personal acoustic models, personal SLMs may be represented as an adaptation layer within a neural SLM, as a neural model trained by transfer learning from a more general model, or as language embedding vectors that can be used as input to an SLM along with the input of a speech or other tokenization model.

[0047] Personalized acoustic models and SLMs improve ASR accuracy and increase user experience, retention, and user discovery of data sources, product, and advertising engagement, all of which provide additional revenue opportunities for members of the voice ecosystem.

[0048] Audio Fingerprinting

[0049] Another type of PEKD information is one or more recordings of a user's voice, one or more audio fingerprints extracted from the recordings, or both. Audio fingerprints are useful for user authentication, diarization within multi-user conversations, and pseudonymization. It has various applications, such as customization keying of virtual assistants. A virtual assistant keyed to a voice fingerprint can be calculated in real time. The system can automatically respond to one user differently from another according to a mapping from the best match of the generated fingerprint to the PEKD and therefrom accessing the user-specific information pointed to by the PEKD.

[0050] Figure 6 shows a diagram of one method of audio fingerprinting. Speech audio 61 is captured from a microphone or received over a network connection. Processing can occur locally in the device with the microphone or remotely, such as in a cloud data center server. A window of speech audio having a size of approximately 25 milliseconds is processed frame by frame at a rate of approximately 10 milliseconds per frame. For each frame, spectral energy is estimated in the mel scale for several frequency ranges, such as the 40-80 range. The sequence of calculated mel-scale measurements generates a spectrogram 62. A neural network 63, trained to convert the sequence of speech audio into vectors, processes the spectrogram to generate an audio feature vector, which is a speaker's voice signature 64. Some systems may use a neural network trained directly on the sampled speech audio without computing a spectrogram.

[0051] Vectors can be generated, for example, by: Different hidden Markov models (HMMs) or neural methods The generated vector may represent the speech in a multi-dimensional space with dimensions learned according to the speech feature embedding.

[0052] The accuracy of an audio vector signature generally improves with longer amounts of speech or phrase length. It can be phrase-dependent, such as when speaking a specific wake phrase or prefix of a wake phrase. It can also be phrase-independent, in which case it is calculated without regard to words. Phrase-dependent audio fingerprinting is generally more accurate for smaller pieces of speech, while phrase-independent fingerprinting can take advantage of longer pieces of speech, if available.

[0053] Some audio fingerprinting methods are language-dependent. Language-dependent methods are generally more accurate but require models trained separately for each language, such as English, Mandarin, German, Japanese, French, or Korean. Language-independent audio fingerprinting is also possible, which only requires training a single model for all human speech, but is less accurate for short phrase lengths.

[0054] Authentication via voice fingerprinting requires a proper enrollment procedure. During operation, the VVS captures audio and computes a fingerprint. It compares the computed fingerprint with fingerprints associated with the user's claimed identity. To support this, the user's voice must be captured. The audio must be stored, or the voice fingerprint must be stored in the same vector space as the voice fingerprint computed during normal operation. Storing a record of the enrolled voice makes it possible to retrain the system on an improved embedding space and recompute the user's profile fingerprint as technology or system resources improve in the future.

[0055] During enrollment, a sufficient amount of speech must be captured to provide sufficient information for accurate fingerprint comparison. For a personalized wake phrase in a phrase-dependent voice fingerprinting system, having the user speak the wake phrase three times is typically sufficient. The more speech, the better to capture variations in the user's voice. For phrase-independent voice fingerprinting, 10 seconds or more of speech is typically desirable for accurate fingerprint-based authentication.

[0056] The open source Kaldi package for speech processing provides many of its is an example of a software package that can support audio fingerprinting within its processing capabilities. The neural model used for fingerprinting may include a convolutional neural network (CNN) layer with maxpooling between layers, combined with one or more recurrent layers and a linear attention model.

[0057] Unlike call center voice fingerprinting, which requires manual dialing and operates within the limited environment of the telephone system, voice fingerprinting for interconnected VVS enables a fully voice-based user interface that is robust to various noise environments and distortions, sampling rates and accuracy of device voice components, as well as the latency and bandwidth limitations of network connections.

[0058] Figure 7 shows how the identification of a unique wake phrase can be combined with a voice fingerprint to provide strong authentication. The figure shows the process of combining ASR with VVS printing. ASR performs speech recognition (71). Recognition can be performed locally on the device using text sent to the VVS, or within the VVS on audio sent from the device. Recognition detects the wake phrase and performs a real-time lookup in a database 72 where user-specific wake phrases are stored. The database contains one or more voice signatures captured during enrollment associated with each unique wake phrase and a connection to the corresponding PEKD data.

[0059] An audio fingerprinting model, such as the neural model 63 described above, performs extraction of an audio fingerprint as a feature vector of the captured speech utterance (73). The fingerprint calculation can be performed within the device with the fingerprint vector sent to the VVS, or the fingerprint calculation can be performed directly within the VVS.

[0060] The VVS then performs a comparison between the extracted signature and one or more registered signatures (74). The VVS can perform the comparison by calculating the cosine distance between the vectors or by other multidimensional feature comparison algorithms. If the features match within an acceptable similarity threshold, the user's voice is verified and authentication is provided. Once authentication is successful, the VVS grants access to the user's PEKD (75). The VVS may be separate from the provider of the PEKD database. If so, a signal indicating authorization is sent to the database provider to provide access.

[0061] If the audio comparison does not meet the similarity threshold, authentication fails and the VVS sends a signal to the device informing it that no fingerprints were matched. The device may respond appropriately to the user, such as by providing an error message, asking the user to try again, signaling that the database has been locked due to an unauthorized access attempt, or other appropriate action.

[0062] The combination of unique wake phrase recognition and voice fingerprint authentication provides the convenience of allowing users to have a device-independent, ambient virtual assistant while providing sufficient security for users to trust it with the storage of sensitive information. This convenience enables financial, health, and other personal services that would not otherwise be possible.

[0063] POS scenario

[0064] The interconnected VVS approach has a wide range of applications. Figure 8 shows one example: a touchless, voice-enabled POS system in a supermarket 81 tallies the purchases in a user's bag or cart. Such automated tallies are made possible using a variety of detection techniques, such as RFID tags on products or machine vision cameras observing shoppers selecting products from shelves.

[0065] When a user comes to a POS device to check out and pay, rather than having to put down a bag to use their hands, insert a card into the POS device, tap their phone, or scan a QR code, the shopper can simply speak their individual wake phrase to the POS system: "Hello, Mr. Big. I'm Juliet." The POS device is connected to the VVS via the internet. The POS device sends the individual wake phrase, "Hello, Mr. Big. I'm Juliet," to the VVS. The VVS uses the wake phrase to verify the shopper's identity and the voice The VVS performs fingerprint authentication. If authentication is successful, the VVS signals the user's bank to provide payment to the POS system. This can be direct, or the VVS can provide a key that allows the POS to gain access to certain financial information, such as account numbers, stored in the shopper's PEKD. In this way, the POS system obtains confirmation that the shopper has sufficient financial assets to make the purchase.

[0066] The POS market is generally large. Voice-enabled POS devices are a necessary upgrade across the entire range of vendors with checkout machines, from grocery stores to restaurants and subway stations. Consumers will demand vendors offer voice-enabled POS for the great convenience of not having to put anything they have in their hands, the ability to avoid germs that would be shared through physical contact and interfaces, and the speed of access. Innovation in this ecosystem is only possible with the secure authentication methods described.

[0067] Double Authentication

[0068] A simple VVS may be subject to spoofing attacks. For example, an impostor might record a legitimate user speaking their personal wake phrase. By playing back the recording, a simple VVS would allow the impostor to access the user's PEKD. Some systems circumvent this with double authentication. One form of double authentication is to prompt the user to say an unpredictable word or phrase. The impostor would have no record of the legitimate user saying that word or phrase and would therefore not gain access. Supporting this requires an enrollment process that asks the user to speak an additional word.

[0069] FIG. 9 illustrates one possible method of enrollment with support for double authentication. It is based on the method shown in FIG. 5. Specifically, the method begins (90) by determining the user's identity (91). The user's identity may be provided by entering an email address via a keyboard, by the user speaking a government-issued identification number, or by the user scanning a badge with an optical or magnetic scanner. Next, the system requests a desired wake phrase from the new user (92). The system then receives the user's desired wake phrase (93), such as "Hello, Mr. Big. I'm Juliet." The wake phrase may be provided via a text input method, such as typing, or a spoken input method, such as automatic speech recognition (ASR). The VVS stores a database of all users' personal wake phrases. The VVS proceeds to search the database for the user's desired wake phrase to verify (94) that the user's desired wake phrase does not exist in the database and is therefore a unique wake phrase. If not, the VVS returns to step (92) of requesting the desired wake phrase. If the wake phrase is unique, the VVS proceeds to associate the wake phrase with the user's identity and store it in a database (95).

[0070] The VVS registration process is available for Penelope, Encyclopedia, Avenue, and Fifty-Three. The VVS then proceeds to pseudo-randomly select or create a set of additional words or phrases, such as "." The VVS then requests the user to speak each of the additional words or phrases (96). The VVS records and stores a sample of the phrases spoken by the user (97). The VVS can optionally compute and store phrase-dependent speech verification vectors for each of the additional words or phrases. This completes wake phrase enrollment (98).

[0071] In a POS scenario, during authentication, after the VVS determines the shopper's identity through the wake phrase, the VVS sends one of a randomly selected authentication word (or a series of words, such as numbers, that can be easily recognized using voice recognition) to the POS system. The POS system displays the authentication word on a display screen. The user then reads and speaks the authentication word. The POS system sends the authentication word to the VVS. The VVS compares the authentication word with the authentication word fingerprint stored in its memory. If there is a match, the VVS authenticates the identity to the POS system. If there is no match, the POS system rejects the user.

[0072] Using double authentication significantly reduces or eliminates fraud.

[0073] To double-authenticate with a personal device such as a smartphone, the VVS can send a series of numbers or letters and a numeric verification code to the shopper's personal device. The user can read the code. By using phrase-independent voice recognition, an imposter cannot play back the recording, as long as the code is long enough to force the user to provide a long enough voice sample.

[0074] Figure 10 illustrates a method for phrase-independent authentication with a VVS. It begins (100) by requesting the user's identity (101). The user can provide their identity by providing unique identifying information such as an email address, ID card number, or a unique wake phrase. The VVS then generates (102) any simple voice query, such as "What is two times two?", "Say the word dragon?", or "What's your zip code?" By ensuring that the answer to the question is voluntary, the VVS can address the risk of an imposter using recorded audio. The VVS receives (103) the user's voice response and performs (104) a voice match. If the match is within a threshold distance, the user is authenticated (105). Otherwise, authentication fails (106), and the VVS sends a corresponding signal to the device requesting authentication.

[0075] This particular form of double authentication does not require push notifications.

[0076] Another form of double authentication that thwarts spoofing via playback of voice recordings is to verify small variations in the voice recording. Figure 11 illustrates such a method. It begins (110) by receiving (111) the audio of the wake phrase. The VVS then stores (112) the audio of the wake phrase for future reference. The VVS then compares (113) the received audio of the wake phrase with previous stored audio samples. This comparison can be done by comparing fingerprints calculated for the wake phrase audio samples and verifying that they are below a threshold required to confirm the same user, but above a threshold that would indicate the same recording. Because multiple recordings may produce very similar fingerprints, it is possible to consider the length of the wake phrase audio, or to stretch the audio samples to a matching length and calculate the cumulative vector distance of small audio segments across the entire wake phrase audio. After verifying that there is sufficient difference between the wake phrase audio samples (114), if the distance is sufficient, the user is authenticated (115). Otherwise, the authentication fails (116) and the VVS sends a corresponding signal to the device requesting authentication.

[0077] Using this method, the same recording of a user's voice by an impostor cannot be used repeatedly to circumvent the system.

[0078] As an additional security feature to identify spoofing attacks in which an imposter records a legitimate user's voice, the VVS may further employ a voice verification module capable of distinguishing live voice from recorded voice. In particular, various acoustic differences exist between voice received from a live person (referred to herein as live voice) and voice played from a recording device (referred to herein as played voice). The verification module may operate according to any of a wide variety of techniques for detecting these differences between live and played voice.

[0079] In one embodiment, the audio verification module detects the presence of so-called pop noise. Pop noise is a distortion of the audio waveform that occurs when a human breath reaches a microphone. This same distortion phenomenon does not occur in played-back audio. Methods for using pop noise to detect differences between live and played-back audio are described, for example, in a 2015 Interspeech publication by Shioto et al. entitled "Voice Liveness Detection Algorithms Based on Pop Noise Caused by Human Breath for Automatic Speaker Verification," which is incorporated herein by reference. by detecting the presence or absence of pop noises, the audio verification module may identify whether the audio is live audio from a live user or played audio from playback device 102. The audio verification module may operate according to other techniques in further embodiments.

[0080] Vending Machine Scenario

[0081] Figure 12 illustrates another scenario. A voice-enabled vending machine 121 receives Juliet's utterance. She says, "Hello, Mr. Big. I'm Juliet." The vending machine connects to the VVS. It verifies Juliet's identity. The vending machine then gains access to certain financial information stored in Juliet's PEKD. The vending machine allows Juliet to make a purchase.

[0082] Building Security Scenario

[0083] Figure 13 illustrates another scenario. Consider a voice-activated lock / security system for a commercial building. Dave walks up to the building. Dave is an imposter who does not have permission to enter the building. Dave knows that Juliet works in the building. Dave utters the phrase, "Hello, Mr. Big. I'm Juliet." The voice-activated lock / security system sends a wake phrase to the VVS. The VVS checks the wake phrase against its stored fingerprint and finds that it does not match. The VVS signals an authentication failure. Dave is not allowed access to the building. Now Juliet walks up to the building. Juliet utters the wake phrase, "Hello, Mr. Big. I'm Juliet." The VVS recognizes Juliet's voice and allows Juliet access to the building.

[0084] Edge Processing

[0085] 14 shows another exemplary configuration: A user 141 verbally interacts with an edge device 142, which stores known users' wake phrases locally as text. Whenever a new user is registered or a user changes their wake phrase, the system transmits the new wake phrase to all devices the user may use, including the edge device 142.

[0086] During normal operation, it buffers audio. It also performs speaker-independent, continuous ASR and comparison of transcribed words with the wake phrase text. Upon finding a match between the spoken words and any user's wake phrase, the edge device transmits the buffered audio and either the wake phrase text and corresponding identifier to a VVS server 144 over a network 143, such as the Internet. The VVS retrieves the user's audio fingerprint corresponding to the wake phrase or identifier from a database and compares the audio with the stored fingerprint. The VVS sends an acceptance or rejection response to the device. The device uses the response to perform application-specific functions, such as unlocking a door for successful authentication or activating an alarm for unsuccessful authentication.

[0087] In another exemplary configuration, the edge device continuously transmits speech to the VVS, which performs speech recognition. This approach provides greater accuracy and continuously improves the server-based ASR. It also avoids the need to send wake phrase updates to the device.

[0088] In another exemplary configuration, the device performs voice activity detection and transmits the audio to the VVS upon detection, avoiding the high network bandwidth requirements that may be required for continuous audio streaming.

[0089] The device may detect the presence of a user using any combination of visual, infrared, ultrasonic, or other sensors such as RFID signals, possibly in combination with audio signals.

[0090] Database Management

[0091] Any product vendor can design their own virtual assistant (VA) Implementing a voice-over interface can be an inefficient use of know-how. VA platforms, such as Houndify by SoundHound, have been developed to meet the need for full-featured VA capabilities in many Internet-connected products. They allow products to provide a user interface that associates the user with the product or its brand, possibly without the knowledge that the platform server provides the VA. These services support voice interfaces, other modes of user interface, or a combination of these modes.

[0092] VAs are much more useful to their users when they use PEKD information such as the voice of the user's past queries, the textual transcriptions of those queries, the user's address book, location history, and the user's scores in a virtual game of blackjack. Some server-based VAs collect and store that information and use it to improve the user experience.

[0093] However, some users may wish to delete their PEKD from the VA server or obtain a copy of their PEKD. The user may make a request to delete or obtain a copy of the PEKD. The VA platform collects the information, but the user makes their request to the product vendor. The product vendor needs a way to make a corresponding request to the VA platform to delete or request a copy of at least some of the user data.

[0094] Figure 15 shows a VA platform 151 that includes multiple servers dedicated to different purposes. It receives requests from a network 15, such as the Internet. Proxy 152 verifies authorization. Authorization may be granted by a VVS or other verification system. In this exemplary system, contact server 155 stores user address book information, text server 156 stores user transcriptions and documents, game server 157 stores blackjack and other game scores, and parallel voice server system 158 stores user speech recordings. A virtual assistant platform that captures many user speech recordings may require distributed storage to process the voice recordings. Distributing storage has the benefit of distributing data access, which limits bandwidth requirements and improves latency and therefore system responsiveness by placing user data closer to them topologically in the network. The parallel voice servers use a map-reduce cluster of storage. This is achieved using Apache Hadoop, can be implemented using an open protocol, such as a cloud service specific protocol.

[0095] Such a VA platform may provide an API for providing delete and retrieve requests. An authentication proxy interface receives delete and retrieve requests from an external network, along with an optional indication of the data format. If the request is permitted, the authentication proxy forwards the delete and retrieve requests to the appropriate server. If the data format is not specified, the authentication proxy may forward the request to all servers. In this manner, the VA platform may use a distributed request architecture for the authentication proxy to execute delete and retrieve requests across all servers.

[0096] Figure 16 shows another implementation of the virtual assistant platform 161. Delete and retrieve requests from the network 150 go to the authentication proxy 152, which confirms authorization. If denied, the authentication proxy responds with an error signal. If authorization is confirmed, the authentication proxy sends the request to an open-source Kafka queue with a user data retrieval and deletion request topic. Other open-source or proprietary queuing systems are suitable for integration into different systems. All servers that store user data, such as the contact server 165, text server 166, game server 167, and servers in the parallel voice map-reduce cluster 168 system, must subscribe to the user data retrieval and deletion request topic with their own unique group ID. The authentication proxy subscribes to Kafka. After processing the delete and retrieve requests, each server posts a confirmation record to Kafka. If the server has the user data, it posts it as the result of the retrieval request via the authentication proxy API. The authentication proxy keeps track of the progress of the delete and retrieve requests and provides an API for reporting the request status.

[0097] Some servers, especially those dealing with storing large amounts of data, may use a distributed array of map-reduce clusters communicating via frameworks such as Apache Hadoop. Such servers must handle the distribution of delete requests in response to delete API requests via an authenticating proxy.

[0098] The Authentication Proxy API may require the requester to use a mutual authentication protocol such as Mutual Transport Layer Security (mTLS). Quasi- or proprietary mutual authentication protocols are suitable for different systems.

[0099] Boilerplate

[0100] Computer-readable medium

[0101] 17A shows an example of a non-transitory computer-readable medium 171 that is a rotating magnetic disk. Data centers typically use magnetic disks to store data and code, including instructions for server processors. The non-transitory computer-readable medium 171 stores code, including instructions, that, when executed by one or more computers, cause the computers to perform the steps of the methods described herein. Rotating optical disks and other mechanically movable storage media are possible.

[0102] 17B shows an example of non-transitory computer-readable medium 172 that is a flash random access memory (RAM) chip. Data centers typically use flash memory to store data and code for server processors. Mobile devices typically use flash memory to store data and code for processors in system-on-chip devices. Non-transitory computer-readable medium 172 stores code including instructions that, when executed by one or more computers, cause the computers to perform the steps of the methods described herein. Other non-movable storage media packaged with leads or solder balls are also possible.

[0103] System-on-Chip

[0104] 18A shows the bottom view of a packaged system-on-chip device 180 with a ball grid array for surface mount soldering to a printed circuit board. A variety of package shapes and sizes are possible for various chip implementations. System-on-chip (SoC) devices control many embedded system and IoT device embodiments as described herein.

[0105] FIG. 18B shows a block diagram of system-on-chip 180. It comprises a multi-core cluster of computer processor (CPU) cores 181 and a multi-core cluster of graphics processor (GPU) cores 182. The processor connects via network-on-chip 183 to an off-chip dynamic random access memory (DRAM) interface 184 for volatile storage of programs and data and a flash interface 185 for non-volatile storage of computer program code in a flash RAM non-transitory computer-readable medium. SoC 180 also has a display interface 186 for displaying a GUI and an I / O interface module 187 for connecting to various I / O interface devices as needed for different peripheral devices. The I / O interfaces enable sensors such as touchscreen sensors, positioning receivers, microphones, speakers, Bluetooth peripherals, and USB devices such as keyboards and mice, among others. SoC 180 also supports wireless communication standards such as WiFi, 3G, 4G long-term evolution (LTE), 5G, and other wireless interface standards. The processor 181 includes a network interface 188 for enabling the processor to access the Internet through a wired or wireless connection, such as Ethernet connection hardware, and a wired or wireless connection, such as Ethernet connection hardware. The CPU 181 and the GPU 182 perform the steps of the methods described herein by executing instructions stored in a RAM device via interface 184 or instructions stored in a flash device via interface 185.

[0106] server

[0107] 19A illustrates a rack-mounted server blade multiprocessor server system 190 according to some embodiments. The server system X0 includes multiple networked computer processors that execute software in parallel.

[0108] 19B shows a block diagram of server system 190. It comprises a multi-core cluster of computer processor (CPU) cores 191 and a multi-core cluster of graphics processor (GPU) cores 192. The processors connect via board-level interconnect 193 to random access memory (RAM) devices 194 for storage of program code and data. Server system 190 also comprises a network interface 198 that allows the processors to access the Internet. By executing instructions stored in RAM devices 194, CPU 191 and GPU 192 perform steps of the methods described herein.

[0109] Special Notes

[0110] The examples shown and described use a certain spoken language. Various embodiments work similarly for other languages ​​or combinations of languages. The examples shown and described use a certain domain with knowledge and capabilities. Various systems work similarly for other domains or combinations of domains.

[0111] Some systems are screenless, such as an earpiece. Some systems are stationary, such as a vending machine. Some systems are mobile, such as in an automobile. Some systems are portable, such as a cell phone. Some systems are implantable in the human body. Some systems have a manual interface, such as a keyboard or touchscreen.

[0112] Some systems function by running software on general-purpose programmable processors (CPUs), such as those with ARM or x86 architectures. Some power-sensitive systems and systems requiring particularly high performance, such as those for neural network algorithms, use hardware optimizations. Some systems use application-customizable processors with configurable instruction sets in dedicated systems-on-chips, such as Synopsys' ARC processors and Cadence's Xtensa processors. Some systems use dedicated hardware blocks baked into field-programmable gate arrays (FPGAs). Some systems use arrays of graphics processing units (GPUs). Some systems use application-specific integrated circuits (ASICs) with customized logic to provide higher performance.

[0113] Some systems of physical machines described and claimed herein are programmable in a multitude of variables, the combinations of which provide essentially an infinite variety of operational behaviors. Some systems herein are configured by software tools that provide a multitude of parameters, the combinations of which provide essentially an infinite variety of machine embodiments.

[0114] The hardware blocks, custom processor instructions, coprocessors, and hardware accelerators execute neural network processing, or portions of neural network processing algorithms, with particularly high performance and power efficiency, thereby extending battery life in battery-powered devices and reducing heat removal costs in data centers that simultaneously serve many client devices.

Claims

1. 1. A computer-implemented method for authenticating a user by voice, the method comprising: receiving a voice query from a voice-enabled device associated with a commercial system, the voice query including a wake phrase; authenticating a user based on the voice query; sending an authentication message to a personal data service provider including a personal database having personal data of a plurality of users, wherein authenticating the user based on the voice query includes authentication based on the wake phrase and an additional authentication; the commercial system is a POS (Point of Sale) system, The authentication message grants the commercial system access to at least a portion of the user's personal data.

2. The method of claim 1 , wherein the additional authentication comprises additional word authentication from the user.

3. The method of claim 1 , wherein the additional authentication comprises phrase-independent authentication.

4. The method of claim 1 , wherein the further authentication comprises comparing a fingerprint of the wake phrase with a fingerprint of a previous voice sample of the user.

5. The method of claim 1 , wherein the additional authentication comprises detecting a pop noise.

6. The method of claim 1 , wherein the user's personal data includes the user's financial information.

7. Authenticating a user is Computing audio feature vectors in a machine-learned embedding space; and The method of any one of claims 1 to 6, comprising comparing the speech feature vector in a machine-learned embedding space with user-specific speech feature vectors in a database.

8. Authenticating a user is Computing audio feature vectors in a machine-learned embedding space; and comparing the speech feature vector in the machine-learned embedding space with user-specific speech feature vectors in a database; The method of any one of claims 1 to 7, wherein the user-specific audio feature vector corresponds to a user having the wake phrase.

9. 1. An authentication system for authenticating a user by voice, the authentication system comprising: Equipped with personal data service providers, the personal data service provider includes a personal database; the personal database includes personal data of a plurality of users; the personal data service provider is capable of communicating with the commerce system via one or more networks; the commerce system is associated with a voice-enabled device; a voice verification server capable of granting said commercial system access to at least a portion of the user's personal data as a result of authenticating said user using said voice-enabled device; An authentication system, wherein the commercial system is a POS (Point of Sale) system.

10. 10. The authentication system of claim 9, wherein granting access to at least a portion of the user's personal data includes providing the commercial system with a key that enables it to gain access to the personal data service provider.

11. 11. The authentication system of claim 9 or claim 10, wherein the user's personal data includes the user's financial information.

12. 11. The authentication system according to claim 9 or claim 10, wherein the user's personal data is stored in a memory within the personal data service provider.

13. The authentication system of any one of claims 9 to 12, wherein authentication by the voice verification server includes calculation of a voice feature vector in a machine-learned embedding space.

14. The authentication system according to any one of claims 9 to 13, wherein the authentication by the voice verification server includes recognizing a wake phrase unique to the user.

15. The authentication system of claim 14 , wherein authentication by the voice verification server further comprises recognizing additional words after recognizing the wake phrase.

16. The authentication system of claim 14 , wherein authentication by the voice verification server further comprises phrase-independent authentication.

17. 15. The authentication system of claim 14, wherein authentication by the voice verification server includes comparing a fingerprint of the wake phrase with a fingerprint of a previous voice sample of the user.

18. 11. The authentication system of claim 9 or 10, wherein authentication by the voice verification server includes detecting the presence of pop noises.

19. The authentication system according to any one of claims 9 to 18, wherein the authentication system is distributed.

20. A program that, when executed by a computer, causes the computer to implement the method according to any one of claims 1 to 8.

Images