Voice privacy in far-field voice control devices using remote voice services
By using a multi-module system and integrated circuits to perform audio transformation and separation processing on voice data, the privacy issue when voice control devices transmit data to remote computers is resolved. This enables voice recognition that transmits only user intent-related data, thus protecting user privacy.
Patent Information
- Application Number
- CN202410096611.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2023-01-27
- Filing Date
- 2024-01-23
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-01-23
AI Technical Summary
Existing voice control devices pose privacy concerns when transmitting voice data to remote computers, potentially leading to the unauthorized processing of users' personal information for data mining purposes, such as the detection of identity, gender, age, emotional state, and accent.
It employs a multi-module system and integrated circuits to anonymize voice data through audio transformation, separates and combines data to generate anonymized audio data, and transmits only information related to the user's intent, avoiding the uploading of personally identifiable information.
It achieves the goal of protecting user privacy while maintaining voice recognition functionality, ensuring that only data related to the user's intent is transmitted, and preventing the leakage of personal information.
Smart Images

Figure CN118411995B_ABST
Abstract
Description
Technical Field
[0001] The exemplary embodiments disclosed herein may generally relate to systems, integrated circuits, and methods for far-field voice processing, and more specifically, to voice privacy of far-field voice control devices using remote voice services. Background Technology
[0002] Voice-controlled devices (e.g., smart speakers and voice assistants) typically have limited processing power and provide voice data to remote (e.g., cloud) computers for processing, such as speech recognition to interpret user commands. To this end, voice-controlled devices may include various software development kits (SDKs) for transmitting voice data to remote computers. However, using SDKs and offloading voice data to external devices allows remote computers to perform other processes unknown to the user to collect more information about the user, such as data mining of personally identifiable information.
[0003] Voice data can be used to extract various information, including identity, gender, age, emotional state, location, and accent. To protect user privacy, it is desirable to store or upload only data relevant to the speaker's direct intent / command to a remote computer. Therefore, private information should be removed from the voice signal before it is provided to the voice processing SDK. For example, if the intent is a voice recognition task, information about other irrelevant recognition activities (e.g., identity, gender, age, emotion, and accent) should be masked / retained before the voice signal is provided to the voice processing SDK. Summary of the Invention
[0004] An exemplary embodiment includes a system comprising a first module and a second module. The first module is configured to perform operations including: generating speech data based on input audio; anonymizing the speech data by applying a first audio transform; and transmitting the anonymized speech data to a first remote automatic speech recognition (ASR) module for generating speech recognition data. The second module is configured to perform operations including: separating the input audio into first data and second data; anonymizing the first data by applying a second audio transform; generating anonymized audio data by combining the anonymized first data and the second data; and transmitting the anonymized audio data to a second remote ASR module for generating speech recognition data.
[0005] An exemplary embodiment also includes an integrated circuit comprising a first module and a second module. The first module is configured to perform operations including: generating speech data based on input audio; anonymizing the speech data by applying a first audio transform; and transmitting the anonymized speech data to a first remote ASR module for generating speech recognition data. The second module is configured to perform operations including: separating the input audio into first data and second data; anonymizing the first data by applying a second audio transform to the first data; generating anonymized audio data by combining the anonymized first data and the second data; and transmitting the anonymized audio data to a second remote ASR module for generating speech recognition data.
[0006] An exemplary implementation further includes a method comprising receiving input audio and a reference signal and selecting one or more modules from a set of modules. In response to selecting a first module, the method includes: generating speech data based on the input audio; anonymizing the speech data by applying a first audio transform; and transmitting the anonymized speech data to a first remote ASR module for generating speech recognition data. In response to selecting a second module, the method further includes: separating the input audio into first data and second data; anonymizing the first data by applying a second audio transform to the first data; generating anonymized audio data by combining the anonymized first data and the second data; and transmitting the anonymized audio data to a second remote ASR module for generating speech recognition data. Attached Figure Description
[0007] The appended claims set forth certain features of the present technology. However, for illustrative purposes, several exemplary embodiments of the present technology are illustrated in the following figures.
[0008] Figure 1 This describes an exemplary network configuration for a voice control device based on one or more aspects of the present technology.
[0009] Figure 2 This describes an exemplary voice control device based on one or more aspects of the present technology.
[0010] Figure 3 A schematic diagram illustrating the first method for far-field voice control.
[0011] Figure 4 A schematic diagram illustrating the second method for far-field voice control.
[0012] Figure 5 A schematic diagram illustrating the third method for far-field voice control.
[0013] Figure 6A schematic diagram illustrating a voice anonymization system for far-field voice control based on one or more aspects of this technology.
[0014] Figure 7 A schematic diagram illustrating a voice privacy module based on one or more aspects of this technology.
[0015] Figure 8 A schematic diagram illustrating the data separation process according to one or more aspects of this technology.
[0016] Figure 9 A schematic diagram illustrating the data merging process according to one or more aspects of this technology.
[0017] Figure 10 A flowchart illustrating an example process for voice privacy in a far-field voice control device according to one or more aspects of the present technology.
[0018] For illustrative purposes only, the accompanying drawings depict various embodiments. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods described herein can be employed without departing from the principles described herein.
[0019] However, not all depicted components are available in all embodiments, and one or more embodiments may include additional components or components different from those shown in the figures. Variations in the arrangement and type of components may be made without departing from the spirit or scope of the claims set forth herein. Additional components, different components, or fewer components may be provided. Detailed Implementation
[0020] The detailed description below is intended to describe various configurations of the present technology, and not to represent the only configurations in which the present technology can be practiced. The accompanying drawings are incorporated herein and form part of the detailed description. The detailed description contains specific details to provide a thorough understanding of the present technology. However, the present technology is not limited to the specific details set forth herein and can be practiced using one or more other embodiments. In one or more embodiments, structures and components are shown in block diagram form to avoid obscuring the concept of the present technology.
[0021] Voice control devices (e.g., smart speakers and voice assistants) may include an SDK for transmitting voice data to a remote computer. However, using such an SDK and offloading voice data to an external device—as is done in current methods utilized by voice control devices—may allow the remote computer to perform other processing unknown to the user to collect more information about the user, such as data mining of personally identifiable information.
[0022] The use of far-field voice (FFV) control devices is for the effective recognition of commands spoken by users located far from the FFV control device in environments with varying noise levels, such as environments with background music or people talking. To achieve this, microphone arrays are typically used. An FFV processing module may first process the audio data captured by the microphone array to enhance the speech content of the audio data (e.g., by removing noise) and then provide the enhanced speech to an ASR module (e.g., a cloud ASR SDK) for recognizing commands within the speech content. If the audio data is sent to a remote computer (e.g., a cloud service), privacy concerns may arise because the audio samples may contain sufficient information for secondary purposes (e.g., detection of identity, gender, age, emotional state, accent, and similar traits).
[0023] Therefore, this aspect of the technology provides a single system for anonymizing audio data from multiple audio processing pipelines, which can use a remote computer for audio processing.
[0024] Figure 1 This describes an exemplary network configuration 100 of a voice control device according to one or more aspects of the present technology. The voice control device 102 may be a computer device (e.g., a set-top box, voice assistant, or the like) for receiving audio data that may contain commands 114 from a user 118. The audio data may be near-field or far-field audio data, wherein near-field audio data is close to the voice control device 102 (e.g., within 10 feet) and far-field audio data is far from the voice control device 102 (e.g., beyond 10 feet). The command 114 may be a process performed by the voice control device 102, such as a search query, setting a timer, playing music, or the like. The environment 116 in which the user provides the command 114 to the voice control device 102 may include noise 112, such as music, conversation, and any other ambient sounds.
[0025] Voice control device 102 receives audio data, which may include commands 114 from user 118 and noise 112 from environment 116. Voice control device 102 may anonymize the audio data before providing it to one or more ASR SDKs. Voice control device 102 may include multiple pipelines for anonymizing audio data, for example, to provide cross-compatibility between different ASR platforms. Voice control device 102 may also include pipelines for anonymizing audio data for local ASRs, wherein the local ASR includes ASRs that can be executed on voice control device 102 without the data leaving the device.
[0026] The ASR SDK of the voice control device 102 can provide anonymized audio data to a remote computer 106 (e.g., a cloud server) via network 104 for processing. The remote computer 106 can be any device external to the voice control device 102. For example, the remote computer 106 can be a cloud server located remotely from the voice control device 102. Processing may include speech separation, speech recognition, command recognition, and the like. The processing result can be used by the remote computer 106 to execute command 114 and provide the result of command 114 to the voice control device 102. The processing result may also, or alternatively, be sent to the voice control device 102 to execute command 114 and / or provide the result of command 114 as feedback 115 to the user 118.
[0027] Figure 2 A block diagram illustrating an exemplary computing system 200 according to one or more aspects of the present technology. For example... Figure 1 As shown, computing system 200 may be voice control device 102, and / or may be a part of voice control device 102. Computing system 200 may include various types of computer-readable media and interfaces for various other types of computer-readable media. Computing system 200 includes bus 210, processing unit 220, storage device 202, system memory 204, input device interface 206, output device interface 208, FFV module 212, ASR module 214, voice privacy module 216, and / or network interface 218.
[0028] Bus 210 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of computing system 200. In one or more embodiments, bus 210 communicatively connects processing unit 220 to other components of computing system 200. Processing unit 220 retrieves instructions to be executed and data to be processed from various memory locations to perform the operations of this disclosure. In various embodiments, processing unit 220 may be a controller and / or a single-core or multi-core processor.
[0029] Bus 210 is also connected to input device interface 206 and output device interface 208. Input device interface 206 enables the system to receive input. For example, input device interface 206 allows a user to convey information and select commands on system 200. Input device interface 206 can be used with input devices such as keyboards, mice, and other user input devices, as well as microphones (e.g., microphone arrays), cameras, and other sensor devices. For example, output device interface 208 can allow the display of images generated by computing system 200. Output devices that can be used with output device interface 208 may include, for example, printers and display devices such as liquid crystal displays (LCDs), light-emitting diode (LED) displays, organic light-emitting diode (OLED) displays, flexible displays, flat panel displays, solid-state displays, projectors, speakers (e.g., speaker arrays), haptic or any other device for outputting information. One or more embodiments may include devices that function as both input and output devices, such as touchscreens.
[0030] Bus 210 also connects system 200 to one or more networks and / or one or more network nodes via network interface 218. Network interface 218 may include one or more interfaces that allow system 200 to be part of a computer network (e.g., a local area network (LAN), a wide area network (WAN), or a network of networks (“Internet”). Any or all components of system 200 may be used in conjunction with this disclosure.
[0031] FFV module 212 may include hardware and / or software for processing far-field speech data. FFV module 212 may include one or more algorithms (e.g., computer-readable instructions) that include accessing audio input captured from a microphone array (e.g., input device interface 206) and separating and / or enhancing audio data from a target source (e.g., user 118) for applications such as ASR, which may use remote (e.g., cloud) voice services and / or local (e.g., edge) voice services. FFV module 212 may include one or more algorithms (e.g., computer-readable instructions) for acoustic echo cancellation (AEC), which can be used to remove audio output by system 200 that is subsequently captured back by system 200. For example, when a user command is received, system 200 may be playing music; the audio captured by the system's microphone may include music and user speech (e.g., commands). The AEC may include access to one or more reference signals to an integrated and / or external speaker (e.g., output device interface 208), receiving audio from a microphone (e.g., input device interface 206), and removing a portion of the received audio generated from the system 200 based on the reference signals, which is known to the system.
[0032] ASR module 214 may include hardware and / or software for performing ASR on speech data. Performing ASR on speech data may include receiving speech data and extracting speech recognition data. Speech recognition data may include speech, words, commands, intentions, and the like. ASR may be performed using hidden Markov models, dynamic time warping, machine learning models (e.g., neural networks), end-to-end ASR, and the like.
[0033] The voice privacy module 216 may include hardware and / or software for preparing voice data for input into one or more ASR SDKs. The voice privacy module 216 may include one or more algorithms for anonymizing voice data according to one or more voice processing pipelines, each pipeline comprising one or more steps, the output of which is the input of the next step. The pipelines may share resources with other pipelines and can operate concurrently. The following references... Figure 3 , Figure 4 , Figure 5 and Figure 6 Describe an example pipeline. In one or more implementations, the speech processing pipeline may be divided into its own modules, such that each module contains hardware and / or software for performing the speech processing pipeline.
[0034] Storage device 202 may be a read-and-write memory device. Storage device 202 may be a non-volatile memory cell that stores instructions and data (e.g., static and dynamic instructions and data) even when computing system 200 is powered off. In one or more embodiments, a mass storage device (e.g., a solid-state drive, disk drive, or optical disk drive and its corresponding disk drive) may be used as storage device 202. In one or more embodiments, a removable storage device (e.g., a floppy disk, flash drive, and its corresponding disk drive) may be used as storage device 202.
[0035] Similar to storage device 202, system memory 204 may be a read-write storage device. However, unlike storage device 202, system memory 204 may be a volatile read-write memory, such as random access memory. System memory 204 may store any of the instructions and data that one or more processing units 220 may need to perform operations during runtime. In one or more embodiments, the processes of this disclosure are stored in system memory 204 and / or storage device 202. One or more processing units 220 retrieve instructions to be executed and data to be processed from these various memory units to perform the processes of one or more embodiments.
[0036] The embodiments within the scope of this technology may be implemented in whole or in part using tangible computer-readable storage media (or multiple tangible computer-readable storage media of one or more types) encoding one or more instructions. Tangible computer-readable storage media may also be non-transitory in nature.
[0037] Computer-readable storage media can be any storage medium that can be read, written, or otherwise accessed by a general-purpose or special-purpose computing device, including any processing electronic device and / or processing circuitry capable of executing instructions. For example, but not limited to, computer-readable media can include any volatile semiconductor memory (e.g., system memory 204), such as RAM, DRAM, SRAM, T-RAM, Z-RAM, and TTRAM. Computer-readable media can also include any non-volatile semiconductor memory (e.g., storage device 202), such as ROM, PROM, EPROM, EEPROM, NVRAM, flash, SSD, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM, NRAM, racetrack memory, FJG, and millipede memory.
[0038] Furthermore, the computer-readable storage medium may comprise any non-semiconductor memory, such as optical disc storage, magnetic disk storage, magnetic tape, other magnetic storage devices, or any other medium capable of storing one or more instructions. In one or more embodiments, the tangible computer-readable storage medium may be directly coupled to a computing device, while in other embodiments, the tangible computer-readable storage medium may be indirectly coupled to a computing device, for example, via one or more wired connections, one or more wireless connections, or any combination thereof.
[0039] Instructions can be directly executable or can be used to develop executable instructions. For example, instructions can be implemented as executable or non-executable machine code or can be compiled to produce executable or non-executable machine code in a high-level language. Furthermore, instructions can also be implemented as data or can contain data. Computer executable instructions can also be organized in any format, including routines, subroutines, programs, data structures, objects, binaries, modules, applications, applets, functions, SDKs, frameworks, and the like. As those skilled in the art will recognize, the number, structure, order, and organization details of the instructions can vary significantly without altering the underlying logic, functionality, processing, and output. As used herein, the term "module" refers not only to computer executable instructions but also, or alternatively, to hardware (e.g., computer circuitry) capable of performing the processes described herein.
[0040] While the above discussion primarily concerns microprocessors or multi-core processors that execute software, one or more implementations are executed by one or more integrated circuits, such as DSPs, ASICs, or FPGAs. In one or more implementations, such integrated circuits execute instructions stored on the circuit itself.
[0041] Figure 3This diagram illustrates method 300 for FFV control. Method 300 includes separate FFV and ASR SDK modules. Voice anonymization is not performed in either module. Method 300 can be executed on a chip within voice control device 102. Microphones 302 and 304 of voice control device 102 can receive audio data from user 118 and / or environment 116. It should be understood that method 300 is not limited to a dual-microphone array and can control more or fewer microphones.
[0042] Microphones 302 and 304 transmit audio data to FFV module 212. The audio data is processed by FFV module 212, and the output of FFV module 212 is provided to ASR SDK 306. Processing may include speech extraction / enhancement. The output of FFV module 212 may include speech data of user 118 extracted from the audio data. ASR SDK 306 may send the speech data to remote computer 106 (e.g., a cloud server), and remote computer 106 may perform ASR on the speech data (e.g., determine commands in the speech data). The results of ASR from remote computer 106 may be sent to ASR SDK 306 for further processing and / or for providing speech recognition data 308 as output to the remainder of voice control device 102.
[0043] ASR SDK 306 may be provided by a third party (e.g., an ASR provider). From the perspective of the voice control device 102, ASR SDK 306 is a black box that receives audio data in one form and outputs speech recognition data 308, which may include commands, text, audio, and the like from the audio data, such as those determined by the remote computer 106 via ASR SDK 306.
[0044] Figure 4 A schematic diagram illustrating another method 400 for FFV control is provided. Method 400 includes a combined FFV and ASR SDK module (combined module 402). Voice anonymization is not performed in any module (e.g., combined module 402, FFV module 212, and / or ASR SDK 406) before voice data is sent to a remote computer (e.g., remote computer 106). Method 400 can be executed on a chip within a voice control device 102. Microphones 302, 304 of the voice control device 102 can receive audio data from a user 118 and / or an environment 116. It should be understood that method 400 is not limited to a dual-microphone array and can control more or fewer microphones.
[0045] Microphones 302 and 304 transmit audio data to the combination module 402. Like the ASR SDK 306, the combination module 402 may be provided by a third party and is considered a black box from the perspective of the voice control device 102. In method 400, how the audio data is processed (e.g., how speech data is generated from the audio data) and how the audio data is moved between the voice control device 102 and the remote computer 106 depends almost entirely on the combination module 402. The combination module 402 receives the audio data and outputs speech recognition data 408, which may include commands, text, and the like determined by the remote computer 106 via the ASR SDK 406 from the audio data. In one or more embodiments, the functionality of the FFV module 212 is performed at the remote computer 106.
[0046] Figure 5 A schematic diagram illustrating another method 500 for FFV control is provided. Method 500 includes an FFV module 212 and an ASR module 214. Method 500 can also be executed on a chip within a voice control device 102. Microphones 302 and 304 of the voice control device 102 can receive audio data from a user 118 and / or an environment 116. It should be understood that method 500 is not limited to a dual-microphone array and can control more or fewer microphones.
[0047] Microphones 302 and 304 transmit audio data to FFV module 212. The audio data is processed by FFV module 212, and its output is provided to ASR module 214. The output of FFV module 212 may include voice data from user 118. ASR module 214 may receive voice data for locally performing ASR on voice control device 102. In one or more embodiments, FFV module 212 and ASR module 214 may be combined modules, similar to combined module 402. The output speech recognition data 506 of ASR module 214 may include commands, text, and the like from the voice data.
[0048] and Figure 3 and Figure 4 Compared to other methods, this method offers the highest level of privacy because the voice samples do not leave the voice control device 102 (e.g., to a remote computer 106). Figure 5 The drawback of this method is that the voice control device 102 is limited by its own computing resources, which can negatively impact the performance of ASR.
[0049] Figure 6This illustration depicts a voice anonymization system 600 for FFV control according to one or more aspects of the present technology. The voice anonymization system 600 prepares audio data for ASR processing and allows the voice control device 102 to have compatibility across multiple ASR platforms while maintaining the privacy of the user 118. System 600 may be part of the voice control device 102 and may include an FFV module 212 and a voice privacy module 216. In one or more embodiments, system 600 may also include one or more of microphones 302, 304. For example, microphones 302, 304 may be connected to or integrated with system 600. System 600 may be a single integrated circuit on the voice control device 102. The microphones 302, 304 of the voice anonymization system 600 may receive audio data from the user 118, the environment 116, and / or a speaker 604 (e.g., integrated or connected to the voice control device 102). It should be understood that system 600 is not limited to a dual-microphone array as shown and may control more or fewer microphones.
[0050] Microphones 302 and 304 transmit audio data to the FFV module 212. The audio data is processed by the FFV module 212, and the output of the FFV module 212 is provided to the voice privacy module 216. Microphones 302 and 304 may also, or alternatively, transmit audio data to the voice privacy module 216 for use in subsequent modules that can be incorporated into the FFV.
[0051] At FFV module 212, audio data can be processed and provided to voice privacy module 216. Processing may include speech extraction / enhancement. Processing may also include acoustic echo cancellation, which removes reference signal 602 (e.g., audio output by voice control device 102) from the audio data. Reference signal 602 may be a signal received by output device interface 208 for output via speaker 604. Output device interface 208 may be external to 600 or part of 600 itself. The output of FFV module 212 may be voice data of user 118. In one or more embodiments, the output of FFV module 212 may also be output to ASR module 214 (internal or external to system 600), which in turn outputs speech recognition data 506 (e.g., words, commands, and the like) without using any remote computer or cloud service.
[0052] At the voice privacy module 216, the audio data is anonymized. The voice privacy module 216 receives audio data and anonymizes the audio data for the ASR SDK 306. The ASR SDK 306 can send the anonymized audio data to a remote computer 106 for ASR, receive speech recognition data 308 from the remote computer 106, and subsequently output speech recognition data 308 for the application. The voice privacy module 216 also or alternatively processes the audio data to separate the user 118's voice data from the remainder of the audio data, anonymize the voice data, and combine the anonymized voice data with the remainder of the audio data such that the audio data is not touched except for the user 118's voice data being anonymized. The voice privacy module 216 can output anonymized voice data to the combining module 402 (containing the ASR SDK 406), which can send the anonymized voice data to the remote computer 106 for ASR, receive speech recognition data 408 from the remote computer 106, and subsequently output speech recognition data 408 for the application. In one or more embodiments, the voice privacy module 216 also removes the reference signal 602 via AEC. Details of the process performed by the voice privacy module 216 will be provided below. Figure 7 , Figure 8 , Figure 9 and Figure 10 Let's discuss this in more detail.
[0053] In one or more implementations, one or more ASR SDKs and / or one or more ASR modules may be included as part of system 600. For example, extension 601 may be an extension of system 600, such that system 600 and extension 601 are on the same chip. As another example, extension 601 may be a separate chip connected to system 600.
[0054] Figure 7 A schematic diagram illustrating a voice privacy module 216 according to one or more aspects of the present technology is provided. The voice privacy module 216 anonymizes the voice data in the audio data, and then sends the anonymized audio data to the ASR SDK (e.g., for remote ASR).
[0055] Audio data may be received from at least one microphone (e.g., a microphone array). An FFV processing module (e.g., FFV module 212) may receive the audio data and output voice data of user 118 based on the audio data. A voice privacy module 216 may receive the voice data and anonymize the voice data in example 710 of the voice anonymization algorithm of anonymization module 706. The voice anonymization algorithm may be any voice anonymization method, such as a vocoder, x-vector-based speech transform, and the like, and may include transformations of any acoustic properties, such as pitch, formants, pitch variations, timbre, and the like, and / or non-acoustic properties, such as pitch variations and grammar. Voice privacy module 216 outputs anonymized voice data.
[0056] Alternatively or concurrently, the voice privacy module 216 receives audio data received from at least one microphone (e.g., a microphone array). The voice privacy module 216 may separate the audio data into at least voice data and noise data at the audio source separation module 704, and anonymize the voice data at the voice anonymization module 706 in example 708 of the voice anonymization algorithm of the anonymization module 706. Example 708 may utilize the same or different voice anonymization algorithm as example 710 and may be performed in the same or separate voice anonymization module 706. The anonymization module 706 may output anonymized voice data. The anonymized voice audio is combined with other audio data and then output as anonymized audio data.
[0057] In one or more embodiments, the audio data may be preprocessed at the microphone array preprocessing module 702 prior to audio source separation. The preprocessing module 702 is configured to enhance the performance of the audio source separation module 704 by modifying the audio data to emphasize the speech data of user 118. For example, the preprocessing module 702 may perform transformations on the audio, such as increasing the gain of the audio data and applying a high-pass filter to reduce the frequency to below the frequency of user 118's speech data. The preprocessed audio data may be output to the audio source separation module 704 to separate user 118's speech data from the remainder of the audio data.
[0058] In one or more embodiments, the preprocessing transformation applied at preprocessing module 702 can be reversed at inverse preprocessing module 714. Inverse preprocessing module 714 can modify anonymized audio data to return the audio data to its original state, but the speech data is anonymized. For example, if preprocessing module 702 applies gain enhancement and a high-pass filter to the audio data, then inverse preprocessing module 714 applies the inverse of the high-pass filter and gain enhancement to the anonymized audio data.
[0059] Figure 8A schematic diagram illustrating an audio source separation process according to one or more aspects of the present technology is provided. The audio source separation process may be performed by an audio source separation module 704. The audio source separation module 704 may include an echo cancellation module 802 configured to eliminate audio feedback (e.g., reference signal 602) from audio data played on the voice control device 102. To eliminate audio feedback, the audio source separation module 704 may perform acoustic echo cancellation to remove echoes generated based on reference signal 602 from the audio data (e.g., preprocessed audio data).
[0060] The audio source separation module 704 can also be configured to separate the audio data into at least speech data and noise data at the demixing module 804. To separate the audio data, the demixing module 804 can perform blind source separation, beamforming, or any other audio data separation algorithm. The output of the audio separation may contain speech data and noise data (e.g., background noise, output noise). The audio source separation module 704 can also be configured to apply further processing to the speech data at the post-gain module 806 to enhance the anonymization process (e.g., improve anonymization efficiency, reduce noise that may result from anonymization, and the like). For example, the post-gain module 806 can increase the gain of the audio data. The speech audio can be output to the speech anonymization module 706.
[0061] Figure 9 This diagram illustrates a data merging process according to one or more aspects of the present technology. The data merging process can be performed by an audio merging module 712. The audio merging module 712 can be configured to combine speech data, noise data, and / or reference signal 602 such that the output of the audio merging module is substantially the same as the audio data input to the speech privacy module 216, but the speech data is anonymized. To combine the audio data, the post-gain applied by the post-gain module 806 can be reversed by the inverse post-gain module 902. For example, if the post-gain module 806 increases the gain of the speech data, then the inverse post-gain module 902 can apply a decrease in the gain of the speech data.
[0062] The audio merging module 712 can also be configured to mix (e.g., combine) speech data with noise data. For example, mixing speech and noise data may involve reversing the separation performed at the demixing module 804. Therefore, the output of the mixing module 904 is anonymized audio data (e.g., an anonymized version of the original audio data), wherein the speech data is anonymized. The echo addition module 906 of the audio merging module 712 can mix (e.g., combine) the anonymized audio data with the reference signal 602 by reversing the acoustic echo cancellation algorithm used at the echo cancellation module 802. The output of the audio merging module 712 is substantially the same as the audio data input to the audio source separation module 704, but the speech data is anonymized.
[0063] Figure 10 A flowchart illustrating a process 1000 for voice privacy in an FFV control device according to one or more aspects of the present technology is provided. For illustrative purposes, process 1000 is described herein primarily with reference to the preceding figures. However, process 1000 is not limited to system 600, and one or more blocks (or operations) of process 1000 may be performed by one or more other components of other suitable devices. Furthermore, for illustrative purposes, the blocks of process 1000 are described herein as occurring sequentially or linearly. However, multiple blocks of process 1000 may occur in parallel. Moreover, blocks of process 1000 do not need to be performed in the order shown and / or one or more blocks of process 1000 do not need to be performed and / or may be replaced by other operations.
[0064] In process 1000, the speech data contained in the audio data may be anonymized while maintaining the audio data in a format compatible with one or more ASR SDKs (e.g., single-stream or multi-stream). In block 1002, input audio may be received by system 600. The input audio data may be received by one or more microphones (e.g., microphones 302, 304). The input audio data may include audio data from a user (e.g., user 118) or other sources (e.g., environment 116). In one or more embodiments, a reference signal (e.g., reference signal 602) may also be received from an output device interface 208 corresponding to the audio played on speaker 604, which is internal and / or external to the voice control device 102.
[0065] In box 1004, one or more audio data anonymization pipelines (or "pipelines") can be selected. The audio data anonymization pipeline can be embodied by one or more pipeline modules. A pipeline module may contain hardware and / or software that performs one or more processes described herein and may include other modules, including FFV module 212 and voice privacy module 216. System 600 may have a default selection of one or more audio data anonymization pipelines. For example, system 600 may be hardware-configurable to route audio data to one or more pipelines. Selection may also be configured or configured for an application (e.g., an ASR SDK, an audio recorder, a voice assistant, or the like) that utilizes system 600 to route audio data through one or more pipelines leading to a combination module 402, ASR SDK 306, and / or ASR module 214 suitable for the application. For example, an application may receive audio data as input instructing combination module 402 (containing ASR SDK 406) to be utilized by system 600. As another example, installing one or more components of an application onto a voice control device may involve sending a message to a controller (e.g., processing unit 220), which then sends an instruction to system 600 instructing system 600 to route audio data to one or more pipelines. As yet another example, combination module 402, ASR SDK 306, and / or ASR module 214 may send instructions to system 600 instructing system 600 to send audio data to one or more pipelines.
[0066] In one or more embodiments, the audio data anonymization pipeline used may depend on the type of ASR SDK utilized by the application of the voice control device. If the ASR SDK receives voice data (similar to method 300 for FFV control), then the process may proceed to block 1006. If the ASR SDK also or alternatively utilizes audio data (similar to method 400 for FFV control), then the process may also or alternatively proceed to block 1012. In one or more embodiments, the ASR may be performed locally, in which case the system 600 may also or alternatively perform FFV processing on the audio data via FFV module 212 and pass the output to local ASR module 214 to generate speech recognition data 506. In one or more embodiments, multiple methods may run in parallel to (e.g.) utilize multiple ASRs simultaneously (e.g., locally and / or externally).
[0067] In block 1006, processed audio data can be generated based on the received audio data. The audio data can be input to an FFV module 212 configured to extract, isolate, or enhance one or more sources (e.g., user 118's voice data) from the audio data. For example, noise data (e.g., background noise, device output noise) can be removed from the audio data to isolate user 118's voice data. The output of FFV module 212 can be user 118's voice data. FFV module 212 can perform one or more audio processing techniques, including acoustic echo cancellation, audio source separation, and the like. Acoustic echo cancellation can include removing echoes (e.g., audio output from voice control device 102) from the audio data based on reference signal 602.
[0068] In box 1008, anonymized audio data can be generated based on the transformation from processed audio data to audio data. A voice privacy module (e.g., voice privacy module 216) can be configured to anonymize the voice data and output the anonymized voice data to the ASR SDK (e.g., ASR SDK 306) such that personally identifiable information (e.g., identity, gender, age, accent, and the like) cannot be determined based on the anonymized voice data. The anonymized voice data may include the application of one or more transformations to the voice data. Transformations may include modulation frequencies, pitch, formants, and amplitude. Transformations may also, or alternatively, include non-acoustic changes, such as alterations to grammar and neutralization of pitch. The voice privacy module may include analog and / or digital signal processing methods, such as vocoders, x-vector-based speech conversion, and the like. It should be understood that voice privacy module 216 is not limited to any particular voice anonymization technique.
[0069] In box 1010, anonymized voice data can be transmitted to a remote ASR module (e.g., remote computer 106) for outputting speech recognition data (e.g., speech recognition data 308). An ASR SDK (e.g., ASR SDK 306) can be used in the voice control device (e.g., via hardware and / or software) to manage communication between the voice control device and the remote ASR module. The ASR SDK can receive anonymized voice data from the voice privacy module. The anonymized voice data can be sent to the remote ASR module associated with the ASR SDK for performing ASR. The ASR SDK can receive data (e.g., speech recognition data) from the remote ASR module and can also output the received data for further use (e.g., by an application on the voice control device).
[0070] For example, the first company can manufacture and configure voice control devices for consumer use. The second company can operate a cloud computing infrastructure with voice recognition capabilities. The second company can provide an ASR SDK to manufacturers (such as the first company) to leverage the second company's cloud computing infrastructure, which reduces the computing power required for the voice control device. Voice control devices utilizing this technology can extract, enhance, and / or anonymize user voice data to protect user privacy because the ASR SDK performs unknown operations on the voice data that may not include privacy protection measures.
[0071] In block 1012, audio data can be separated (e.g., by audio source separation module 704) into at least one speech data and one noise data (e.g., the remainder of the audio data). To separate the audio data, audio source separation algorithms, such as blind source separation and / or beamforming, can be used.
[0072] In one or more embodiments, the audio data may first be processed by an echo cancellation process (e.g., via echo cancellation module 802) to eliminate audio feedback (e.g., reference signal 602) from the audio data played on the voice control device. To eliminate audio feedback, an AEC algorithm may remove at least the audio created by the voice control device from the audio data based on the reference signal.
[0073] In one or more embodiments, after the audio source is separated, the speech data can be processed (e.g., via post-gain module 806) to improve the anonymization process. For example, the gain of the speech data can be increased to improve anonymization efficiency and / or reduce noise that may be caused by anonymization.
[0074] In one or more embodiments, the audio data may be preprocessed (e.g., via microphone array preprocessing module 702) before separation. The purpose of preprocessing the data is to enhance the performance of at least the audio source separation process. Transformations may be applied to the audio data to produce preprocessed audio data. The transformations may include the same or different aspects of the transformation from block 1008. The transformations applied to the audio data may include gain adjustment (e.g., gain boosting) and / or the application of a high-pass filter to cut off frequencies of the speech data below those of the user (e.g., user 118).
[0075] In block 1014, speech data may be anonymized (e.g., via speech anonymization module 706). Speech data may be anonymized in a manner similar to that used for the speech data in block 1008. For example, the anonymization process performed on the speech data may be a second example (e.g., example B708) of the anonymization process performed on the processed audio data in block 1008 (e.g., example A 710). In one or more embodiments, the anonymization at block 1008 may be modified randomly or otherwise for anonymizing the speech data. In one or more embodiments, the speech data may be anonymized in a manner different from that used for the processed audio data in block 1008 (e.g., a second, separate speech anonymization module may exist for the speech data, rather than a separate example of the same speech anonymization module).
[0076] In box 1016, anonymized speech data can be combined with the remaining audio data to form anonymized audio data. An audio merging module (e.g., audio merging module 712) can combine the speech data with noise data and / or a reference signal. The result will be anonymized audio data, similar to the audio data input to the speech privacy module, except that the user 118's speech data is anonymized so that no personally identifiable information about the user can be extracted from the audio data.
[0077] The audio merging module can invert the post-gain applied to the speech data at block 1012 (e.g., at post-gain module 806). For example, if the gain of the speech data increases by 0.5 dB, then the gain of the anonymized speech data can be reduced by 0.5 dB (e.g., at inverse post-gain module 902). Speech data and other audio data can also be mixed together. Mixing (e.g., combining) the components of the audio data together can include inverting the data separation performed at block 1012. For example, a portion of the blind source separation algorithm includes generating a set of demixing weights that can be used to derive the original audio mix, although speech anonymization has been performed. Echo cancellation can also be inverted from anonymized audio data (e.g., mixed audio data with anonymized speech data). For example, a portion of the acoustic echo cancellation can include generating a set of acoustic echo cancellation filter taps that can be used to derive the original audio mix containing a reference signal.
[0078] In one or more embodiments, the preprocessing applied at block 1012 is invertible. For example, if gain adjustment (e.g., gain boost) and a high-pass filter are applied to the audio data, then similar gain adjustment and high-pass filter will be inverted from the anonymized audio data.
[0079] In block 1010, anonymized audio data can be transmitted to a remote ASR module (e.g., remote computer 106) to generate speech recognition data (e.g., speech recognition data 408). An ASR SDK (e.g., ASR SDK 406 of combination module 402) can be used in a voice control device (e.g., via hardware and / or software) to manage communication between the voice control device and the remote ASR module. In one or more embodiments, the ASR SDK may be the same ASR SDK used after block 1008.
[0080] Those skilled in the art will understand that the various illustrative boxes, modules, elements, components, methods, and algorithms described herein can be implemented as electronic hardware, computer software, or a combination of both. To illustrate this hardware-software interchangeability, the various illustrative boxes, modules, elements, components, methods, and algorithms have been described above according to their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art can implement the described functionality in different ways for each specific application. For example, the described functionality can be implemented as various subroutines in a single software program executed by a processor. Various components and boxes can be arranged in different ways (e.g., in different orders or in different ways) without departing from the scope of this art.
[0081] It should be understood that any particular order or hierarchy of the boxes in the disclosed process is an illustration of the example method. Based on design preferences, it should be understood that the particular order or hierarchy of the boxes in the process may be rearranged, or all of the illustrated boxes may be executed. Any boxes may be executed concurrently. In one or more embodiments, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system components in the embodiments described above should not be construed as requiring this separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
[0082] As used in this specification and any claims of this application, the terms "base station," "receiver," "computer," "server," "processor," and "memory" all refer to electronic or other technical devices. These terms do not include persons or groups of people. For the purposes of this specification, the terms "display" or "showing" mean display on an electronic device.
[0083] As used herein, the phrase “at least one of…” preceding a series of items (where any items are separated by the terms “and” or “or”) modifies the entire list, not each member of the list (i.e., each item). The phrase “at least one of…” does not require selection of at least one of every listed items; rather, the phrase allows for the inclusion of at least one of any one item, and / or at least one of any combination of items, and / or at least one of each item. For example, the phrases “at least one of A, B, and C” or “at least one of A, B, or C” respectively refer to only A, only B, or only C; any combination of A, B, and C; and / or at least one of each of A, B, and C.
[0084] The predicates “configured to,” “operable to,” and “programmed to” do not imply any particular tangible or intangible limitation on the subject, but are intended to be used interchangeably. In one or more embodiments, a processor configured to monitor and control operations or components may also mean a processor programmed to monitor and control operations or a processor operable to monitor and control operations. Similarly, a processor configured to execute code may be interpreted as a processor programmed to execute code or operable to execute code.
[0085] Phrases such as "one aspect," "the other aspect," "some aspects," "one or more aspects," "an embodiment," "the other embodiment," "some embodiments," "one or more embodiments," "an example," "the other example," "some embodiments," "one or more embodiments," "a configuration," "the other configuration," "some configurations," "one or more configurations," "the present technology," "the present disclosure," "the content of the present disclosure," and other variations thereof and the like are for convenience purposes and do not imply that the disclosure associated with such phrases is necessary to the present technology, nor does it imply that such disclosure applies to all configurations of the present technology. The disclosure associated with such phrases may apply to all configurations, or one or more configurations. The disclosure associated with such phrases may provide one or more instances. For example, the phrase "aspect" or "some aspects" may refer to one or more aspects, and vice versa, and this also applies to other foregoing phrases.
[0086] The word “exemplary” is used herein to mean “serving as an example, illustration, or description.” Any embodiment described herein as “exemplary” or “example” is not necessarily to be construed as preferred or advantageous over other embodiments. Furthermore, with regard to the terms “comprising,” “having,” or the like as used in this description or claims, this term is intended to be inclusive in a manner similar to how the phrase “comprising” is interpreted when used as a transitional word in the claims.
[0087] All structural and functional equivalents of the elements described in this disclosure that are known or will be known thereafter by one of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended to be offered to the public, whether or not such disclosure is expressly stated in the claims. No claim element should be construed in accordance with 35 U.S.SC 112(f) unless the element is expressly stated using the phrase “component for…” or, in the case of a method claim, using the phrase “step for…”.
[0088] The foregoing description is intended to enable any person skilled in the art to implement the various aspects described herein. Various modifications to these aspects will readily be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects presented herein, but rather to conform to the full scope consistent with the language of the claims, wherein, unless specifically stated otherwise, references to singular elements do not mean “one and only one,” but rather “one or more.” Unless specifically stated otherwise, the term “some” refers to one or more. A masculine pronoun (e.g., his) includes a feminine pronoun (e.g., she has) and vice versa. Titles and subtitles (if any) are used for convenience only and do not limit this disclosure.
Claims
1. A system for speech processing, comprising: The first module in a set of modules is configured to perform the following operations: Generate speech data based on input audio; The speech data is anonymized by applying a first audio transformation; and The anonymized voice data is transmitted to a first remote automatic speech recognition (ASR) module to generate speech recognition data. and The second module in a set of modules is configured to perform operations including the following: The input audio is separated into first data and second data; The first data is anonymized by applying a second audio transformation to the first data; Anonymized audio data is generated by combining the anonymized first data and the second data; and The anonymized audio data is transmitted to a second remote ASR module to generate speech recognition data.
2. The system according to claim 1, further comprising: The controller is configured to perform operations including the following: Receive instructions from the application; and Select one or more modules from the group of modules based on the instructions.
3. The system according to claim 1, further comprising: The third module in the set of modules is configured to perform the following operations: Second voice data is generated based on the input audio; and The second voice data is provided as input to the local ASR module to generate voice recognition data.
4. The system of claim 1, wherein one or more of the first audio transformation or the second audio transformation comprises at least one of modulation frequency, pitch, formants, amplitude, pitch variation or grammar.
5. The system of claim 1, wherein separating the input audio into the first data and the second data comprises: Determine whether there is an echo in the input audio based on the reference signal; In response to determining that an echo exists in the input audio, the echo is removed from the input audio based on the reference signal; The first data and the second data are mixed from the input audio; and Increase the gain of the first data.
6. The system of claim 5, wherein combining the anonymized first data and the second data comprises: Reduce the gain of the anonymized first data; The anonymized first data and the second data are mixed to generate the anonymized audio data; and The reference signal is added to the anonymized audio data.
7. An integrated circuit for voice processing, comprising: The first module in a set of modules is configured to perform the following operations: Generate speech data based on input audio; The speech data is anonymized by applying a first audio transformation; and The anonymized voice data is transmitted to a first remote automatic speech recognition (ASR) module to generate speech recognition data. and The second module in the set of modules is configured to perform the following operations: The input audio is separated into first data and second data; The first data is anonymized by applying a second audio transformation to the first data; Anonymized audio data is generated by combining the anonymized first data and the second data; and The anonymized audio data is transmitted to a second remote ASR module to generate speech recognition data.
8. The integrated circuit according to claim 7, further comprising: The controller is configured to perform operations including the following: Receive instructions from the application; and Select one or more modules from the group of modules based on the instructions.
9. The integrated circuit according to claim 7, further comprising: The third module in the set of modules is configured to perform the following operations: Second voice data is generated based on the input audio; and The second voice data is provided as input to the local ASR module to generate voice recognition data.
10. The integrated circuit according to claim 7, wherein: One or more of the first audio transformation or the second audio transformation includes at least one of modulation frequency, pitch, formant, amplitude, pitch variation or grammar.
11. The integrated circuit of claim 7, wherein separating the input audio into the first data and the second data comprises: Determine whether there is an echo in the input audio based on the reference signal; In response to determining that an echo exists in the input audio, the echo is removed from the input audio based on the reference signal; The first data and the second data are mixed from the input audio; and Increase the gain of the first data.
12. The integrated circuit of claim 11, wherein combining the anonymized first data and the second data comprises: Reduce the gain of the anonymized first data; The anonymized first data and the second data are mixed to generate the anonymized audio data; and The reference signal is added to the anonymized audio data.
13. A method for speech processing, comprising: Receive input audio and reference signals; Select one or more modules from a set of modules; In response to selecting the first module from the set of modules, the following operations are performed: The first module is used to generate speech data based on the input audio and the reference signal; The first module is used to anonymize the speech data by applying a first audio transformation; and The first module is used to transmit the anonymized speech data to the first remote automatic speech recognition (ASR) module to generate speech recognition data. and In response to selecting the second module from the set of modules, the following operations are performed: The second module is used to separate the input audio into first data and second data; The second module is used to anonymize the first data by applying a second audio transformation to the first data; Anonymized audio data is generated by combining the anonymized first data and the second data; and The anonymized audio data is transmitted to a second remote ASR module to generate speech recognition data.
14. The method of claim 13, wherein separating the input audio into the first data and the second data comprises: The second module is used to remove the reference signal from the input audio; The second module is used to mix the first data and the second data from the input audio; and Use the second module to increase the gain of the first data.
15. The method of claim 14, wherein combining the anonymized first data and the second data comprises: Use the second module to reduce the gain of the anonymized first data; The second module is used to mix the anonymized first data and the second data to generate the anonymized audio data; and The reference signal is added to the anonymized audio data using the second module.
16. A system for speech processing, comprising: The first module in a set of modules is configured to perform the following operations: The input audio is separated into first data and second data; Anonymize the first data by applying a first audio transformation to the first data; Anonymized audio data is generated by combining the anonymized first data and the second data; as well as The anonymized audio data is transmitted to a first remote ASR module to generate speech recognition data. as well as The second module in a set of modules is configured to perform operations including the following: Voice data is generated based on the input audio; The speech data is anonymized by applying a second audio transformation; as well as The anonymized voice data is transmitted to a second remote automatic speech recognition (ASR) module to generate speech recognition data.
17. The system of claim 16, wherein separating the input audio into the first data and the second data comprises: Determine whether there is an echo in the input audio based on the reference signal; In response to determining that an echo exists in the input audio, the echo is removed from the input audio based on the reference signal; Mix the first data and the second data from the input audio; as well as Increase the gain of the first data.
18. The system of claim 16, further comprising: The third module in the set of modules is configured to perform the following operations: Second voice data is generated based on the input audio; as well as The second voice data is provided as input to the local ASR module to generate voice recognition data.
19. An integrated circuit for voice processing, comprising: The first module in a set of modules is configured to perform the following operations: The input audio is separated into first data and second data; Anonymize the first data by applying a first audio transformation to the first data; Anonymized audio data is generated by combining the anonymized first data and the second data; as well as The anonymized audio data is transmitted to a first remote ASR module to generate speech recognition data. as well as The second module in a set of modules is configured to perform operations including the following: Voice data is generated based on the input audio; The speech data is anonymized by applying a second audio transformation; as well as The anonymized voice data is transmitted to a second remote automatic speech recognition (ASR) module to generate speech recognition data.
20. The integrated circuit of claim 19, wherein separating the input audio into the first data and the second data comprises: Determine whether there is an echo in the input audio based on the reference signal; In response to determining that an echo exists in the input audio, the echo is removed from the input audio based on the reference signal; Mix the first data and the second data from the input audio; as well as Increase the gain of the first data.
21. The integrated circuit of claim 19, further comprising: The third module in the set of modules is configured to perform the following operations: Second voice data is generated based on the input audio; as well as The second voice data is provided as input to the local ASR module to generate voice recognition data.
22. A method for speech processing, comprising: Receive input audio and reference signals; The first module is used to separate the input audio into first data and second data; The first module is used to anonymize the first data by applying a first audio transformation to the first data; The first module is used to generate anonymized audio data by combining the anonymized first data with the second data; The first module is used to transmit the anonymized audio data to a first remote ASR module for generating speech recognition data. The second module generates speech data based on the input audio and the reference signal; The second module is used to anonymize the speech data by applying a second audio transformation; as well as The second module is used to transmit the anonymized speech data to a second remote automatic speech recognition (ASR) module to generate speech recognition data.
23. The method of claim 22, wherein separating the input audio into the first data and the second data comprises: Determine whether there is an echo in the input audio based on the reference signal; In response to determining that an echo exists in the input audio, the echo is removed from the input audio based on the reference signal; Mix the first data and the second data from the input audio; as well as Increase the gain of the first data.
24. The method of claim 22, further comprising: Second voice data is generated based on the input audio; as well as The second voice data is provided as input to the local ASR module to generate voice recognition data.
Citation Information
Patent Citations
Method for transmitting tone modulation voice in social network
CN108417223A
Privacy mode for wireless audio device
CN111527530A
