Speaker identity and content de-identification

Abstract

One embodiment of the present invention provides a method for speaker identity and content de-identification under privacy guarantees. The method includes receiving an input indicating a privacy protection level to enforce, extracting features from speech recorded in an audio recording, recognizing and extracting textual content from the speech, analyzing the textual content to recognize privacy-sensitive personal information about an individual, generating de-identified textual content by anonymizing the privacy-sensitive personal information to a degree that satisfies the privacy protection level and conceals the individual's identity, and mapping the de-identified textual content to a speaker who produced the speech. The method further includes generating a synthetic speaker identity based on other features that are dissimilar to the features to a degree that satisfies the privacy protection level, and synthesizing a new speech waveform based on the synthetic speaker identity to generate the de-identified textual content. The new speech waveform conceals the individual's identity.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE

[0002] Embodiments of the present invention relate generally to data privacy protection, and more particularly to methods and systems for speaker identity and content de-identification under data privacy assurances. Summary of the Invention

[0002] One embodiment of the present invention provides a method for speaker identity and content de-identification under data privacy assurance. The method includes receiving an input indicating at least one level of privacy protection requiring speaker identity and content de-identification to be enforced, and extracting features corresponding to a first speaker from a first speech emitted by the first speaker and recorded in a first speech recording. The method further includes recognizing and extracting text content from the first speech, analyzing the text content to recognize privacy-sensitive personal information corresponding to the first individual, and performing utility-preserving content de-identification on the text content to anonymize the privacy-sensitive personal information to a degree that satisfies the at least one level of privacy protection, thereby generating de-identified text content. The de-identified text content conceals the personal identity of the first individual. The method further includes mapping the de-identified text content to the first speaker, generating a synthetic speaker identity corresponding to the first speaker based on other features corresponding to the at least one other speaker, and synthesizing a new speech waveform to emit the de-identified text content based on the synthetic speaker identity. The other features corresponding to the at least one other speaker are dissimilar to the features corresponding to the first speaker to a degree that satisfies at least one level of privacy protection. The new speech waveform is different from the speech waveform of the first voice, and the new speech waveform conceals the personal identity of the first speaker.

[0003] Another embodiment of the present invention provides a system for speaker identity and content de-identification under data privacy assurance. The system includes at least one processor and a non-transitory processor-readable memory device storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations. The operations include receiving an input indicating at least one level of privacy protection requiring enforcement of speaker identity and content de-identification, and extracting features corresponding to a first speaker from a first speech emitted by the first speaker and recorded in a first speech recording. The operations include recognizing and extracting textual content from the first speech, analyzing the textual content to recognize privacy-sensitive personal information corresponding to a first individual, and performing utility-preserving content de-identification on the textual content to anonymize the privacy-sensitive personal information to a degree that satisfies the at least one level of privacy protection, thereby generating de-identified textual content. The de-identified textual content conceals the personal identity of the first individual. The operations further include mapping the de-identified text content to the first speaker, generating a synthetic speaker identity corresponding to the first speaker based on other characteristics corresponding to the at least one other speaker, and synthesizing a new speech waveform to emit the de-identified text content based on the synthetic speaker identity. The other characteristics corresponding to the at least one other speaker are dissimilar to the characteristics corresponding to the first speaker to a degree that satisfies at least one level of privacy protection. The new speech waveform differs from the speech waveform of the first speech, such that the new speech waveform conceals the personal identity of the first speaker.

[0004] One embodiment of the present invention provides a computer program product for speaker identity and content de-identification under data privacy assurance. The computer program product includes a computer readable storage medium having program instructions embodied therein. The program instructions, when executed by a processor, cause the processor to receive an input indicating at least one level of privacy protection requiring speaker identity and content de-identification to be enforced, and to extract features corresponding to a first speaker from a first speech emitted by the first speaker and recorded in a first speech recording. The program instructions further cause the processor to recognize and extract text content from the first speech, analyze the text content to recognize privacy-sensitive personal information corresponding to a first individual, and generate de-identified text content by performing utility-preserving content de-identification on the text content to anonymize the privacy-sensitive personal information to a degree that satisfies the at least one level of privacy protection. The de-identified text content conceals the personal identity of the first individual. The program instructions further cause the processor to map the de-identified text content to the first speaker, generate a synthetic speaker identity corresponding to the first speaker based on other characteristics corresponding to the at least one other speaker, and synthesize a new speech waveform based on the synthetic speaker identity to emit the de-identified text content. The other characteristics corresponding to the at least one other speaker are dissimilar to the characteristics corresponding to the first speaker to a degree that satisfies at least one level of privacy protection. The new speech waveform differs from the speech waveform of the first voice, and the new speech waveform conceals the personal identity of the first speaker.

[0005] These and other aspects, features and advantages of embodiments of the present invention will be understood by reference to the drawings and detailed description herein, and will be realized by the various elements and combinations particularly pointed out in the appended claims. It is to be understood that both the general description above, and the brief description of the drawings and the detailed description of the embodiments of the present invention below are exemplary and explanatory descriptions of preferred embodiments of the present invention, and are not limitations of the embodiments of the present invention, as claimed.

[0006] The subject matter which is regarded as embodiments of the present invention is particularly pointed out and distinctly claimed in the claims which follow the specification. The above and other objects, features and advantages of embodiments of the present invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0007] [Figure 1] 1 illustrates a cloud computing environment in accordance with an embodiment of the present invention. [Diagram 2] 1 illustrates an abstract model layer according to an embodiment of the present disclosure. [Diagram 3] 1 illustrates an exemplary computing architecture for implementing speaker identity and content de-identification, according to one embodiment of the present invention. [Figure 4] 1 illustrates an exemplary speaker identity and content de-identification system according to one embodiment of the present invention. [Diagram 5] 1 illustrates an exemplary annotation that provides a mapping between the speech waveform of a speech recording and the words spoken by the speaker to which the speech recording corresponds, according to one embodiment of the present invention. [Figure 6] 4 illustrates an exemplary graphical representation of a speaker's feature vector in a two-dimensional space, according to one embodiment of the present invention. [Figure 7] 4 is a flowchart of an exemplary process for speaker identity and content de-identification according to one embodiment of the present invention. [Figure 8] FIG. 1 is a high-level block diagram illustrating an information processing system useful for implementing embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The detailed description of the invention explains preferred embodiments of the invention, together with advantages and features, by way of example with reference to the drawings.

[0009] SUMMARY OF THE PRESENT EMBODIMENTS The present invention generally relates to data privacy protection, and more particularly to a method and system for speaker identity and content de-identification under data privacy assurance. One embodiment of the present invention provides a method for speaker identity and content de-identification under data privacy assurance. The method includes receiving an input indicating at least one level of privacy protection requiring speaker identity and content de-identification to be enforced, and extracting features corresponding to a first speaker from a first speech emitted by the first speaker and recorded in a first speech recording. The method further includes recognizing and extracting text content from the first speech, analyzing the text content to recognize privacy-sensitive personal information corresponding to a first individual, and performing utility-preserving content de-identification on the text content to anonymize the privacy-sensitive personal information to a degree that satisfies at least one level of privacy protection, thereby generating de-identified text content. The de-identified text content conceals the personal identity of the first individual. The method further includes mapping the de-identified text content to the first speaker, generating a synthetic speaker identity corresponding to the first speaker based on other features corresponding to the at least one other speaker, and synthesizing a new speech waveform to emit the de-identified text content based on the synthetic speaker identity. The other features corresponding to the at least one other speaker are dissimilar to the features corresponding to the first speaker to a degree that satisfies at least one level of privacy protection. The new speech waveform is different from the speech waveform of the first speech, and the new speech waveform conceals the personal identity of the first speaker.

[0010] Another embodiment of the present invention provides a system for speaker identity and content de-identification under data privacy assurance. The system includes at least one processor and a non-transitory processor-readable memory device storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations. The operations include receiving an input indicating at least one level of privacy protection requiring enforcement of speaker identity and content de-identification, and extracting features corresponding to a first speaker from a first speech emitted by the first speaker and recorded in a first speech recording. The operations further include recognizing and extracting textual content from the first speech, analyzing the textual content to recognize privacy-sensitive personal information corresponding to a first individual, and performing utility-preserving content de-identification on the textual content to anonymize the privacy-sensitive personal information to a degree that satisfies the at least one level of privacy protection, thereby generating de-identified textual content. The de-identified textual content conceals the personal identity of the first individual. The operations further include mapping the de-identified text content to the first speaker, generating a synthetic speaker identity corresponding to the first speaker based on other characteristics corresponding to the at least one other speaker, and synthesizing a new speech waveform to emit the de-identified text content based on the synthetic speaker identity. The other characteristics corresponding to the at least one other speaker are dissimilar to the characteristics corresponding to the first speaker to a degree that satisfies at least one level of privacy protection. The new speech waveform differs from the speech waveform of the first speech, such that the new speech waveform conceals the personal identity of the first speaker.

[0011] One embodiment of the present invention provides a computer program product for speaker identity and content de-identification under data privacy assurance. The computer program product includes a computer readable storage medium having program instructions embodied therein. The program instructions, when executed by a processor, cause the processor to receive an input indicating at least one level of privacy protection requiring speaker identity and content de-identification to be enforced, and to extract features corresponding to a first speaker from a first speech emitted by the first speaker and recorded in a first speech recording. The program instructions further cause the processor to recognize and extract text content from the first speech, analyze the text content to recognize privacy-sensitive personal information corresponding to a first individual, and generate de-identified text content by performing utility-preserving content de-identification on the text content to anonymize the privacy-sensitive personal information to a degree that satisfies the at least one level of privacy protection. The de-identified text content conceals the personal identity of the first individual. The program instructions further cause the processor to map the de-identified text content to the first speaker, generate a synthetic speaker identity corresponding to the first speaker based on other characteristics corresponding to the at least one other speaker, and synthesize a new speech waveform based on the synthetic speaker identity to emit the de-identified text content. The other characteristics corresponding to the at least one other speaker are dissimilar to the characteristics corresponding to the first speaker to a degree that satisfies at least one level of privacy protection. The new speech waveform differs from the speech waveform of the first voice, such that the new speech waveform conceals the personal identity of the first speaker.

[0012] For purposes of discussion, the term "de-identification" generally refers to the process of preventing an individual's personal identity from being revealed. An individual's personal identity includes information that reveals one or more private characteristics of the individual (i.e., privacy-sensitive personal information such as, but not limited to, gender, age, health, emotions, education, origin, etc.). A third party may be able to infer an individual's identity if the individual's privacy-sensitive personal information is revealed to the third party.

[0013] For purposes of discussion, the term "speech data" generally refers to data that includes one or more voice recordings of one or more speeches uttered by one or more speakers.

[0014] For purposes of discussion, the terms "speaker de-identification" and "voice de-identification" generally refer to the process of applying de-identification to audio data, including audio recordings of speech uttered by a speaker, to prevent the speaker's personal identity and voice from being revealed.

[0015] For purposes of discussion, the terms “textual content de-identification” and “content de-identification” generally refer to the process of applying de-identification to textual content that contains privacy-sensitive personal information about individuals to prevent the personal identity of the individuals from being revealed from the textual content.

[0016] For purposes of discussion, the term "speaker identity and content de-identification" generally refers to a process that applies both speaker de-identification and content de-identification to audio data.

[0017] For purposes of discussion, the term "direct identifier" generally refers to a data attribute, word, token, or value that can be used alone to identify an individual. A direct identifier may uniquely correspond to an individual such that when present in data, it reveals the identity of the corresponding individual. Examples of direct identifiers include, but are not limited to, a person's name, social security number, national identification number, credit card number, telephone number, medical record number, IP address, account number, etc.

[0018] For purposes of exposition, the term "indirect identifier" or "quasi-identifier" generally refers to a data attribute, word, token, or value that cannot be used alone to identify an individual, but may be used in combination with one or more other indirect / quasi-identifiers to identify an individual. The combination of indirect / quasi-identifiers corresponding to an individual is unique or extremely rare such that the combination, if present in the data, may reveal the identity of the corresponding individual or the combination may be linked to an identity with a record in an external publicly available dataset (e.g., voter registration list, decennial record, US census, etc.) that contains the name of the corresponding individual. Examples of indirect / quasi-identifiers include, but are not limited to, date of birth, gender, zip code, etc. For example, for a large portion of the US population, the combination of date of birth, gender, and a five-digit zip code is unique.

[0019] Embodiments of the present invention provide a method and system for voice and content de-identification of audio recordings that protects the personal identity of the speaker emitting the speech recorded in the audio recording, as well as protecting privacy-sensitive personal information contained in the textual content of the speech.

[0020] A speaker producing a voice produces a human voice that carries with it audio signals indicative of the speaker's privacy-sensitive personal information. For example, the timbre of a speaker's vocalizations typically conveys the speaker's most personal identifying information. Because no two individuals produce identical sounds, an individual's human voice can be used as an individual's identity by combining one or more physiological characteristics of the speaker's vocal tract system (e.g., vocal cords, vocal tract shape, timbre, pitch, etc.) or one or more behavioral characteristics of the human voice (e.g., rhythm, intonation, vocabulary, accent, pronunciation, speaking style, etc.), or a combination thereof, into a unique biometric pattern (signature) for the individual.

[0021] The recent ubiquity of automatic speaker verification (ASV) systems has created a need to effectively protect the personal identity of speakers in audio data. Furthermore, because audio may contain sensitive content by nature, privacy measures to protect the content are necessary in accordance with existing data privacy laws. For example, audio data, including audio clinical data (e.g., audio recordings of clinicians contained in electronic health records and documentation of clinicians' medical encounters with patients), may contain privacy-sensitive personal information, such as protected health information (PHI) about patients, and such data must be de-identified before sharing the data with one or more third parties for secondary use (e.g., sharing to support medical research).

[0022] Conventional solutions for speaker de-identification utilize voice transformation (VT), a technique that alters the original non-linguistic characteristics of the spoken utterance and anonymizes the speaker's voice without affecting the content of the speech. Specifically, VT alters the speaker's voice via (1) source modification, which involves altering the time scale, pitch or energy of the speaker's voice, or a combination thereof, (2) filter modification, which involves altering the speaker's timbre, or (3) a combination of source modification and filter modification.

[0023] Voice conversion is a special form of VT that involves mapping characteristics of a speaker's voice (i.e., the source speaker's voice) to characteristics of another individual's voice (i.e., the target speaker's voice). The source speaker can utilize voice conversion to imitate / mimic the target speaker's voice. Voice conversion requires both the source speaker and the target speaker to use the same corpus to generate voiced utterances of the same text for training purposes.

[0024] The present invention provides a method and system for speaker de-identification that utilizes a state-of-the-art feature vector extraction approach operating on audio data for ASV, building or creating speaker identities for different individuals while providing data privacy guarantees. In one embodiment, privacy is protected by controlling the textual content of the audio data by anonymizing the textual content. The present invention provides a novel approach that combines speaker de-identification and textual content de-identification, concealing the speaker's identity and anonymizing the textual content of the audio while providing state-of-the-art data privacy guarantees. This approach can be used in a wide range of real-world applications to anonymize audio data and audio recordings in an effective and provable manner and to facilitate secondary uses of the resulting anonymized audio data and audio recordings.

[0025] Although this disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings detailed herein is not limited to a cloud computing environment. Rather, embodiments of the invention may be implemented in conjunction with any other type of computing environment now known or hereafter developed.

[0026] Cloud computing is a model of service delivery that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with a service provider. In one embodiment, this cloud model includes at least five characteristics, at least three service models, and at least four deployment models.

[0027] The characteristics are as follows: On-Demand Self-Service: Cloud consumers can unilaterally provision computing power, such as server time and network storage, as they need it automatically, without the need for human interaction with the service provider. Broadband Network Access: Capabilities are available over the network and accessed via standard mechanisms that facilitate use by heterogeneous thin-client or thick-client platforms (e.g., mobile phones, laptops, personal digital assistants). Resource Pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where various physical and virtual resources are dynamically allocated and reallocated according to demand. Consumers generally have no control or knowledge of the exact location of the resources provided to them, but there is a sense of location independence in the sense that they can specify location at a higher level of abstraction (e.g. country, state, or data center). Rapid Elasticity: Capacity can be rapidly and elastically, in some cases automatically, provisioned to quickly scale out and rapidly released to quickly scale in. To the consumer, the provisioned available capacity often appears externally unlimited, and any amount can be purchased at any time. Measured Services: Cloud systems automatically control and optimize resource usage by utilizing metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, number of active users). Resource usage is monitored, controlled, and reported, providing transparency to both providers and consumers of the services being used.

[0028] The service model is as follows: Software as a Service (SaaS): The capability offered to the consumer is to use a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying infrastructure, including networks, servers, operating systems, storage, or even individual application capabilities with the potential exception of limited user-specific application configuration settings. Platform as a Service (PaaS): The capability offered to a consumer is to deploy consumer-created or acquired applications, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does have control over the deployed applications and possibly the configuration of the application hosting environment. Infrastructure as a Service (IaaS): The capability provided to the consumer is to provide processing, storage, networking, and other basic computing resources onto which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does have control over the operating systems, storage, deployed applications, and in some cases, limited control over selected networking components (e.g., host firewalls).

[0029] The deployment model is as follows: Private Cloud: The cloud infrastructure is used exclusively for one organization. In one embodiment, this may be managed by the organization or a third party and may exist on-premise or off-premise. Community Cloud: Cloud infrastructure is shared by several organizations to support a specific community with common concerns (e.g., mission, security requirements, policy and compliance considerations). In one embodiment, this may be managed by the organization or a third party and may exist on-premise or off-premise. Public Cloud: The cloud infrastructure is available to the general public or large industry entities and is owned by organizations that sell cloud services. Hybrid Cloud: A cloud infrastructure is a blend of two or more clouds (private, community or public) that remain unique entities but are joined by standardized or proprietary technologies that allow data and application portability (e.g. cloud bursting for load balancing between clouds).

[0030] Cloud computing environments are service-oriented with an emphasis on statelessness, low coupling, modularity and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0031] FIG. 1 illustrates a cloud computing environment 50 according to an embodiment of the present invention. As shown, in one embodiment, the cloud computing environment 50 includes one or more cloud computing nodes 10 that communicate with local computing devices used by cloud consumers, such as, for example, a PDA or cell phone 54A, a desktop computer 54B, a laptop computer 54C, or an automobile computer system 54N, or a combination thereof. In one embodiment, the nodes 10 communicate with each other. In one embodiment, they are physically or virtually grouped (not shown) in one or more networks, such as private, community, public, or hybrid clouds, as described above, or a combination thereof. This enables the cloud computing environment 50 to provide infrastructure, platform, or software, or a combination thereof, as a service, for which the cloud consumer does not need to maintain resources on a local computing device. The types of computing devices 54A-54N shown in FIG. 1 are for illustrative purposes only, and it will be understood that the computing node 10 and cloud computing environment 50 can communicate with any type of computerized device via any type of network, a network-addressable connection (e.g., using a web browser), or both.

[0032] 2 illustrates a set of functional abstraction layers provided by cloud computing environment 50, in accordance with an embodiment of the present invention. It will be understood that the components, layers, and functions illustrated in FIG. 2 are for illustrative purposes only, and that embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0033] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframes 61, servers based on RISC (Reduced Instruction Set Computing) architecture 62, servers 63, blade servers 64, storage devices 65, and networks and networking components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0034] In one embodiment, the virtualization layer 70 provides an abstraction layer from which examples of virtualized entities are provided, such as virtualized servers 71, virtualized storage 72, virtualized networks including virtual private networks 73, virtualized applications and operating systems 74, and virtual clients 75.

[0035] In one embodiment, the management layer 80 provides the functions described below. Resource provisioning 81 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 provides tracking of costs as resources are utilized within the cloud computing environment and billing or invoicing for consumption of these resources. In one embodiment, these resources include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides allocation and management of cloud computing resources to meet required service levels. Service level agreement (SLA) planning and fulfillment 85 provides pre-allocation and procurement of cloud computing resources in anticipation of future demand in accordance with SLAs.

[0036] In one embodiment, the workload layer 90 provides examples of functionality for which a cloud computing environment is utilized. In one embodiment, example workloads and functions provided from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instructional delivery 93, data analytics processing 94, transaction processing 95, and speaker identity and content de-identification 96 (e.g., speaker identity and content de-identification system 330, described in more detail later herein).

[0037] 3 illustrates an exemplary computing architecture 300 for implementing speaker identity and content de-identification according to one embodiment of the present invention. In one embodiment, the computing architecture 300 is a centralized computing architecture. In another embodiment, the computing architecture 300 is a distributed computing architecture.

[0038] In one embodiment, computing architecture 300 includes computational resources such as, but not limited to, one or more processor units 310 and one or more storage units 320. One or more applications may execute / run on computing architecture 300 utilizing the computational resources of computing architecture 300. In one embodiment, applications on computing architecture 300 include, but are not limited to, a speaker identification and content de-identification system 330 for speaker identification and content de-identification.

[0039] As described in more detail later herein, in one embodiment, the system 330 is configured to receive a dataset (e.g., a collection of audio data) including a set of audio recordings of a set of speakers, where textual content of the audio recordings includes privacy-sensitive personal information about the set of individuals. The system 330 is configured to apply speaker de-identification and content de-identification to at least one of the audio recordings to conceal the identity of at least one of the speakers and to anonymize the privacy-sensitive personal information about at least one of the individuals, resulting in at least one de-identified audio recording from which the identity of the at least one speaker and the privacy-sensitive personal information about the at least one individual cannot be inferred. Each resulting de-identified audio recording may be shared with one or more third parties for secondary uses (e.g., to support medical research).

[0040] For example, in one embodiment, the dataset is audio clinical data including audio recordings of a clinician contained in an EHR documenting the clinician's medical encounter with a patient, where the textual content of the audio recordings encompasses PHI about the patient. The system 330 extracts a subset of the audio clinical data (i.e., extracts one or more of the audio recordings), de-identifies the extracted subset (i.e., applies speaker identity and content de-identification to the extracted audio recordings), conceals the identity of the clinician, and anonymizes the PHI about the patient. Because privacy-sensitive personal information (e.g., PHI) about the patient cannot be inferred from the resulting de-identified extracted subset (i.e., the resulting de-identified audio recordings), the de-identified extracted subset may be shared with one or more third parties to support medical research.

[0041] In one embodiment, the system 330 is embedded / integrated into a cloud computing environment (such as, for example, IBM Cloud®).

[0042] In one embodiment, the speaker identity and content de-identification system 330 is configured to exchange data with one or more electronic devices 350 or one or more remote server devices 360, or both, via a connection (e.g., a Wi-Fi® connection or a cellular data connection, a wired connection, or a combination of the two).

[0043] In one embodiment, electronic device 350 includes one or more computational resources, such as one or more processor units 351 and one or more storage units 352. One or more applications may execute / run on electronic device 350 utilizing the one or more computational resources of electronic device 350, such as, but not limited to, one or more software applications 354 that are loaded or downloaded to electronic device 350. Examples of software applications 354 include, but are not limited to, artificial intelligence (AI) applications, big data analytics applications, etc.

[0044] Examples of electronic devices 350 include, but are not limited to, desktop computers, mobile electronic devices (e.g., tablets, smartphones, laptops, etc.), wearable devices (e.g., smart watches, etc.), Internet of Things (IoT) devices, smart appliances such as smart televisions, etc.

[0045] In one embodiment, electronic device 350 includes one or more input / output (I / O) units 353, such as a keyboard, keypad, touch interface, display screen, etc., incorporated into or coupled to electronic device 350. A user may utilize I / O unit 353 of electronic device 350 to configure one or more user settings, configure one or more parameters (e.g., thresholds, boundaries, etc.), provide input (e.g., requests), etc.

[0046] In one embodiment, the electronic device 350 or the remote server device 360 ​​or both may be a data source providing the data set for speaker identification and content de-identification.

[0047] In one embodiment, the speaker identity and content de-identification system 330 may be accessed or utilized by one or more online services (e.g., AI services, big data analytics services, data processing services) hosted on a remote server device 360 ​​and / or one or more software applications 354 (e.g., AI applications, big data analytics applications, data processing applications) running on the electronic device 350.

[0048] FIG. 4 illustrates an exemplary speaker identity and content de-identification system 330 according to an embodiment of the present invention. In one embodiment, the system 330 includes an input unit 400 configured to receive a dataset including a collection of audio data. In one embodiment, the collection of audio data includes R audio recordings, where R is a positive integer. The R audio recordings are original audio recordings of speech produced by S speakers, where S is a positive integer. Specifically, for each of the S speakers, the R audio recordings include at least one corresponding audio recording of at least one speech produced by the speaker. The textual content of the speech produced by the S speakers includes privacy-sensitive personal information about (i.e., relating to, involving) P individuals, such as PHI or other personally identifiable information (PII), where P is a positive integer.

[0049] In one embodiment, the system 330 includes a feature extraction unit 410. For each of the S speakers, the feature extraction unit 410 is configured to generate a corresponding feature vector based on at least one audio recording (of the R audio recordings) corresponding to the speaker. Specifically, in one embodiment, the feature extraction unit 410 is configured to: (1) for each of the R audio recordings, extract language and speaker identity features from the audio recording (i.e., features are extracted from the audio recorded in the audio recording); and (2) for each of the S speakers, generate a corresponding feature vector based on the language and speaker identity features extracted from all audio recordings corresponding to the speaker. For example, if the R audio recordings include two audio recordings that both correspond to the same speaker, the feature extraction unit 410 generates a feature vector corresponding to the speaker based on the language and speaker identity features extracted from the audio recorded in the two audio recordings.

[0050] Examples of linguistic and speaker identity features that may be extracted from an audio recording corresponding to a speaker include, but are not limited to, one or more physiological characteristics of the speaker's vocal tract system (e.g., vocal cords, vocal tract shape, tone, pitch, etc.), one or more behavioral characteristics of the speaker's voice (e.g., rhythm, intonation, vocabulary, accent, pronunciation, conversational style).

[0051] In one embodiment, feature extraction unit 410 generates a feature vector corresponding to a speaker by training the feature vector based on language and speaker identity features extracted from all audio recordings corresponding to the speaker.

[0052] In one embodiment, the feature extraction unit 410 utilizes a feature vector extraction approach such as, but not limited to, x-vector, i-vector, etc. In another embodiment, the feature extraction unit 410 utilizes a Fourier transform or a fast Fourier transform (FFT) to decompose the unique tones of the S speakers and isolate the voiceprints of the S speakers. The unique speakers may be split into separate audio files for processing.

[0053] In one embodiment, all feature vectors generated by feature extraction unit 410 are persisted in a database (eg, storage unit 320).

[0054] In one embodiment, the system 330 includes a text content extraction unit 420. For each of the R speech recordings, the text content extraction unit 420 is configured to recognize and extract text content (i.e., a transcript) from the speech recording (i.e., the text content is extracted from the speech recorded in the speech recording), where the text content includes words (or terms) spoken or uttered by the speaker to which the speech recording corresponds (and which uttered the speech). In one embodiment, the text content extraction unit 420 utilizes a speech recognition approach or engine such as, but not limited to, Dragon®, IBM® Speech To Text, Temi, Transcribe, etc. to recognize and extract text content from the speech recordings.

[0055] In one embodiment, for each of the R speech recordings, the textual content extraction unit 420 is optionally configured to generate a corresponding annotated text based on the textual content recognized and extracted from the speech recording, where the annotated text is an annotation that provides a mapping between the speech waveform of the speech recording and the words (or terms) spoken or uttered by the speaker to which the speech recording corresponds.

[0056] In one embodiment, the textual content extraction unit 420 is configured to link an audio waveform of the audio recording with a segment (i.e., a portion) of textual content recognized and extracted from the audio recording, where the segment contains privacy-sensitive personal information (e.g., PHI or other PII) about one of the P individuals. In one embodiment, the segment is pre-filtered to obfuscate the segment before speaker identity and content de-identification of the audio recording is completed. In one embodiment, the audio waveform is flagged or marked to identify the audio recording as one of the R audio recordings that cannot be released or disclosed to a third party for secondary use without inspection (e.g., the audio recording must undergo speaker identity and content de-identification before release).

[0057] In one embodiment, the system 330 includes a masking and tagging unit 430. For each of the R voice recordings, the masking and tagging unit 430 is configured to generate corresponding processed text content by recognizing and processing privacy-sensitive personal information (e.g., PHI or other PII) contained in the recognized and extracted text content from the voice recordings. Specifically, for each of the R voice recordings, the masking and tagging unit 430 is configured to (1) receive (e.g., from the text content extraction unit 420) the recognized and extracted text content from the voice recordings, (2) analyze the text content utilizing at least one natural language processing (NLP) annotator to recognize (i.e., discover) at least one direct identifier and / or at least one quasi-identifier in the text content, and (3) process each identifier (i.e., direct identifier and / or quasi-identifier) ​​recognized in the text content based on the type of identifier, resulting in processed text content corresponding to the voice recording.

[0058] For purposes of discussion, the term "PII words" generally refers to words (or terms) in text content that are either direct identifiers or quasi-identifiers. For purposes of discussion, the term "non-PII words" generally refers to words (or terms) in text content that are neither direct identifiers nor quasi-identifiers. Non-PII words cannot be linked to an individual's personal identity.

[0059] In one embodiment, the masking and tagging unit 430 processes direct identifiers recognized in the text content by masking (i.e., replacing) the direct identifier in the text content with a masked value (i.e., a replacement value) based on the type of the direct identifier. For example, in one embodiment, if the direct identifier recognized in the text content is a name, the masking and tagging unit 430 replaces the direct identifier in the text content with a random name (e.g., extracted from a dictionary or from a publicly available dataset such as a voter registration list) or a pseudonym (e.g., "Patient 1234"). Alternatively, the masking and tagging unit 430 processes direct identifiers recognized in the text content by hiding the direct identifier from the text content.

[0060] In one embodiment, the masking and tagging unit 430 processes quasi-identifiers recognized in the textual content by tagging the quasi-identifiers with one or more tags based on the type of quasi-identifier (e.g., age, gender, date, zip code, etc.) For example, in one embodiment, if a quasi-identifier recognized in the textual content is an age, the masking and tagging unit 430 is configured to tag the quasi-identifier with one or more tags indicating that the quasi-identifier is an age.

[0061] In one embodiment, if a segment (i.e., portion) of the text content contains an unidentified concept (e.g., not recognized by an NLP annotator) or indecipherable audio, the masking and tagging unit 430 is configured to annotate or mark the segment as “unknown” such that the segment is ignored for further processing by the system 330.

[0062] A speaker's word choice (i.e., phrasing) may be characteristic of the speaker's identity and may reveal the speaker's identity. In one embodiment, the system 330 includes a word replacement unit 440. For each of the R speech recordings, the word replacement unit 440 is configured to replace some words in the corresponding processed text content with synonyms to preserve the speaker's phrasing to which the speech recording corresponds. Specifically, for each of the R speech recordings, the word replacement unit 440 is configured to (1) receive the processed text content corresponding to the speech recording (e.g., from the masking and tagging unit 430), (2) select at least one word in the processed text content that is either a quasi-identifier or a non-PII word, and (3) replace each selected word with a synonym (i.e., synonym) using a thesaurus, lookup table, or vocabulary database (e.g., Wordnet). In one embodiment, if the processed textual content includes healthcare terminology (e.g., the audio data is audio clinical data), the word replacement unit 440 may replace medical terms in the processed textual content with SNOMED (Systematically Organized Computer Processable Collection of Medical Terms) codes or ICD-9 (International Classification of Diseases, Ninth Revision) codes to which the medical terms are mapped.

[0063] In one embodiment, for each of the S speakers, the synonyms used as replacement words are the same across (i.e., globally) all processed textual content corresponding to the same speaker (i.e., all processed textual content corresponding to all audio recordings corresponding to the same speaker), thereby protecting the identity of the speaker and preserving the usefulness of all processed textual content. For example, in one embodiment, the word replacement unit 440 may replace each occurrence of the word "found" in all processed textual content with the same synonym "discovered" and may replace the word "elevated" in all processed textual content with the same synonym "increased" (the same synonyms are used across all processed textual content).

[0064] In one embodiment, the system 330 includes a text document generation unit 450. For each of the P individuals, the text document generation unit 450 is configured to generate a corresponding text document by combining all processed text content corresponding to the same individual (i.e., all processed text content including privacy-sensitive personal information about the same individual). Specifically, in one embodiment, for each of the P individuals, the text document generation unit 450 is configured to (1) receive (e.g., from the masking and tagging unit 430 or the word replacement unit 440, or both) all processed text content corresponding to the same individual, where all processed text content includes privacy-sensitive personal information about the individual that is masked, replaced, hidden, or tagged, or a combination thereof, and (2) generate a corresponding text document by combining all of the processed text content into a text document.

[0065] In one embodiment, the text document generation unit 450 generates a corresponding text document for each of the P individuals, resulting in a total collection of P generated text documents. In one embodiment, the collection of P text documents is maintained in a database (e.g., on the storage unit 320). Alternatively, in one embodiment, the text document generation unit 450 generates corresponding text documents only for individuals that have multiple corresponding processed text content (i.e., only for individuals that are the subject of multiple voice recordings).

[0066] In one embodiment, if an individual is the subject of multiple audio recordings (i.e., all the processed textual content corresponding to the multiple audio recordings includes privacy-sensitive personal information about the individual), the text document generation unit 450 is configured to merge all the processed textual content corresponding to the multiple audio recordings into a corresponding text document based on timestamp vectors or category similarity. For example, in one embodiment, all the processed textual content is arranged in the text document in chronological order based on a timestamp vector indicating the timestamps of the multiple audio recordings. As another example, in one embodiment, all the processed textual content is arranged in the text document in category similarity order.

[0067] In one embodiment, the system 330 includes a content de-identification unit 460. For each of the P individuals, the content de-identification unit 460 is configured to (1) receive a corresponding text document (e.g., from the text document generation unit 450), where the text document includes all processed text content corresponding to the same individual (i.e., all of the processed text content includes privacy-sensitive personal information about the individual that is masked, replaced, hidden, or tagged, or a combination thereof), and (2) generate corresponding de-identified text content by applying content de-identification to the text document. The applied content de-identification anonymizes all processed text content included in the text document to a degree that preserves the usefulness of all processed text content and does not reveal privacy-sensitive personal information about the individual. All de-identified text content generated by the content de-identification unit 460 is suitable for disclosure to third parties for secondary use.

[0068] In one embodiment, the content de-identification unit 460 calculates, for each of the P individuals, k c k from P individuals, where P is the number of individuals that are equal to or smaller than P. c The method is configured to generate corresponding de-identified text content that protects the individual among other individuals of the person. These conditions provide data privacy assurances against potential re-identification (by a third party) of the original identity of the individual. If the corresponding de-identified text content is disclosed to or intercepted by a third party, the probability that the third party will successfully identify the individual (i.e., infer the identity of the individual) from the de-identified text content is 1 / k c Therefore, the likelihood that a third party (e.g., an attacker) can successfully re-identify an individual from de-identified text content is bounded by 1 / kc. c In one embodiment, k c is set by the data owner or de-identification expert (e.g., via I / O unit 353).c is the re-identification risk threshold, provided as input by the data owner or de-identification expert, used to enforce the desired / required level of privacy protection (i.e., the likelihood of re-identification).

[0069] In one embodiment, the content de-identification applied by the content de-identification unit 460 includes the following steps: First, the content de-identification unit 460 analyzes each of the P text documents and generates a union of terms / tokens that appear in the text documents and excludes recognized PII terms (i.e., each direct identifier and / or quasi-identifier recognized via the masking and tagging unit 430). For each term / token in the union, the content de-identification unit 460 maintains a term frequency list that includes a corresponding frequency indicating the number of times these terms / tokens appear (occur) across the P text documents.

[0070] Second, the content de-identification unit 460 selects one or more infrequent terms in the P text documents for pruning based on at least one blacklist / dictionary for direct identifiers (e.g., a list of names extracted from a publicly available dataset such as a voter registration list). In one embodiment, the content de-identification unit 460 utilizes the at least one blacklist / dictionary to determine a maximum frequency F associated with the recognized direct identifiers in the P text documents, where the maximum frequency F is selected as a threshold for use in selecting infrequent terms in the P text documents for pruning. For example, in one embodiment, the content de-identification unit 460 selects for pruning all terms / tokens in the union that have a corresponding frequency that does not exceed the threshold F, such that the remaining terms / tokens of the union that were not selected for pruning have a corresponding frequency that exceeds the threshold F.

[0071] For each infrequent term / token selected for pruning, the content de-identification unit 460 is configured to prune (ie, filter out) the infrequent term / token from the P text documents.

[0072] The content de-identification unit 460 selects unique terms and low-frequency terms (collectively, non-frequent words) occurring in the P text documents for pruning. The content de-identification unit 460 first assumes that each non-frequent word selected for pruning is a PII word. However, the non-frequent words selected for pruning may actually be non-PII words that do not need to be pruned (i.e., filtered out) from the P text documents. Third, to take into account the non-frequent words that are actually non-PII words, the content de-identification unit 460 is optionally configured to return one or more non-frequent words selected for pruning to the P text documents based on at least one whitelist of harmless terms. Each non-frequent word selected for pruning but included in at least one whitelist is recognized by the content de-identification unit 460 as a known non-PII word and returned to the P text documents. Examples of whitelists that the content disidentification unit 460 can utilize may include, but are not limited to, known whitelists for content disidentification, vocabulary databases (e.g., Wordnet).

[0073] Fourth, the content de-identification unit 460 is configured to extract each quasi-identifier recognized in the P text documents and generate a corresponding record of structured data (i.e., structured representation) based on the list of known quasi-identifiers. The list of known quasi-identifiers identifies one or more structured representations for holding one or more values ​​for one or more known types of quasi-identifiers. For example, if the list of known quasi-identifiers is defined as {date of birth, gender, 5-digit zip code}, the list identifies a first structured representation for holding values ​​of known type date of birth, a second structured representation for holding values ​​of known type gender, and a third structured representation for holding values ​​of known type 5-digit zip code. In one embodiment, the list of known quasi-identifiers is derived based on a publicly available dataset in the domain in which the text documents reside (i.e., related to). In another embodiment, the list of known quasi-identifiers is provided by a data owner or a de-identification expert (e.g., via the I / O unit 353).

[0074] Specifically, for each known type included in the list of known quasi-identifiers, the content de-identification unit 460 is configured to: (1) locate all text documents in the P text documents that contain at least one quasi-identifier tagged with one or more tags indicating the known type; and (2) generate, for each located text document, a corresponding record of structured data that holds the value of the known type. For example, if the known type is date of birth and the quasi-identifier is "November 2, 1980", the record created includes the following structured data: date of birth="11 / 2 / 1980". As another example, if the known type is gender and the quasi-identifier is "He", the record created includes the following structured data: gender="M". As yet another example, if the known type is zip code and the quasi-identifier is "12345", the record created includes the following structured data: zip code="12345".

[0075] In one embodiment, the content de-identification unit 460 hides each quasi-identifier in the P text documents that are tagged with one or more tags indicating a type of quasi-identifier that is not included in the list of known quasi-identifiers.

[0076] In one embodiment, the content de-identification unit 460 supports multiple anonymization algorithms. For each record of generated structured data, the content de-identification unit 460 selects an appropriate syntactic anonymization approach (i.e., algorithm) from among multiple anonymization algorithms and applies it to the record to anonymize at least one value held in the record, resulting in an anonymized structured data record holding the anonymized value. In one embodiment, for known types included in the list of quasi-identifiers, the content de-identification unit 460 is optionally configured to apply a micro-aggregation approach to all structured data records holding values ​​for the known type, resulting in an anonymized structured data record having a random value for the known type, where the random value is calculated across the micro aggregates. For each of the P text documents, the content de-identification unit 460 is configured to replace each quasi-identifier recognized in the text document with an anonymized / random value for the known type of the quasi-identifier, where the anonymized / random value is obtained from an anonymized structured data record corresponding to the text document.

[0077] Finally, for each of the P individuals, the content de-identification unit 460 is configured to remove from the corresponding text document each tag that is tagged with each quasi-identifier recognized in the text document (e.g., remove start and end tags), resulting in corresponding de-identified text content.

[0078] In one embodiment, system 330 includes a mapping unit 470. For each of the P individuals, mapping unit 470 is configured to (1) receive corresponding de-identified text content (e.g., from content de-identification unit 460) and (2) map, based on the R audio recordings, one or more segments of the identified text content to both one or more speakers of the S speakers and one or more sounds uttered by the one or more speakers, where the one or more sounds contain privacy-sensitive personal information about the individual (i.e., the one or more sounds are recorded in one or more audio recordings to which the individual is subject).

[0079] In one embodiment, the system 330 includes a synthetic speaker identity generation unit 480. For each of the S speakers, the synthetic speaker identity generation unit 480 is configured to apply speaker de-identification to each audio recording (of the R audio recordings) corresponding to the speaker. In one embodiment, for each of the S speakers, the speaker de-identification applied by the synthetic speaker identity generation unit 480 includes (1) generating a corresponding synthetic speaker identity, and (2) for each audio recording (of the R audio recordings) corresponding to the speaker, synthesizing a new audio waveform based on the synthetic speaker identity to generate de-identified text content that maps to both the speaker and the audio generated by the speaker (and recorded in the audio recording). The new audio waveform sounds very different from the audio waveform of each audio recording corresponding to the speaker. In one embodiment, if the de-identified text content includes one or more hidden values, each hidden value is pronounced / generated or converted into a beep in the new audio waveform. A beep can notify the viewer that the new audio waveform has one or more missing words.

[0080] In one embodiment, for each of the S speakers, the synthetic speaker identity generation unit 480 is configured to generate a corresponding synthetic speaker identity that satisfies the following conditions: (1) the synthetic speaker identity is generated for k of the S speakers; sprotect the speaker among k other speakers, where s ≦S, and (2) the synthetic speaker identity is far from the speaker's original speaker identity (i.e., the new speech waveforms synthesized with the synthetic speaker identity sound very different from the speech waveforms of each audio recording corresponding to the speaker). These conditions ensure data privacy against potential re-identification (by a third party) of the speaker to its original speaker identity. If the de-identified audio recording containing the new speech waveforms synthesized with the synthetic speaker identity is released to or intercepted by a third party, the probability that the third party will successfully identify the speaker from the de-identified audio recording (i.e., infer the speaker's original speaker identity) is 1 / k s Therefore, the likelihood that a third party (attacker) can successfully re-identify a speaker from a de-identified audio recording is bounded by 1 / k s In one embodiment, k s is set by the data owner or de-identification expert (e.g., via I / O unit 353). s is the re-identification risk threshold provided as input by the data owner or de-identification expert and is used to enforce the desired / required level of privacy protection (i.e., the likelihood of re-identification).

[0081] In one embodiment, the synthetic speaker identity generation unit 480 clusters the S speakers into clusters (i.e., groups) by clustering each feature vector corresponding to each of the S speakers (extracted via the feature extraction unit 410) based on a vector similarity measure / metric (e.g., Euclidean distance or cosine similarity measure), where each resulting cluster contains at least k features from the S speakers. sThe S speakers include feature vectors corresponding to similar speakers of the S speakers. For each of the S speakers, the synthetic speaker identity generation unit 480 is configured to generate a corresponding synthetic speaker identity by (1) selecting a cluster (i.e., a target cluster) that is as far away as possible from other clusters that include feature vectors corresponding to the speaker (i.e., the speech waveforms of all feature vectors included in the selected cluster sound significantly different from the speech waveforms of each speech recording corresponding to the speaker), (2) applying an aggregation function across the speaker identity features (i.e., speech waveforms) of all feature vectors included in the selected cluster, and (3) generating a synthetic speaker identity based on the resulting aggregated speaker identity features. The feature vectors included in the selected cluster are different from the feature vectors corresponding to the speaker to an extent that satisfies the desired / required level of privacy protection.

[0082] In one embodiment, the system 330 includes an output unit 490. For each of the R original voice recordings, the output unit 490 is configured to release a corresponding de-identified voice recording to a third party for secondary use, where the de-identified voice recording includes a synthesized voice waveform that releases de-identified text content mapped to both the speaker corresponding to the original voice recording and the voice produced by the speaker and recorded in the original voice recording. The output unit 490 releases only the de-identified information, i.e., the de-identified voice recording and the de-identified text content. The output unit 490 does not release the original voice recording or the original text content recognized and extracted from the original voice recording.

[0083] In an example application scenario, assume R=10, S=6, and P=20. The R audio recordings include a total of 10 audio recordings, i.e., audio recording 1, audio recording 2, ..., audio recording 10. The S speakers include a total of 6 speakers, i.e., speaker 1, speaker 2, ..., speaker 6. The P individuals include a total of 20 individuals, i.e., individual 1, individual 2, ..., individual 20. In one embodiment, for each of the 6 speakers, the feature extraction unit 410 generates a corresponding feature vector based on all language and speaker identity features extracted from all audio recordings corresponding to the speaker (of the 10 audio recordings) and persists the feature vector in a database (e.g., the storage unit 320).

[0084] Assume that the system 330 receives a request via the input unit 400 to apply speaker de-identification and content de-identification to three given audio recordings out of 10 audio recordings. Assume that the three given audio recordings correspond to two specific speakers out of six speakers, and that the text content recognized and extracted from the three audio recordings contains privacy-sensitive personal information about three specific individuals out of 20 individuals.

[0085] Table 1 below provides examples of transcripts recognized and extracted by the textual content extraction unit 420 from three given audio recordings. [Table 1]

[0086] As shown in Table 1, three given audio recordings include: (1) audio recording 1 corresponding to speaker 1, who is a clinician, Carmen Dudley, MD, where a transcription recognized and extracted from audio recording 1 includes privacy-sensitive personal information (e.g., PHI or other PII) about a patient, individual, Jane Alan; (2) audio recording 2 corresponding to speaker 1 (Carmen Dudley, MD), where a transcription recognized and extracted from audio recording 2 includes privacy-sensitive personal information about a patient, individual, Mr. Ted Borret; and (3) audio recording 3 corresponding to speaker 2, who is a clinician, Veep Bob, MD, where a transcription recognized and extracted from audio recording 3 includes privacy-sensitive personal information about a patient, individual, Cathie Trian.

[0087] 5 illustrates an exemplary annotation that provides a mapping between the speech waveform of a speech recording and the words spoken by the speaker to which the speech recording corresponds, according to one embodiment of the present invention. Specifically, the annotation provides a mapping between the speech waveform of speech recording 1 and a transcription recognized and extracted from speech recording 1 (see Table 1). The annotation is generated by text content extraction unit 420.

[0088] Table 2 below provides example direct identifiers and quasi-identifiers recognized in the transcripts of Table 1 by the masking and tagging unit 430. For reference, each direct identifier recognized by the masking and tagging unit 430 is shown underlined and in bold, and each quasi-identifier recognized by the masking and tagging unit 430 is shown in bold with a tag indicating the type of quasi-identifier. [Table 2]

[0089] As shown in Table 2, names in the transcript are recognized as direct identifiers (e.g., the names "Jane Alan," "Becket," "Ted Borret," "Cathie Trian," and "Boris" are underlined and bolded). As further shown in Table 2, one or more words (or terms) in the transcript that indicate a particular age, a particular gender, a particular date, a particular diagnosis, or a particular procedure are recognized as quasi-identifiers (e.g., the date "August 14, 2013" in the transcript recognized and extracted from audio recording 3 has a start tag <date> and the closing tag< / date> will be tagged).

[0090] As further shown in Table 2, the one or more NLP annotators applied to the transcript may not recognize all direct identifiers and / or all quasi-identifiers in the text. For example, the direct identifiers "ID43729" and "ID53265" and quasi-identifiers such as "Crohn's Disease" are not recognized by the NLP annotators. The system 330 is configured to conceal direct identifiers and quasi-identifiers (e.g., IDs) in the transcript that are not recognized by the NLP annotators. For example, the content de-identification unit 460 selects unique terms and low frequency words (e.g., IDs) occurring in the transcript for pruning (see Table 4 below).

[0091] Table 3 below provides example masked values ​​that the masking and tagging unit 430 uses to replace direct identifiers recognized in the transcript of Table 2, and also provides example synonyms that the word replacement unit 440 uses to replace some of the words in the transcript of Table 2. For reference, each masked value is shown underlined and in bold, and each synonym is shown in bold and italics. [Table 3]

[0092] As shown in Table 3, the direct identifiers "Jane Alan," "Becket," "Ted Borret," "Cathie Trian," and "Boris" recognized in the transcript of Table 2 are replaced with the masked values ​​"Mary Quinn," "Capeman," "Albert Somaya," "Ted Burner," and "Rott," respectively. As further shown in Table 3, the occurrences of "presented," "instructed," "experienced," and "elevated" in the transcript of Table 2 are replaced with the synonyms "came," "asked," "had," and "Increased," respectively.

[0093] Table 4 below provides an example union of terms / tokens produced by content dis-identification unit 460. For reference, each infrequent word / token selected for pruning by content dis-identification unit 460 is shown with a strikethrough. [Table 4]

[0094] Table 5 below provides example infrequent words / tokens that are selected for pruning but are subsequently recognized as harmless and reverted by content de-identification unit 460. For reference, each infrequent word / token that is subsequently recognized as harmless and reverted by content de-identification unit 460 is shown in bold. [Table 5]

[0095] As shown in Table 5, the non-frequent words selected for pruning (see Table 4), namely “care”, “department”, “evaluate”, “found”, “had”, “found”, “give”, “History”, “no”, “of”, “prescription”, “come”, “went”, “were”, “who”, and “Instructed”, are subsequently recognized as harmless and reinstated.

[0096] A PLQ generally denotes a list of known quasi-identifiers. In one example, assume that the PLQ is represented according to list (1) provided below:

number

[0097] Table 6 below provides example quasi-identifiers extracted by the content de-identification unit from the transcript of Table 2. For reference, each quasi-identifier that is obscured in the transcript by content de-identification unit 460 is shown with a strikethrough. [Table 6]

[0098] Each extracted quasi-identifier is tagged with a tag indicating a known type contained in the PLQ, as shown in Table 6. The quasi-identifier "laparoscopic partial nephrectomy" tagged with a tag indicating procedure is suppressed in the transcription, since procedure is not a known type contained in the PLQ.

[0099] Table 7 below provides an example structured data record created by content de-identification unit 460 according to PLQ element 1. [Table 7]

[0100] Each record created has a corresponding ID indicating the transcript from which the quasi-identifiers were extracted, where the record maintains its original value, as shown in Table 7. The quasi-identifiers are tagged with a tag indicating their known type, age or gender.

[0101] Table 8 below provides an example structured data record created by content de-identification unit 460 according to PLQ element 2. [Table 8]

[0102] Each record created has a corresponding ID indicating the transcript from which the quasi-identifier was extracted, where the record maintains its original value, as shown in Table 8. The quasi-identifier is tagged with a tag indicating the known type date.

[0103] Table 9 below provides an example structured data record created by content de-identification unit 460 according to PLQ element 3. [Table 9]

[0104] As shown in Table 9, each record created has a corresponding ID that indicates the transcript from which the quasi-identifiers were extracted, where the record maintains its original value. The quasi-identifiers are tagged with tags indicating the types, diagnoses, for which they are known.

[0105] k c Assume that ρ = 2. In one embodiment, for each of the individuals (i.e., patients) Jane Alan, Ted Borret, and Cathie Trian, the content de-identification unit 460 is configured to generate corresponding de-identified text content such that the likelihood of a third party (attacker) successfully re-identifying an individual from the de-identified text content is limited to ½.

[0106] Table 10 below provides an example anonymized structured data record resulting from content de-identification unit 460 applying relationship 2 anonymization to the record in Table 7 to anonymize the original values ​​in the record to generalized values. For reference, the original values ​​obscured by content de-identification unit 460 are indicated with an asterisk (*). [Table 10]

[0107] Table 11 below provides an example anonymized structured data record resulting from content de-identification unit 460 applying Sequence 2 anonymization (order preserving) to the record in Table 8 to anonymize the original values ​​in the record into generalized values. [Table 11]

[0108] Table 12 below provides an example anonymized structured data record resulting from content de-identification unit 460 applying Set 2 anonymization to the record in Table 9 to anonymize the original values ​​in the record to generalized values. For reference, the original values ​​obscured by content de-identification unit 460 are indicated with an asterisk (*). [Table 12]

[0109] Table 13 below provides example anonymized structured data records that result when content de-identification unit 460 applies the micro-aggregation approach to the records in Table 10 to obtain random values ​​that are calculated across the micro-aggregates. For reference, each random value is shown in parentheses. [Table 13]

[0110] Table 14 below provides example anonymized structured data records that result when content de-identification unit 460 applies the micro-aggregation approach to the records in Table 11 to obtain random values ​​that are calculated across the micro-aggregates. For reference, each random value is shown in parentheses. [Table 14]

[0111] Table 15 below provides example anonymized structured data records that result when content de-identification unit 460 applies the micro-aggregation approach to the records in Table 12 to obtain random values ​​that are calculated across the micro-aggregates. For reference, each random value is shown in parentheses. [Table 15]

[0112] As shown in Tables 13-15, each random value for each record is a likely replacement value that may be used to replace the corresponding quasi-identifier in the transcript from which it was extracted. Each random value is randomly selected from the generalized values ​​generated as a result of applying the anonymization algorithm. For categorical values, the random value may be randomly selected from the original set of values, or may be randomly selected from the subtree rooted at the node with the generalized value (e.g., "head-related medical problem").

[0113] Table 16 below provides an example de-identified transcript obtained by content de-identification unit 460 replacing some quasi-identifiers recognized in the transcript of Table 2 with replacement values ​​from Tables 13-15 and removing the quasi-identifier tags. For reference, the original values ​​hidden by content de-identification unit 460 are indicated with an asterisk (*). [Table 16]

[0114] As shown in Table 16, the de-identified transcripts of Audio Recording 1 and Audio Recording 2 are mapped to Speaker 1 (Clinician Carmen Dudley, MD) and the de-identified transcript of Audio Recording 3 is mapped to Speaker 2 (Clinician Veep Bob, MD). As shown in Table 16, the de-identified transcripts do not reveal privacy-sensitive personal information about the individuals (i.e., patients), Jane Alan, Ted Borret, and Cathie Trian.

[0115] 6 shows an exemplary graphical representation of speaker feature vectors in a two-dimensional space according to an embodiment of the present invention. The synthetic speaker identity generation unit 480 retrieves each feature vector corresponding to each of the six speakers from the database and clusters the resulting feature vectors based on a vector similarity measure / metric (e.g., Euclidean distance or cosine similarity measure). sAssume that s = 3. The synthetic speaker identity generation unit 480 clusters all the resulting feature vectors into two separate clusters, cluster X and cluster Y, where each cluster contains feature vectors corresponding to three similar speakers. As shown in FIG. 6, cluster X contains feature vector S-vector-1 corresponding to speaker 1 (clinician Carmen Dudley, MD), feature vector S-vector-3 corresponding to speaker 3, and feature vector S-vector-4 corresponding to speaker 4. Cluster Y contains feature vector S-vector-2 corresponding to speaker 2 (clinician Veep Bob, MD), feature vector S-vector-5 corresponding to speaker 5, and feature vector S-vector-6 corresponding to speaker 6.

[0116] For each cluster, the synthetic speaker identity generation unit 480 applies an aggregation function across the speaker identity features of all feature vectors included in the cluster, and generates a synthetic speaker identity corresponding to the cluster based on the resulting aggregated speaker identity features. In one embodiment, the synthetic speaker identity generation unit 480 generates a synthetic vector V corresponding to cluster X by calculating the minimum, maximum, and average of all feature vectors included in cluster X. X Then, we calculate the minimum, maximum and average of all the feature vectors in cluster Y to obtain a composite vector V Y In another embodiment, the synthetic speaker identity generation unit 480 constructs a synthetic vector V corresponding to cluster X by randomly selecting values ​​for each speaker identity feature from all feature vectors contained in cluster X. X Then, we randomly select values ​​for each speaker identity feature from all feature vectors in cluster Y to create a composite vector V Y Construct the composite vector V X and V Y denote the synthetic speaker identities corresponding to cluster X and cluster Y, respectively.

[0117] Since feature vector S-vector-1 corresponding to speaker 1 (clinician Carmen Dudley, MD) is included in cluster X, the synthetic speaker identity generation unit 480 selects cluster Y as the cluster as far away as possible from cluster X and generates a synthetic vector V for generating de-identified transcripts of audio recording 1 and audio recording 2 (see Table 16 above). Y Since feature vector S-vector-2 corresponding to speaker 2 (clinician Veep Bob, MD) is included in cluster Y, the synthetic speaker identity generation unit 480 selects cluster X as the cluster as far away as possible from cluster Y, and synthesizes synthetic vector V to generate a de-identified transcript of speech recording 3 (see Table 16 above). X In summary, the synthetic speaker identity generation unit 480 generates two different synthetic speaker identities (i.e., synthetic vectors V X , V Y ), which are used to synthesize new speech waveforms to generate de-identified transcripts of the three audio recordings (i.e., audio recording 1, audio recording 2, and audio recording 3). In one embodiment, for each of two different speakers (i.e., clinician Carmen Dudley, MD, and Veep Bob, MD), synthetic speaker identity generation unit 480 is configured to generate a corresponding synthetic speaker identity such that the likelihood of a third party (e.g., an attacker) successfully re-identifying a speaker using a new speech waveform synthesized with the synthetic speaker identity is bounded by 1 / 3.

[0118] Table 17 below shows the composite vector V X and V Y 4 provides an example de-identified speech recording that includes a new speech waveform synthesized by synthetic speaker identity generation unit 480 using: For reference, the de-identified transcript produced by the new speech waveform is shown in quotation marks. [Table 17]

[0119] Output unit 490 releases the de-identified audio recordings of Table 17 to third parties for secondary use. As shown in Table 17, the de-identified audio recordings do not reveal the identities of Speaker 1 and Speaker 2 (i.e., clinicians Carmen Dudley, MD and Veep Bob, MD), and the de-identified transcripts do not reveal privacy-sensitive personal information about individuals (i.e., patients) Jane Alan, Ted Borret, and Cathie Trian.

[0120] FIG. 7 is a flow chart of an example process 700 for speaker identity and content de-identification, according to an embodiment of the present invention. Processing block 701 includes receiving an input indicating at least one level of privacy protection to enforce via speaker identity and content de-identification. Processing block 702 includes extracting features corresponding to a first speaker from a first speech emitted by the first speaker and recorded in a first speech recording. Processing block 703 includes recognizing and extracting text content from the first speech. Processing block 704 includes analyzing the text content to recognize privacy-sensitive personal information corresponding to a first individual. Processing block 705 includes generating de-identified text content by performing utility-preserving content de-identification on the text content to anonymize the privacy-sensitive personal information to a degree that satisfies at least one level of privacy protection, where the de-identified text content conceals the personal identity of the first individual. Processing block 706 includes mapping the de-identified text content to the first speaker. Processing block 707 includes generating a synthetic speaker identity corresponding to the first speaker based on other features corresponding to at least one other speaker, where the other features corresponding to the at least one other speaker are dissimilar to the features corresponding to the first speaker to a degree that satisfies at least one level of privacy protection. Processing block 708 includes synthesizing a new speech waveform based on the synthetic speaker identity to produce the de-identified text content, where the new speech waveform differs from the speech waveform of the first voice, where the new speech waveform conceals the personal identity of the first speaker.

[0121] In one embodiment, process blocks 701 - 708 are performed by one or more components of system 330 .

[0122] 8 is a high-level block diagram illustrating an information processing system 800 useful for implementing embodiments of the present invention. The computer system includes one or more processors, such as processor 802. Processor 802 is connected to a communications infrastructure 804 (e.g., a communications bus, crossbar, or network).

[0123] The computer system may also include a display interface 806 that transfers graphics, text, and other data from the voice communications infrastructure 804 (or from a frame buffer, not shown) for display on a display unit 808. In one embodiment, the computer system also includes a main memory 810, preferably random access memory (RAM), and a secondary memory 812. In one embodiment, the secondary memory 812 includes, for example, a hard disk drive 814 and / or a removable storage drive 816, for example, a floppy disk drive, magnetic tape drive, or optical disk drive. The removable storage drive 816 reads from and / or writes to a removable storage unit 818 in a manner well known to those skilled in the art. The removable storage unit 818 may represent, for example, a floppy disk, compact disk, magnetic tape, or optical disk that is read from and written to by the removable storage drive 816. As will be appreciated, the removable storage unit 818 includes a computer readable medium having stored thereon computer software and / or data.

[0124] In alternative embodiments, the secondary memory 812 includes other similar means for allowing computer programs or other instructions to be loaded into the computer system. Such means include, for example, a removable storage unit 820 and an interface 822. Examples of such means include program packages and package interfaces (such as found in video game devices), removable memory chips (e.g., EPROM or PROM) and associated sockets, and other removable storage units 820 and interfaces 822 that allow software and data to be transferred from the removable storage unit 820 to the computer system.

[0125] In one embodiment, the computer system also includes a communications interface 824. The communications interface 824 allows software and data to be transferred between the computer system and external devices. In one embodiment, examples of the communications interface 824 include a modem, a network interface (e.g., an Ethernet card), a communications port, or a PCMCIA slot and card. In one embodiment, the software and data transferred through the communications interface 824 are in the form of signals, which may be, for example, electronic, electromagnetic, optical, or other signals capable of being received by the communications interface 824. These signals are provided to the communications interface 824 via a communications path (or channel) 826. In one embodiment, the communications path 826 carries signals and is implemented using wire or cable, fiber optics, a phone line, a cellular phone link, an RF link, or other communications channels, or a combination thereof.

[0126] The embodiments of the invention may be a system, method, or computer program product, or combination thereof, at any possible level of technical detail of integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to perform aspects of the embodiments of the invention.

[0127] A computer readable storage medium may be a tangible device that holds and stores instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. Illustrative enumeration of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, punch cards or mechanically encoded devices such as ridge structures in grooves having instructions recorded thereon, and any suitable combination of the above. Computer-readable storage media, as used herein, is not to be construed as a transitory signal per se, such as an electric wave, a freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0128] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to each computing / processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions to a computer readable storage medium within the respective computing / processing device for storage.

[0129] The computer readable program instructions for carrying out the operations of embodiments of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object oriented languages ​​such as Smalltalk®, C++ or the like, procedural languages ​​such as the C programming language or similar programming languages. The computer readable program instructions may be executed as a stand-alone software package, entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, the electrical circuitry may execute computer readable program instructions by utilizing state information of the computer readable program instructions to individualize the electrical circuitry, which may include, for example, programmable logic circuitry, a field programmable gate array (FPGA), or a programmable logic array (PLA), to perform aspects of embodiments of the present invention.

[0130] Aspects of embodiments of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, may be implemented by computer readable program instructions.

[0131] These computer readable program instructions, when provided to a computer processor or other programmable data processing apparatus, cause the instructions to execute on the computer processor or other programmable data processing apparatus to produce a machine-generated means for implementing the functions / acts identified in the block or blocks of the flowchart diagrams and / or blocks. These computer readable program instructions may also be stored on a computer readable storage medium capable of directing a computer, programmable data processing apparatus or other device or combination thereof to function in a particular manner, such that a computer readable storage medium having instructions stored thereon includes an article of manufacture including instructions that implement aspects of the functions / acts identified in the block or blocks of the flowchart diagrams and / or blocks.

[0132] The computer readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device and cause the computer, other programmable data processing apparatus, or other device to perform a series of operational steps to generate a computer implemented process such that the instructions executing on the computer, other programmable data processing apparatus, or other device implement aspects of the functions / actions identified in a block or blocks of the flowchart and / or blocks.

[0133] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment or part of instructions, including one or more executable instructions for implementing a particular logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order depicted in the drawings. For example, two blocks shown in succession may in fact be realized as a single step, may be executed simultaneously, may be executed substantially simultaneously in a partially or fully overlapping manner, or the blocks may be executed in reverse order depending on the functionality involved. It should be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations may be implemented by a special purpose hardware-based system that performs a particular function or action or implements a combination of special purpose hardware and computer instructions.

[0134] From the above description, it can be seen that the embodiments of the present invention provide a system, a computer program product, and a method for implementing the embodiments of the present invention. The embodiments of the present invention further provide a non-transitory computer usable storage medium for implementing the embodiments of the present invention. The non-transitory computer usable storage medium has a computer readable program, which when executed on a computer, causes the computer to perform the steps of the embodiments of the present invention described herein. References to elements in the singular form in the claims are not intended to mean "only one" unless so expressly stated, but rather "one or more." All structural and functional equivalents to the elements of the above exemplary embodiments now known or that become known in the future to those skilled in the art are intended to be encompassed within the scope of the claims. No claim element herein should be construed under the provisions of 35 U.S.C. 112, paragraph 6, unless the element is expressly recited using the phrase "means for" or "step for."

[0135] The terms used herein are for the purpose of describing particular embodiments of the invention only and are not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise. Furthermore, it is to be understood that the terms "comprise" and / or "comprising", when used in this specification, specify the presence of stated features, integers, steps, operations, elements or components, or combinations thereof, and do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof, or combinations thereof.

[0136] In the following claims, equivalents of corresponding structures, materials, acts, and all means or step-plus-function elements are intended to include any structure, material, or act for performing a function in combination with other claimed elements as expressly claimed.

[0137] The description of various embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications, or technical improvements to the technology found in the market, or to allow other skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A method for speaker identity and content de-identification under data privacy assurance, comprising: at least one processor comprising: receiving an input indicating at least one level of privacy protection that requires enforcement of said speaker identity and content de-identification; extracting features corresponding to a first speaker from a first speech uttered by said first speaker and recorded in a first speech recording; Recognizing and extracting text content from the first speech; analyzing the text content to recognize privacy-sensitive personal information corresponding to a first individual; generating de-identified text content by performing utility-preserving content de-identification on the text content to anonymize the privacy-sensitive personal information to a degree that satisfies at least one level of privacy protection, the de-identified text content concealing a personal identity of the first individual; mapping the de-identified text content to the first speaker; generating a synthetic speaker identity corresponding to the first speaker based on other features corresponding to at least one other speaker, the other features corresponding to the at least one other speaker being dissimilar to the features corresponding to the first speaker to a degree sufficient to satisfy at least one level of privacy protection; synthesizing a new speech waveform to produce the de-identified text content based on the synthesized speaker identity, the new speech waveform being different from the speech waveform of the first voice, the new speech waveform masking the personal identity of the first speaker; A method for performing.

2. The method of claim 1, further comprising the step of processing the text content by the at least one processor replacing at least one word in the text content with at least one replacement value, the at least one word including a portion of the privacy-sensitive personal information.

3. 3. The method of claim 1 or 2, wherein the other features corresponding to the at least one other speaker are extracted from at least one other speech uttered by the at least one other speaker and recorded in at least one other speech recording.

4. The method includes the at least one processor performing the steps of receiving a plurality of voice recordings including the first voice recording and the at least one other voice recording; 4. The method of claim 3, wherein a plurality of voices including the first voice and the at least one other voice are recorded in the plurality of voice recordings, each of the plurality of voices being uttered by a speaker of a plurality of speakers, the plurality of speakers including the first speaker and the at least one other speaker, and text content of each of the plurality of voices includes privacy-sensitive personal information corresponding to at least one of a plurality of individuals, the plurality of individuals including the first individual.

5. The at least one processor, for each of the plurality of speech recordings, extracting features from speech recorded in the speech recording; generating, for each of the plurality of speakers, a corresponding feature vector based on features extracted from at least one of the plurality of sounds produced by the speaker; The method of claim 4 , further comprising:

6. The at least one processor, For each of the plurality of voice recordings, recognizing and extracting text content from speech recorded within the speech recording; analyzing the textual content by applying at least one natural language processing (NLP) annotator to the textual content to recognize privacy-sensitive personal information in the textual content corresponding to at least one of the plurality of individuals; The method of claim 5 , further comprising:

7. The method of claim 6 , wherein each privacy-sensitive personal information corresponding to each individual includes at least one of a direct identifier or a quasi-identifier.

8. The at least one processor, For each of the plurality of voice recordings, processing the textual content of the speech recorded in the speech recording by masking each direct identifier recognized in the textual content, tagging each quasi-identifier recognized in the textual content, and replacing at least one word in the textual content with at least one synonym.

8. The method of claim 7, further comprising: wherein each word replaced in the text content is one of a quasi-identifier or a word that is neither a direct identifier nor a quasi-identifier.

9. The at least one processor, For each of the plurality of individuals, combining the processed text content of at least one of the plurality of audios, including corresponding privacy-sensitive personal information, into a text document; generating corresponding de-identified text content by performing the utility-preserving content de-identification on the text document to anonymize the corresponding privacy-sensitive personal information to a degree that satisfies at least one level of privacy protection, wherein the corresponding de-identified text content does not reveal the corresponding privacy-sensitive personal information; mapping one or more segments of the corresponding de-identified textual content to at least one of the plurality of speakers, where at least one textual content of the plurality of voices produced by the at least one speaker includes the corresponding privacy-sensitive personal information; The method of claim 8 , further comprising:

10. The at least one processor, For each of the plurality of individuals, generating a corresponding synthetic speaker identity based on at least one feature vector corresponding to at least one other speaker of the plurality of speakers, the at least one feature vector corresponding to the at least one other speaker being dissimilar to the feature vector corresponding to the speaker to a degree that satisfies at least one level of the privacy protection; for each of the plurality of speeches produced by the speaker and recorded in one of the plurality of speech recordings, synthesizing a corresponding new speech waveform to produce a segment of de-identified text content mapped to the speaker based on the corresponding synthesized speaker identity, the corresponding new speech waveform being different from the speech waveform of the speech, such that the corresponding new speech waveform does not reveal the personal identity of the speaker; The method of claim 9 , further comprising:

11. The at least one processor, For each of the plurality of speakers, Releasing the corresponding de-identified voice recording to a third party, the corresponding de-identified voice recording including the corresponding new voice waveform synthesized based on the corresponding synthesized speaker identity. The method of claim 10 further comprising:

12. 1. A system for speaker identity and content de-identification under privacy guarantees, comprising: At least one processor; a non-transitory processor-readable memory device for storing instructions; the instructions, when executed by the at least one processor, cause the at least one processor to perform an operation, the operation comprising: receiving an input indicating at least one level of privacy protection requiring that the speaker identity and content de-identification be enforced; extracting features corresponding to a first speaker from a first speech uttered by said first speaker and recorded in a first speech recording; Recognizing and extracting text content from the first speech; analyzing the text content to recognize privacy-sensitive personal information corresponding to a first individual; generating de-identified text content by performing utility-preserving content de-identification on the text content to anonymize the privacy-sensitive personal information to a degree that satisfies at least one level of privacy protection, the de-identified text content concealing a personal identity of the first individual; mapping the de-identified text content to the first speaker; generating a synthetic speaker identity corresponding to the first speaker based on other features corresponding to at least one other speaker, the other features corresponding to the at least one other speaker being dissimilar to the features corresponding to the first speaker to a degree that satisfies at least one level of the privacy protection; synthesizing a new speech waveform for producing the de-identified text content based on the synthesized speaker identity, the new speech waveform being different from a speech waveform of the first voice, the new speech waveform concealing the personal identity of the first speaker. and a system.

13. The operation includes:

13. The system of claim 12, further comprising: receiving a plurality of voice recordings including the first voice recording and the at least one other voice recording, wherein a plurality of voices including the first voice and the at least one other voice are recorded in the plurality of voice recordings, each of the plurality of voices being uttered by a speaker of a plurality of speakers, the plurality of speakers including the first speaker and the at least one other speaker, and wherein text content of each of the plurality of voices includes privacy-sensitive personal information corresponding to at least one of a plurality of individuals, the plurality of individuals including the first individual.

14. The operation includes: for each of the plurality of speech recordings, extracting features from speech recorded in the speech recording; generating, for each of the plurality of speakers, a corresponding feature vector based on features extracted from at least one of the plurality of sounds produced by the speaker; The system of claim 13 , comprising:

15. The above operation is For each of the plurality of voice recordings, Recognizing and extracting text content from speech recorded within the speech recording; analyzing the textual content by applying at least one natural language processing (NLP) annotator to the textual content to recognize privacy-sensitive personal information in the textual content corresponding to at least one of the plurality of individuals; The system of claim 14 , comprising:

16. 20. The system of claim 15, wherein each privacy-sensitive personal information corresponding to each individual includes at least one of a direct identifier or a quasi-identifier.

17. The operation includes: For each of the plurality of voice recordings, processing the textual content of the speech recorded in the speech recording by masking each direct identifier recognized in the textual content, tagging each quasi-identifier recognized in the textual content, and replacing at least one word in the textual content with at least one synonym.

17. The system of claim 16, wherein each word replaced in the text content is one of a quasi-identifier or a word that is neither a direct identifier nor a quasi-identifier.

18. The operation includes: For each of the plurality of individuals, combining the processed text content of at least one of the plurality of audios, including corresponding privacy-sensitive personal information, into a text document; generating corresponding de-identified text content by performing the utility-preserving content de-identification on the text document to anonymize the corresponding privacy-sensitive personal information to a degree that satisfies at least one level of the privacy protection, wherein the corresponding de-identified text content does not reveal the corresponding privacy-sensitive personal information; Mapping one or more segments of the corresponding de-identified textual content to at least one of the plurality of speakers, where at least one textual content of the plurality of voices produced by the at least one speaker includes the corresponding privacy-sensitive personal information; The system of claim 17 , comprising:

19. For each of the plurality of individuals, generating a corresponding synthetic speaker identity based on at least one feature vector corresponding to at least one other speaker of the plurality of speakers, wherein the at least one feature vector corresponding to the at least one other speaker is dissimilar to the feature vector corresponding to the speaker to a degree that satisfies at least one level of privacy protection; synthesizing a corresponding new speech waveform for each of the plurality of speeches produced by the speaker and recorded in one of the plurality of speech recordings based on the corresponding synthesized speaker identity to produce a segment of de-identified text content mapped to the speaker, the corresponding new speech waveform being different from a speech waveform of the speech, such that the corresponding new speech waveform does not reveal the personal identity of the speaker. The system of claim 18 , comprising:

20. 1. A computer program for speaker identity and content de-identification under privacy guarantees, the computer program comprising: receiving an input indicating at least one level of privacy protection requiring that the speaker identity and content de-identification be enforced; extracting features corresponding to a first speaker from a first speech uttered by said first speaker and recorded in a first speech recording; Recognizing and extracting text content from the first speech; analyzing the text content to recognize privacy-sensitive personal information corresponding to a first individual; generating de-identified text content by performing utility-preserving content de-identification on the text content to anonymize the privacy-sensitive personal information to a degree that satisfies at least one level of privacy protection, the de-identified text content concealing a personal identity of the first individual; mapping the de-identified text content to the first speaker; generating a synthetic speaker identity corresponding to the first speaker based on other features corresponding to at least one other speaker, the other features corresponding to the at least one other speaker being dissimilar to the features corresponding to the first speaker to a degree that satisfies at least one level of the privacy protection; synthesizing a new speech waveform to produce the de-identified text content based on the synthesized speaker identity, the new speech waveform being different from the speech waveform of the first voice, the new speech waveform masking the personal identity of the first speaker; A computer program for performing the above.

Images