Participant-tuned filtering using deep neural network dynamic spectral masking for speech separation and security in noisy environments

Abstract

A method is provided for providing separation and amplification of speech between selected participants. A plurality of spectral masks are received, each spectral mask corresponding to a respective participant in a group of selected participants included in the conversation. A composite spectral mask is generated by additively superposing the plurality of spectral masks. The composite spectral mask is applied to audio captured by a microphone to remove audio that does not match the composite spectral mask and to amplify remaining audio that matches the composite spectral mask.

Description

[Technical field]

[0001] The present disclosure relates generally to spectral masking, and more particularly to conversation participant adjusted filtering using deep neural network dynamic spectral masking for participant conversation separation and security in noisy environments. [Background technology]

[0002] In telecommunications, a spectral mask is a mathematically defined filter that is applied to audio. The purpose of a spectral mask is generally to attenuate audio that does not match the spectral mask. Such attenuation of unwanted audio is usually achieved by only allowing audio that matches the spectral mask to pass. Summary of the Invention

[0003] According to one exemplary embodiment, a method is provided for isolating and amplifying a conversation between selected participants. A plurality of spectral masks are received. Each spectral mask in the plurality of spectral masks corresponds to a respective participant of the group of selected participants included in the conversation. An additive superposition of the plurality of spectral masks generates a composite spectral mask. The composite spectral mask is applied to audio captured by a microphone to remove audio that does not match the composite spectral mask and amplify remaining audio that matches the composite spectral mask. According to another exemplary embodiment, a mobile device and computer program product are provided for isolating and amplifying a conversation between selected participants.

[0004] Various exemplary embodiments also transmit the amplified remaining audio that matches the composite spectral mask to audio output devices that correspond to the participants in the conversation, the audio output devices being located proximate to the ears of the participants in the conversation.

[0005] Further, various exemplary embodiments send voice samples of participants of the conversation to a deep neural network server in a cloud environment to generate a spectral mask personalized for the participant, and receive the spectral mask personalized for the participant from the deep neural network server. Various exemplary embodiments combine the spectral mask personalized for the participant with a plurality of spectral masks corresponding to the selected group of participants included in the conversation to form a composite spectral mask, use the composite spectral mask to filter the incoming audio signal to allow only the conversation between the selected group of participants and the participant to remain in the audio signal, and transmit the audio signal including only the conversation between the selected group of participants and the participant to the audio output device. Further, various exemplary embodiments share the spectral mask personalized for the participant and the plurality of spectral masks corresponding to the selected group of participants between the participants and the mobile devices corresponding to the selected group of participants, such that each mobile device generates its own composite spectral mask for filtering the audio signal incoming to each mobile device.

[0006] Viewed from a first aspect, the present invention provides a method for isolating and amplifying a conversation between selected participants, the method comprising: receiving a plurality of spectral masks, each spectral mask among the plurality of spectral masks corresponding to a respective participant of a group of selected participants included in the conversation; generating a composite spectral mask by additive superposition of the plurality of spectral masks; and applying the composite spectral mask to audio captured by a microphone to remove audio that does not match the composite spectral mask and to amplify remaining audio that matches the composite spectral mask.

[0007] Preferably, the present invention provides a method further comprising transmitting the amplified remaining audio that matches the composite spectral mask to audio output devices corresponding to the participants in the conversation.

[0008] Preferably, the present invention provides a method further comprising: providing real-time captioning of the conversation; and displaying the real-time captioning of the conversation to participants in the conversation on either a mobile device screen or smart glasses equipped with an audio output device for reading the text of the conversation in addition to listening to the conversation.

[0009] Preferably, the present invention provides a method in which the audio output devices are located near the ears of the participants in the conversation.

[0010] Preferably, the present invention provides a method further comprising sending voice samples of the participants of the conversation to a deep neural network server in a cloud environment to generate a spectral mask personalized for the participant, and receiving the spectral mask personalized for the participant from the deep neural network server.

[0011] Preferably, the present invention provides a method further comprising combining the participant-personalized spectral mask with a plurality of spectral masks corresponding to selected groups of participants included in the conversation to form a composite spectral mask, filtering the incoming audio signal using the composite spectral mask to allow only the conversation between the selected group of participants and the participant to remain in the audio signal, and transmitting the audio signal including only the conversation between the selected group of participants and the participant to an audio output device.

[0012] Preferably, the present invention provides a method further comprising sharing the spectral mask personalized for the participant and the plurality of spectral masks corresponding to selected groups of participants among the participants and the mobile devices corresponding to selected groups of participants such that each mobile device generates its own synthetic spectral mask for filtering audio signals coming into each mobile device.

[0013] Preferably, the present invention provides a method whereby spectral masks personalized for participants in a conversation are subtracted from the synthetic spectral mask based on the participants' preferences.

[0014] Preferably, the present invention provides a method whereby a selected group of participants indicates who is permitted to participate in a conversation.

[0015] Preferably, the present invention provides a method, wherein the method is performed by a mobile device.

[0016] Preferably, the present invention provides a method, wherein a mobile device wirelessly receives a plurality of spectral masks from the mobile device corresponding to a selected group of participants to be included in a conversation.

[0017] Viewed from a second aspect, the present invention provides a mobile device for isolating and amplifying a conversation between selected participants, the mobile device comprising: a bus system; a storage device connected to the bus system, the storage device storing program instructions; and a processor connected to the bus system, the processor executing program instructions for: receiving a plurality of spectral masks, each spectral mask in the plurality of spectral masks corresponding to a respective participant of a group of selected participants included in the conversation; generating a composite spectral mask by additive superposition of the plurality of spectral masks; and applying the composite spectral mask to audio captured by a microphone to remove audio that does not match the composite spectral mask and to amplify remaining audio that matches the composite spectral mask.

[0018] Preferably, the present invention provides a mobile device wherein the processor further executes program instructions to transmit the amplified remaining audio that matches the composite spectral mask to an audio output device corresponding to a participant in the conversation.

[0019] Preferably, the present invention provides a mobile device wherein the processor further executes program instructions for: providing real-time subtitling of the conversation; and displaying the real-time subtitling of the conversation to participants in the conversation on either the mobile device screen or smart glasses having an audio output device attached thereto for reading the text of the conversation in addition to listening to the conversation.

[0020] Preferably, the invention provides a mobile device in which the audio output devices are located near the ears of the participants in a conversation.

[0021] Preferably, the present invention provides a mobile device, wherein the processor further executes program instructions for sending voice samples of the participants of the conversation to a deep neural network server in the cloud environment to generate a spectral mask personalized for the participant, and receiving the spectral mask personalized for the participant from the deep neural network server.

[0022] Preferably, the present invention provides a mobile device wherein the processor further executes program instructions for: combining the participant-personalized spectral mask with a plurality of spectral masks corresponding to selected groups of participants included in the conversation to form a composite spectral mask; filtering the incoming audio signal using the composite spectral mask to allow only the conversation between the selected group of participants and the participant to remain in the audio signal; and transmitting the audio signal including only the conversation between the selected group of participants and the participant to an audio output device.

[0023] Preferably, the present invention provides a mobile device, wherein the processor further executes program instructions for sharing the spectral mask personalized for the participant and the multiple spectral masks corresponding to selected groups of participants among the participants and the mobile devices corresponding to selected groups of participants, such that each mobile device generates its own synthetic spectral mask for filtering audio signals incoming to each mobile device.

[0024] Viewed from a third aspect, the present invention provides a computer program product for isolating and amplifying a conversation between selected participants, the computer program product comprising a computer readable storage medium having embodied thereon program instructions executable by a mobile device to cause the mobile device to perform a method comprising receiving a plurality of spectral masks, each spectral mask among the plurality of spectral masks corresponding to a respective participant of a group of selected participants included in the conversation; generating a composite spectral mask by additive superposition of the plurality of spectral masks; and applying the composite spectral mask to audio captured by a microphone to remove audio that does not match the composite spectral mask and to amplify remaining audio that matches the composite spectral mask.

[0025] Preferably, the present invention provides a computer program product further comprising transmitting the amplified remaining audio that matches the composite spectral mask to an audio output device corresponding to a participant of the conversation.

[0026] Preferably, the present invention provides a computer program product further comprising: generating real-time subtitling of the conversation; and displaying the real-time subtitling of the conversation to participants in the conversation on either a mobile device screen or smart glasses equipped with an audio output device for reading the text of the conversation in addition to listening to the conversation.

[0027] Preferably, the present invention provides a computer program product, wherein the audio output devices are located near the ears of the participants in the conversation.

[0028] Preferably, the present invention provides a computer program product further comprising sending voice samples of participants of a conversation to a deep neural network server in a cloud environment to generate a spectral mask personalized for the participant, and receiving the spectral mask personalized for the participant from the deep neural network server.

[0029] Preferably, the present invention provides a computer program product further comprising: combining the participant-personalized spectral mask with a plurality of spectral masks corresponding to selected groups of participants included in the conversation to form a composite spectral mask; filtering the incoming audio signal using the composite spectral mask to allow only the conversation between the selected group of participants and the participant to remain in the audio signal; and transmitting the audio signal including only the conversation between the selected group of participants and the participant to an audio output device.

[0030] Preferably, the present invention provides a computer program product further comprising sharing the spectral mask personalized for the participant and the multiple spectral masks corresponding to selected groups of participants among the participants and the mobile devices corresponding to selected groups of participants such that each mobile device generates its own synthetic spectral mask for filtering audio signals coming into each mobile device.

[0031] As a result, various exemplary embodiments provide clear and secure conversation between a selected group of participants and the participant by using a shared spectral mask from a mobile device corresponding to the selected group of participants to generate a synthetic spectral mask to remove extraneous sounds so that only the participant's voice remains in the audio signal output to the audio output device. [Brief description of the drawings]

[0032] [Figure 1] 1 is a pictorial representation of a network of data processing systems in which illustrative embodiments may be implemented; [Diagram 2] 1 is an illustration of a data processing system in which an illustrative embodiment may be implemented; [Diagram 3] FIG. 1 illustrates a cloud computing environment in which exemplary embodiments may be implemented. [Figure 4] FIG. 1 illustrates an example of abstraction layers in a cloud computing environment, according to an exemplary embodiment. [Diagram 5] FIG. 1 illustrates an example of an audio filtering system according to an exemplary embodiment. [Figure 6] FIG. 13 illustrates an example of an audio filtering application spectral mask combination process according to an exemplary embodiment. [Figure 7] FIG. 2 illustrates an example of a noisy environment communication process according to an exemplary embodiment. [Figure 8] FIG. 1 illustrates an example of an audio filtering process according to an exemplary embodiment. [Figure 9] FIG. 13 illustrates an example of an alternative audio filtering process according to an exemplary embodiment. [Figure 10] 1 is a flowchart illustrating a process for receiving a personalized spectral mask according to an example embodiment. [Figure 11A] 1 is a flowchart illustrating a process for filtering an incoming audio signal in accordance with an exemplary embodiment. [Figure 11B] 1 is a flowchart illustrating a process for filtering an incoming audio signal in accordance with an exemplary embodiment. [Figure 12] 1 is a flowchart illustrating a process for isolating and amplifying a conversation between selected participants, according to an example embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] The present invention may be a system, method, and / or computer program product at any possible level of integration of technical detail. The computer program product may include a computer readable storage medium(s) having computer readable program instructions for causing a processor to implement aspects of the present invention.

[0034] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), static random access memories (SRAM), portable compact disk read-only memories (CD-ROM), digital versatile disks (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or raised structures in grooves in which instructions are recorded, or any suitable combination of the above. A computer-readable storage medium as used in the present invention should not be construed as a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses through a fiber optic cable), or electrical signals transmitted over electrical wires.

[0035] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to each computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0036] The computer readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk®, C++, and procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server, as a stand-alone software package. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or the connection may be made to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to perform aspects of the invention.

[0037] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or blocks, can be implemented by computer readable program instructions.

[0038] Those computer readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine such that the instructions, executed via the processor of the computer or other programmable data processing apparatus, create means for performing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams. Those computer readable program instructions may be stored in a computer readable storage medium capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner such that the computer readable storage medium on which the instructions are stored constitutes an article of manufacture including instructions implementing aspects of the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0039] The computer readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device, causing the computer, other programmable apparatus, or other device to perform a series of operational steps to generate a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0040] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or part of an instruction, which includes one or more executable instructions for performing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may actually be realized as one step, may be executed simultaneously, may be executed substantially simultaneously, or may be executed with partial or complete overlap in time, or the blocks may sometimes be executed in reverse order depending on the functions involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of each block in the block diagrams and / or flowchart illustrations, may be implemented by a special-purpose hardware-based system that performs the specified functions or operates or implements a combination of special-purpose hardware and computer instructions.

[0041] Referring now to the figures, and in particular to Figures 1-5, diagrams of data processing environments are provided in which illustrative embodiments may be implemented. It should be appreciated that Figures 1-5 are intended as examples only and are not intended to assert or imply any limitation with respect to the environments in which various embodiments may be implemented. Many modifications to the depicted environments may be made.

[0042] 1 depicts a diagrammatic representation of a network of data processing systems in which exemplary embodiments may be implemented. Network data processing system 100 is a network of computers, data processing systems, and other devices in which exemplary embodiments may be implemented. Network data processing system 100 includes network 102, which is the medium used to provide communications links between the computers, data processing systems, and other devices connected together in network data processing system 100. Network 102 may include connections such as, for example, wired communications links, wireless communications links, fiber optic cables, and the like.

[0043] In the depicted example, server 104 and server 106 connect to network 102 along with storage 108. Server 104 and server 106 may be, for example, server computers with high speed connections to network 102. In addition, server 104 and server 106 provide personalized spectral mask generation services to client device users. It should also be noted that server 104 and server 106 may each represent multiple computing nodes in one or more cloud environments. Alternatively, server 104 and server 106 may each represent a cluster of servers in one or more data centers.

[0044] Client 110, client 112, and client 114 also connect to network 102. Clients 110, 112, and 114 are clients of server 104 and server 106. In this example, clients 110, 112, and 114 are shown as mobile communication devices, e.g., cellular telephones, smartphones, etc., with wireless communication links to network 102. However, it should be noted that clients 110, 112, and 114 are merely examples and may represent other types of mobile communication devices, e.g., laptop computers, handheld computers, smart watches, gaming devices, etc., with wireless communication links to network 102. Users of clients 110, 112, and 114 may utilize clients 110, 112, and 114 to access personalized spectral mask generation services provided by server 104 and server 106.

[0045] Storage 108 is a network storage device capable of storing any type of data in a structured or unstructured form. In addition, storage 108 may represent multiple network storage devices. Further, storage 108 may store identifiers and network addresses of multiple different client devices, identifiers of multiple different users, voice samples of multiple different users, personalized spectral masks for multiple different users, etc. Furthermore, storage 108 may store other types of data, such as authentication or credential data, which may include, for example, usernames, passwords, and biometric templates associated with client device users.

[0046] In addition, it should be noted that network data processing system 100 may include any number of additional servers, clients, storage devices, and other devices not shown. Program code located within network data processing system 100 may be stored on a computer readable storage medium and downloaded to a computer or other data processing device for use. For example, program code may be stored on a computer readable storage medium at server 104 and downloaded over network 102 to client 110 for use by client 110.

[0047] In the depicted example, network data processing system 100 may be implemented as any number of different types of communication networks, such as, for example, a telecommunications network, the Internet, an intranet, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), or any combination thereof. Figure 1 is intended as an example only, and not as an architectural limitation for different illustrative embodiments.

[0048] Referring now to Figure 2, a diagram of a data processing system according to an exemplary embodiment is depicted. Data processing system 200 is an example of a mobile communication device, such as client 110 of Figure 1, in which computer readable program code or instructions implementing the processes of the exemplary embodiments may be located. In this example, data processing system 200 includes a communications fabric 202 that provides communications between a processor unit 204, a memory 206, persistent storage 208, a communications unit 210, an input / output (I / O) unit 212, and a display 214.

[0049] The processor unit 204 serves to execute instructions of software applications and programs that may be loaded into the memory 206. The processor unit 204 may be a set of one or more hardware processor devices or a multi-core processor, depending on the particular implementation.

[0050] Memory 206 and persistent storage 208 are examples of storage 216. A computer-readable storage is any hardware capable of storing information, such as, but not limited to, data, computer-readable program code in a functional form, or other suitable information, or a combination thereof, either transiently or persistently. Additionally, computer-readable storage excludes propagation media. Memory 206 in such examples may be, for example, random access memory (RAM), or any other suitable volatile or non-volatile storage, such as flash memory. Persistent storage 208 may take various forms depending on the particular implementation. For example, persistent storage 208 may include one or more devices. For example, persistent storage 208 may be a disk drive, a solid-state drive, or some combination of the above.

[0051] In this example, persistent storage 208 stores audio filtering application 218. However, it should be noted that while audio filtering application 218 is shown as residing in persistent storage 208, in alternative exemplary embodiments, audio filtering application 218 may be a combination of hardware and software components separate from persistent storage 208. In another alternative exemplary embodiment, a first set of components of audio filtering application 218 may be located within data processing system 200 and a second set of components of audio filtering application 218 may be located in a second data processing system, such as server 104 of FIG.

[0052] The audio filtering application 218 controls a process of combining the spectral masks corresponding to the conversation participants by additive superposition to isolate and amplify only the audio signal (i.e., voice) of each conversation participant to improve the intelligibility and security of the conversation. The user 220 represents an identifier of a user corresponding to the data processing system 200. It should be noted that the user 220 may also be a participant in a conversation with one or more other mobile communication device users over a network, such as the network 102 of FIG. 1.

[0053] The voice sample 222 represents a biometric sample of the voice of the user 220. In other words, the voice sample 222 is a recording of the voice of the user 220. The user 220 sends the voice sample 222 from the data processing system 200 to a spectral mask generation computer, such as the server 104 of FIG.

[0054] The spectral mask generation computer utilizes a deep learning neural network to generate the spectral mask 224, which is personalized for the user 220 based on the voice sample 222. Deep learning, also known as deep structured learning or hierarchical learning, is a part of machine learning based on artificial neural networks. Machine learning can be supervised, semi-supervised, or unsupervised. Deep learning architectures such as deep neural networks, recurrent neural networks, and convolutional neural networks have applications in fields including speech recognition, audio recognition, natural language processing, machine translation, and more. Deep neural networks include multiple layers between the input layer and the output layer. Deep neural networks move between the multiple layers to calculate the probability of each output. The spectral mask 224 filters the incoming audio signal so that only signals that match the spectral mask 224 (i.e., only voice signals that approximately correspond to the user 220) remain. In other words, the spectral mask 224 filters out all other audio signals that do not match the spectral mask 224 .

[0055] The set of spectral masks 226 represents one or more pre-shared, stored sets of spectral masks corresponding to other users of mobile communication devices that are equipped with the audio filtering application of the exemplary embodiment. The audio filtering application 218 may display the set of spectral masks 226 to the user 220 on the display 214 immediately prior to or at the start of a conversation with another mobile communication device user to select some, all, or none of the set of spectral masks 226. The group of selected conversation participants 228 represents one or more other mobile communication device users that the user 220 has selected to participate in a conversation with the user 220 via the data processing system 200 and each of their respective mobile communication devices.

[0056] If one or more spectral masks of a selected conversation participant (i.e., the selected group of conversation participants 228) are not listed in the set of spectral masks 226, the audio filtering application 218 may request a download or automatically initiate a wireless download of the spectral masks not included in the set of spectral masks 226 corresponding to the selected conversation participant. The audio filtering application 218 may then generate a composite spectral mask 230, which is a combination of all the spectral masks corresponding to the user 220 and the selected group of conversation participants 228. Alternatively, the audio filtering application 218 may generate the composite spectral mask 230 using a shared spectral mask that corresponds only to the selected group of conversation participants 228 and not to the user 220, based on the preferences of the user 220.

[0057] The audio filtering application 218 generates a composite spectral mask 230 using an additive superposition of the spectral masks corresponding to the user 220 and the selected group of conversation participants 228. The audio filtering application 218 utilizes the composite spectral mask 230 to isolate and amplify only the audio signals of each participant in the conversation to form an audio signal 232. In other words, the audio signal 232 consists only of the voices of the conversation participants and minimal other unrelated audio signals such as background noise. Thus, by applying the composite spectral mask 230 to all incoming audio signals in the conversation, the audio filtering application 218 transforms the incoming audio signals into a different state or thing (i.e., audio signal 232) that contains almost only the voices of the conversation participants.

[0058] The audio filtering application 218 sends the audio signal 232 to an audio output device 234. The audio output device 234 represents an identifier for an audio output device, such as a headset, earpiece, headphones, etc., connected to the data processing system 200 by a wired or wireless connection. The audio output device 234 is placed in, on, or near one or both ears of the user 220 for clear and safe hearing of the audio signal 232.

[0059] The communications unit 210, in this example, provides communication with other computers, data processing systems, and devices over a network. The communications unit 210 may provide communication through the use of both physical and wireless communications links. The physical communications links may utilize, for example, wires, cables, universal serial bus, or any other physical technology for establishing a physical communications link for the data processing system 200. The wireless communications links may utilize, for example, short wave, radio frequency, very high frequency, microwave, Wireless Fidelity (Wi-Fi), Bluetooth® technology, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), second generation (2G), third generation (3G), fourth generation (4G), 4G Long Term Evolution (LTE), LTE Advanced, fifth generation (5G), or any other wireless communications technology or standard for establishing a wireless communications link for the data processing system 200.

[0060] Input / output unit 212 allows for the input and output of data with other devices that may be connected to data processing system 200. For example, input / output unit 212 may provide a connection for user input through a keypad, keyboard, mouse, microphone, or any other suitable input device, or combination thereof. Display 214 provides a mechanism for displaying information to a user and may include touch screen functionality, for example, allowing a user to make on-screen selections or input data through a user interface.

[0061] Instructions for the operating system, applications, and / or programs may be located in storage 216, which is in communication with processor unit 204 through communications fabric 202. In this illustrative example, the instructions are in functional form in persistent storage 208. The instructions may be loaded into memory 206 for execution by processor unit 204. The processes of the various embodiments may be performed by processor unit 204 using computer-implemented instructions, which may be located in a memory, such as memory 206. The program instructions are referred to as program code, computer usable program code, or computer readable program code, which may be loaded and executed by a processor in processor unit 204. The program instructions in different embodiments may be embodied in different physical computer readable storage devices, such as memory 206 or persistent storage 208.

[0062] The program code 236 is located in a functional form on a computer readable medium 238, which is selectively removable, and may be loaded or transferred to the data processing system 200 for execution by the processor unit 204. The program code 236 and the computer readable medium 238 form a computer program product 240. In one example, the computer readable medium 238 may be a computer readable storage medium 242 or a computer readable signal medium 244. The computer readable storage medium 242 may include, for example, an optical or magnetic disk that is inserted or placed into a drive or other device that is part of the persistent storage 208 for transfer to a storage device, such as a hard drive that is part of the persistent storage 208. The computer readable storage medium 242 may take the form of persistent storage, such as a hard drive, thumb drive, or flash memory connected to the data processing system 200. In some cases, the computer readable storage medium 242 may not be removable from the data processing system 200.

[0063] Alternatively, the program code 236 may be transferred to the data processing system 200 using a computer readable signal medium 244. The computer readable signal medium 244 may be, for example, a propagated data signal containing the program code 236. For example, the computer readable signal medium 244 may be an electromagnetic signal, an optical signal, or any other suitable type of signal, or a combination thereof. The signals may be transmitted over a communications link, for example, a wireless communications link, an optical fiber cable, a coaxial cable, a wire, or any other suitable type of communications link, or a combination thereof. In other words, the communications link and / or connection may be physical or wireless in this illustrative example. The computer readable medium may take the form of a non-tangible medium, such as a communications link or a wireless transmission containing the program code.

[0064] In some exemplary embodiments, program code 236 may be downloaded from another device or data processing system through computer readable signal medium 244 over a network to persistent storage 208 for use within data processing system 200. For example, program code stored on a computer readable storage medium within a data processing system may be downloaded from the data processing system over a network to data processing system 200. The data processing system providing program code 236 may be a server computer, a client computer, or some other device capable of storing and transmitting program code 236.

[0065] The different components illustrated for data processing system 200 are not meant to provide architectural limitations with respect to the manner in which different embodiments may be implemented. The various illustrative embodiments may be implemented in a data processing system including components in addition to or in place of the components illustrated for data processing system 200. Other components depicted in FIG. 2 may vary from the illustrative example shown. The various embodiments may be implemented using any hardware device or system capable of executing program code. As an example, data processing system 200 may include organic components integrated with inorganic components, or may consist entirely of organic components excluding humans, or both. For example, the storage device may be comprised of an organic semiconductor.

[0066] As another example, a computer readable storage device in data processing system 200 is any hardware device that can store data. Memory 206, persistent storage 208, and computer readable storage media 242 are examples of physical storage devices in a tangible form.

[0067] In another example, a bus system may be used to implement communications fabric 202 and may be comprised of one or more buses, such as a system bus or an input / output bus. Of course, the bus system may be implemented using any suitable type of architecture that provides for a transfer of data between various components or devices connected to the bus system. In addition, a communications unit may include one or more devices used for transmitting and receiving data, such as a modem or a network adapter. Further, a memory may be, for example, memory 206 or a cache such as found in interfaces and memory controller hubs that may be present in communications fabric 202.

[0068] Although the present disclosure includes detailed descriptions of cloud computing, it is understood that the implementation of the teachings described herein is not limited to a cloud computing environment. Rather, the exemplary embodiments can be implemented in conjunction with any other type of computing environment now known or later developed. Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources, such as networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services, that can be rapidly provisioned and published with minimal administrative effort or interaction with the providers of the services. The cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0069] These characteristics may include, for example, on-demand self-service, broad network access, resource pooling, rapid elasticity, and metered services. On-demand self-service allows cloud consumers to unidirectionally provision computing capacity, such as server time or network storage, as needed, automatically, without the need for human interaction with the service provider. Broad network access provides the ability to be available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms, such as mobile phones, laptops, and personal digital assistants. Resource pooling allows provider computing resources to be pooled to serve multiple consumers using a multi-tenant model, with various physical and virtual resources dynamically allocated and de-allocated as needed. Consumers generally have no control or knowledge regarding the exact location of the resources provided, but there is a sense of location independence in that location can be specified at a high level of abstraction, such as a country, state, or data center. Rapid elasticity provides rapidly and elastically provisionable capacity that is quickly released to scale up and scale down, sometimes automatically. To the consumer, the capacity available to provision often appears infinite and can be purchased at any time and in any quantity. Metered services enable cloud systems to automatically control and optimize resource usage by leveraging metering capabilities at some level of abstraction appropriate to the type of service, e.g., storage, processing, bandwidth, and available user accounts. Resource usage is monitored, controlled, and reported to provide transparency to both providers and consumers of the services being utilized.

[0070] Service models may include, for example, Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS). Software as a Service is the ability offered to a consumer to use a provider's applications running on a cloud infrastructure. Those applications can be accessed from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application capabilities, except possibly limited user-specific application configuration settings. Platform as a Service is the ability offered to a consumer to deploy applications created or acquired by the consumer, created using programming languages ​​and tools supported by the provider, on a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but has control over the deployed applications and possibly the application host environment configuration. Infrastructure as a service is the ability offered to consumers to provision processing, storage, network, and other basic computing resources on which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating systems, storage, deployed applications, and possibly limited control over selected networking components, such as host firewalls.

[0071] Deployment models may include, for example, private cloud, community cloud, public cloud, and hybrid cloud. A private cloud is a cloud infrastructure operated solely for one organization. A private cloud may be managed by an organization or a third party and may exist on-premise or off-premise. A community cloud is a cloud infrastructure shared by several organizations to support a specific community with shared challenges, for example, mission, security requirements, policies, and compliance considerations. A community cloud may be managed by an organization or a third party and may exist on-premise or off-premise. A public cloud is a cloud infrastructure made available to the general public or large industry groups and owned by an organization that sells cloud services. A hybrid cloud is a cloud infrastructure consisting of two or more clouds, for example a private cloud, a community cloud, and a public cloud, that remain unique entities but are coupled together by standardized or proprietary technologies that enable data and application portability, for example cloud bursting for load balancing between clouds.

[0072] A cloud computing environment is a set of services that emphasize statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0073] Referring now to FIG. 3, a diagram illustrating a cloud computing environment in which an exemplary embodiment may be implemented is depicted. In this illustrative example, the cloud computing environment 300 includes a set of one or more cloud computing nodes 310, with which local computing devices used by cloud consumers, such as a smartphone 320A, a desktop computer 320B, a laptop computer 320C, or an automobile mobile computer system 320N, or combinations thereof, can communicate. The cloud computing nodes 310 may be, for example, the servers 104 and 106 of FIG. 1. The local computing devices 320A-320N may be, for example, the clients 110-114 of FIG. 1.

[0074] Cloud computing nodes 310 may communicate with each other and may be physically or virtually grouped as one or more networks, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud, as described herein above, or a combination thereof. This allows cloud computing environment 300 to provide infrastructure, platform, and / or software as a service, for which cloud consumers do not need to maintain resources on a local computing device, such as local computing devices 320A-320N. It is understood that the types of local computing devices 320A-320N are intended to be merely exemplary, and that cloud computing nodes 310 and cloud computing environment 300 may communicate with any type of computerized device, through any type of network or network-addressed connection, or a combination thereof, using, for example, a web browser.

[0075] Referring now to Figure 4, a diagram illustrating abstraction model layers is depicted, according to an exemplary embodiment. The set of functional abstraction layers depicted in this illustrative example may be provided by a cloud computing environment, such as cloud computing environment 300 of Figure 3. It should be understood in advance that the components, layers, and functions depicted in Figure 4 are intended to be merely illustrative, and that embodiments of the present invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0076] The abstraction layers of cloud computing environment 400 include hardware and software layer 402, virtualization layer 404, management layer 406, and workload layer 408. Hardware and software layer 402 includes the hardware and software components of the cloud computing environment. The hardware components may include, for example, mainframe 410, reduced instruction set computing (RISC) architecture based servers 412, servers 414, blade servers 416, storage devices 418, and network and networking components 420. In some exemplary embodiments, the software components may include, for example, network application server software 422 and database software 424.

[0077] The virtualization layer 404 provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers 426, virtual storage 428, virtual networks including virtual private networks 430, virtual applications and operating systems 432, and virtual clients 434.

[0078] In one example, the management layer 406 may provide the following functionality: Resource provisioning 436 provides dynamic procurement of computing and other resources utilized to perform tasks within the cloud computing environment. Metering and pricing 438 provides cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of those resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, and protection for data and other resources. User portal 440 provides consumers and system administrators with access to the cloud computing environment. Service level management 442 provides allocation and management of cloud computing resources such that required service levels are met. Service level agreement (SLA) planning and fulfillment 444 provides proactive configuration and procurement of cloud computing resources in anticipation of future needs in accordance with SLAs.

[0079] The workload tier 408 provides examples of functions for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from the workload tier 408 may include mapping and navigation 446, software development and lifecycle management 448, virtual classroom instructional delivery 450, data analytics processing 452, transaction processing 454, and audio signal filtering 456.

[0080] One of the problems with having a conversation in a noisy environment is the difficulty for participants in the conversation to clearly and reliably hear what is being said by the other participants. Some current solutions use experimental neural networks to amplify speech in noisy environments, but none of these current solutions use hardware and spectral masking techniques that are already available.

[0081] Recent developments in deep learning techniques make it possible to create customized individual spectral masks that embody a detailed characterization of an individual's voice, allowing for the creation of highly specific and effective filters to isolate an individual's voice against background noise.

[0082] Exemplary embodiments isolate and amplify conversations between selected individuals in noisy environments using hardware components already present in, for example, cell phones and wired or wireless earbuds. Exemplary embodiments utilize dynamically generated composite spectral masks that are pre-tuned to the individual participants in the conversation.

[0083] Exemplary embodiments utilize pre-trained deep neural network dynamic spectral masking to enable clear and secure conversations between participants in noisy environments. Exemplary embodiments provide the dynamic spectral mask deep neural network as a cloud service that is coupled with an audio filtering application resident on a mobile phone. A mobile phone user can record voice samples of one or more individuals, upload the voice samples to the cloud service, and have the dynamic spectral mask deep neural network tuned to each individual's voice. Optionally, the individual's voice may be recorded during a conversation and separated using existing speaker diarization techniques or captured individually from a voicemail message. It should be noted that longer voice samples are preferred in exemplary embodiments, and the quality of the individual spectral mask may depend on the length of the voice sample and the bandwidth of the codec used to capture the voice sample. Thus, the quality of the individual spectral mask may be improved when the exemplary embodiments capture longer voice samples in quiet environments with a wide bandwidth encoder, along with multiple voice samples of all phonemes that the speaker's language contains.

[0084] Once the exemplary embodiments generate a personalized dynamic spectral mask for the individual using the dynamic spectral mask deep neural network, the exemplary embodiments download the spectral mask to the individual's mobile device. Additionally, the exemplary embodiments may delete the individual's data in the cloud environment after downloading the spectral mask to the mobile device.

[0085] Prior to or at the start of a conversation, individuals may share their spectrum masks among the mobile devices corresponding to the participants in the conversation such that each individual who wishes to participate in the conversation stores the spectrum masks of all participants on their mobile device. The mobile devices may share the spectrum masks, for example, via Bluetooth, Wi-Fi, near field communications, email, short message service, etc.

[0086] When a participant is ready to converse using the audio filtering application of the exemplary embodiment on their mobile device, the participant places headphones or ear pieces, for example, on their ears and launches the audio filtering application. The participant further instructs the audio filtering application who is permitted to participate in the conversation. The participant may facilitate this selection via their contact list and Bluetooth, Wi-Fi, small message server, email, etc. Alternatively, the participant may select individuals within the user interface of the audio filtering application.

[0087] The audio filtering application then generates a composite or composite spectral mask by additive superposition of the spectrograms of each participant in the conversation. Alternatively, the audio filtering application may generate a composite spectral mask from each participant other than the particular mobile device user according to the user's preferences as to whether the audio filtering application should amplify that particular mobile device user's own voice. The audio filtering application then applies this composite spectral mask to the incoming audio through the mobile device's microphone, removing all audio that does not match the composite spectrogram, and amplifying the remaining audio that matches the composite spectrogram through an earpiece or headphones. Thus, the exemplary embodiment effectively removes the voices of others (e.g., non-participants) and substantially all other extraneous noise during a conversation.

[0088] In an alternative exemplary embodiment, a single audio filtering application may be used with multiple wireless earbuds, headsets, or headphones, or a combination thereof, so that one mobile device can accommodate several participants. In another alternative exemplary embodiment, the audio filtering application may include the capability to perform real-time subtitling of the conversation (optionally with speaker diarization for individual speaker identification) so that the audio filtering application can display the real-time subtitling of the conversation to the participants on a text output device, such as a mobile device screen or smart glasses to which the earpiece is attached, to read the text of the conversation rather than listening to it, or to read and listen at the same time. Additionally, the exemplary embodiment may also include automatic gain control to normalize the volume of selected voices. Additionally, the exemplary embodiment may optionally train an individual's spectral mask to include whispered speech as well as normal speech.

[0089] For conversations involving sensitive content, participants may protect the exchange by purposely moving into a noisy environment and using the audio filtering application of the exemplary embodiment to make it extremely difficult for others to eavesdrop on the conversation. Optionally, the audio filtering application may use the mobile device's external speaker to purposely output out-of-phase, unidentifiable phonemes or speech fragments that the audio filtering application actively removes to reduce the likelihood of eavesdropping or to make eavesdropping more difficult. In an alternative exemplary embodiment, law enforcement may use the audio filtering application to pick up conversations between "persons of interest" in noisy environments using remotely mounted microphones or more specialized voice recording equipment, such as laser microphones.

[0090] Thus, the exemplary embodiments provide one or more technical solutions that overcome technical problems associated with providing clear and secure conversations between mobile communication device users in noisy environments, such that the one or more technical solutions provide technical effects and practical applications in the field of mobile device communications.

[0091] Referring now to Figure 5, a diagram illustrating an example of an audio filtering system according to an exemplary embodiment is depicted. The audio filtering system 500 may be implemented in a network of data processing systems, such as the network data processing system 100 of Figure 1, or in a cloud computing environment, such as the cloud computing environment 300 of Figure 3. The audio filtering system 500 is a system of hardware and software components for performing conversation participant tuned filtering using deep neural network dynamic spectral masking for isolation, amplification and safety of the participants' conversations in noisy environments.

[0092] In this example, the audio filtering system 500 includes a spectral mask generation server 502, a mobile device 504, a mobile device 506, and a mobile device 508. However, it should be noted that the audio filtering system 500 may include any number of servers, mobile devices, and other devices not shown. The spectral mask generation server 502, the mobile device 504, the mobile device 506, and the mobile device 508 may be, for example, the server 104, the client 110, the client 112, and the client 114, respectively, of FIG. 1.

[0093] The user 510 represents a user of the mobile device 504. The mobile device 504 may be, for example, a smartphone or other mobile communication device. The user 510 inputs a voice sample 512 into the mobile device 504. The mobile device 504 records the voice sample 512 and, utilizing an audio filtering application 514, such as the audio filtering application 218 of FIG. 2, sends the voice sample 512 to a spectral mask generation server 502, which in this example resides in a cloud environment 515. The spectral mask generation server 502 utilizes a deep learning neural network component 516 and a graphics processing unit 518 to generate a personalized dynamic spectral mask 520 corresponding to the user 510 based on the voice sample 512. The personalized dynamic spectral mask 520 is a predictive time series model that predicts the next likely frequency signature of the user's 510 voice moment by moment [perhaps approximately once per millisecond] as the voice is processed. In other words, the personalized dynamic spectral mask 520 predicts the user's 510 word speech sequence during a conversation. The spectral mask generation server 502 sends the personalized dynamic spectral mask 520 to the mobile device 504.

[0094] The stored separate personal spectral masks 522 represent a personal dynamic spectral mask 520 corresponding to the user 510 and other spectral masks corresponding to other mobile communication device users that were previously shared with the mobile device 504. In this example, the mobile device 506 and the mobile device 508 correspond to other users selected by the user 510 to be participants in the conversation. As a result, at 524, the mobile device 506 and the mobile device 508 wirelessly share their respective spectral masks, spectral mask 526 and spectral mask 528, with the mobile device 504, e.g., via Bluetooth technology. After receiving the spectral mask 526 and the spectral mask 528, the audio filtering application 514 combines the personal dynamic spectral mask 520 with the spectral mask 526 and the spectral mask 528 to form a composite spectral mask for filtering the microphone feeds of the mobile device 504, the mobile device 506, and the mobile device 508 so that only the voices of the user 510 and the users of the mobile device 506 and the mobile device 508 are heard.

[0095] 6, a diagram illustrating an example of an audio filtering application spectral mask combination process is depicted according to an exemplary embodiment. The audio filtering application spectral mask combination process 600 may be implemented within a mobile device, such as the client 110 of FIG. 1, the data processing system 200 of FIG. 2, the smartphone 320A of FIG. 3, or the mobile device 504 of FIG. 5. In this example, the audio filtering application spectral mask combination process 600 combines a spectral mask 602 for participant 1 and a spectral mask 604 for participant 2. However, it should be noted that the audio filtering application spectral mask combination process 600 may combine any number of spectral masks corresponding to any number of selected conversation participants.

[0096] At the start of a conversation between participant 1 and participant 2, an audio filtering application, such as audio filtering application 218 of FIG. 2 or audio filtering application 514 of FIG. 5, combines participant 1's spectral mask 602 and participant 2's spectral mask 604 into a composite spectral mask 606. The composite spectral mask 606 uses a logical OR in the frequency domain to create a filter that allows only the voices of participant 1 and participant 2, and not other extraneous voices, in the outgoing audio signal.

[0097] 7, a diagram illustrating an example of a noisy environment communication process according to an exemplary embodiment is depicted. The noisy environment communication process 700 includes a mobile device 702 and a mobile device 704. An audio filtering application of the exemplary embodiment running on the mobile device 702 and the mobile device 704 applies a synthetic spectral mask 706 and a synthetic spectral mask 708, respectively, to the incoming audio signals of a conversation between the participants 710 and 712 such that the participants 710 and 712 can clearly and safely hear each other and themselves speaking without other sounds from the noisy environment being included.

[0098] Referring now to FIG. 8, a diagram illustrating an example of an audio filtering process according to an exemplary embodiment is depicted. The audio filtering process 800 includes a mobile device 802 and an audio output device 804 that is wired or wirelessly coupled to the mobile device 802. The audio output device 804 may be, for example, earbuds, smart glasses with ear pieces, etc. It should be noted that an alternative exemplary embodiment may convert the audio signal of the conversation into text format and output the text to the smart glasses or mobile device screen instead of or in addition to the audio.

[0099] The mobile device 802 stores a spectral mask 806 that corresponds to a conversation participant 808, who is a user of the mobile device 802. It should also be noted that the audio output device 804 is worn by the conversation participant 808 near or in one or both ears of the conversation participant 808.

[0100] At 810, the mobile device 802 receives all audio from the noisy environment, including microphone feeds, from the mobile devices corresponding to the conversation participants 812 and 814. At 816, the mobile device 802 filters the incoming audio using a composite spectral mask, which is a combination of the spectral masks selected by the conversation participant 808, including the spectral mask 806 and the spectral masks corresponding to the conversation participants 812 and 814. At 818, the mobile device 802 sends the filtered audio signals of the conversation participants 808 to the audio output device 804 via Bluetooth or wired for clearly and safely hearing and / or viewing the conversation.

[0101] 9, a diagram illustrating an example of an alternative audio filtering process according to an exemplary embodiment is depicted. The alternative audio filtering process 900 includes a mobile device 902 and an audio output device 904 that is wired or wirelessly coupled to the mobile device 902.

[0102] The mobile device 902 stores a spectral mask 906 corresponding to a conversation participant 908, who is a user of the mobile device 902. Additionally, the audio output device 904 is worn by the conversation participant 908 near or in one or both ears of the conversation participant 908.

[0103] At 910, the mobile device 902 receives all audio from the noisy environment 912, including the voices of the conversation participants 914 and 916, via a laser monitoring device 918. The laser monitoring device 918 may be, for example, a laser microphone. At 920, the mobile device 802 filters the incoming audio using a composite spectral mask, which is a combination of the spectral masks selected by the conversation participants 908. The composite spectral mask includes the spectral mask 806 and the spectral masks corresponding to the conversation participants 914 and 916. At 922, the mobile device 902 sends the filtered audio signals of the conversation participants 908 to the audio output device 904 via Bluetooth or wired for clear and safe listening and / or viewing of the conversation.

[0104] 10, a flow chart illustrating a process for receiving a personalized spectral mask is shown, according to an exemplary embodiment. The process shown in FIG. 10 may be implemented within a mobile device, such as the client 110 of FIG. 1, the data processing system 200 of FIG. 2, the smartphone 320A of FIG. 3, or the mobile device 504 of FIG. 5.

[0105] The process begins when the mobile device receives an input to record a voice sample of a user of the mobile device (step 1002). The mobile device records the user's voice sample (step 1004). The mobile device sends the user's voice sample over a network to a dynamic spectral mask deep neural network server in a cloud environment to generate a spectral mask personalized for the user (step 1006).

[0106] The mobile device then receives the user-personalized spectral mask from the dynamic spectral mask deep neural network server over the network (step 1008). The mobile device stores the user-personalized spectral mask in a mobile device storage device (step 1010). The process then ends.

[0107] 11A-11B, a flow chart illustrating a process for filtering an incoming audio signal is shown according to an exemplary embodiment. The process shown in FIGURES 11A-11B may be implemented within a mobile device, such as the client 110 of FIGURE 1, the data processing system 200 of FIGURE 2, the smartphone 320A of FIGURE 3, or the mobile device 504 of FIGURE 5.

[0108] The process begins when the mobile device receives input from a user of the mobile device to activate an audio filtering application on the mobile device (step 1102). In response to activating the audio filtering application on the mobile device in step 1102, the mobile device connects to other mobile devices within wireless range, e.g., Bluetooth range, that are also running an audio filtering application (step 1104). The mobile device then displays a list of spectral masks that correspond to users of other mobile devices within wireless range that are also running an audio filtering application (step 1106).

[0109] The mobile device receives from the mobile device user a selection of one or more spectral masks in the list of spectral masks that correspond to one or more other mobile device users that the mobile device wishes to include in the conversation (step 1108). The mobile device downloads one or more of the one or more spectral masks selected in the list of spectral masks if the one or more are not already stored in the mobile device's storage (step 1110). The mobile device uses an audio filtering application to combine the one or more spectral masks selected in the list of spectral masks with the spectral masks corresponding to the users stored in storage to form a composite spectral mask (step 1112).

[0110] The mobile device activates the mobile device's microphone to capture an incoming audio signal including a conversation between the user and one or more other mobile device users (step 1114). The mobile device filters the incoming audio signal using a synthetic spectral mask to allow only the conversation between the one or more other mobile device users and the user to remain in the audio signal (step 1116). The mobile device transmits the audio signal including only the conversation between the one or more other mobile device users and the user to one of the earpieces, headsets, or headphones connected to the mobile device (step 1118).

[0111] The mobile device makes a determination as to whether an input has been received to stop the audio filtering application (step 1120). If the mobile device determines that an input has not been received to stop the audio filtering application, i.e., the output of step 1120 is no, the process returns to step 1116, where the mobile device continues to filter the incoming audio signal using the synthetic spectral mask. If the mobile device determines that an input has been received to stop the audio filtering application, i.e., the output of step 1120 is yes, the process then ends.

[0112] 12, a flow chart illustrating a process for isolating and amplifying a conversation between selected participants is shown, according to an exemplary embodiment. The process shown in FIG. 12 may be implemented within a mobile device, such as the client 110 of FIG. 1, the data processing system 200 of FIG. 2, the smartphone 320A of FIG. 3, or the mobile device 504 of FIG. 5.

[0113] The process begins when the mobile device receives a plurality of spectral masks (step 1202), each of which corresponds to a respective participant of a selected group of participants included in the conversation, and the mobile device generates a composite spectral mask by additively superposing the plurality of spectral masks (step 1204).

[0114] The mobile device applies the synthetic spectral mask to audio captured by the mobile device's microphone to remove audio that does not match the synthetic spectral mask and amplifies the remaining audio that matches the synthetic spectral mask (step 1206). The mobile device transmits the amplified remaining audio that matches the synthetic spectral mask to an audio output device in or on one or both ears of a participant in the conversation corresponding to the mobile device (step 1208). The process then ends.

[0115] Thus, exemplary embodiments of the present invention provide a computer-implemented method, computer system, and computer program product for conversation participant adjusted filtering using deep neural network dynamic spectral masking for participant speech isolation, amplification, and security in noisy environments. The description of various embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many changes and modifications will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used in this specification has been selected to best explain the principles of the embodiments, practical applications, or technical improvements to the art found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A method for isolating and amplifying a conversation between selected participants, comprising: receiving a plurality of spectral masks, each spectral mask in the plurality of spectral masks corresponding to a respective participant of a group of selected participants included in the conversation; generating a composite spectral mask by additive superposition of the plurality of spectral masks, the composite spectral mask being formed by combining a spectral mask personalized for the participant with the plurality of spectral masks corresponding to the group of selected participants included in the conversation; applying the synthetic spectral mask to audio captured by a microphone to remove audio that does not match the synthetic spectral mask and amplifying remaining audio that matches the synthetic spectral mask, wherein the amplifying is performed by using the synthetic spectral mask to filter an incoming audio signal to remove audio signals that do not match a synthetic spectral mask, so as to allow only the conversation between the selected group of participants and the participant to remain in the audio signal, and thereafter amplifying only the remaining audio signal; The method comprising:

2. 1. A method for isolating a conversation between selected participants, comprising: receiving a plurality of spectral masks, each spectral mask in the plurality of spectral masks corresponding to a respective participant of a group of selected participants included in the conversation; generating a composite spectral mask by additive superposition of the plurality of spectral masks, the composite spectral mask being formed by combining a spectral mask personalized for the participant with the plurality of spectral masks corresponding to the group of selected participants included in the conversation; applying the synthetic spectral mask to audio captured by a microphone to remove audio that does not match the synthetic spectral mask, the removing being performed by using the synthetic spectral mask to filter an incoming audio signal to allow only the conversation between the selected group of participants and the participant to remain in the audio signal; The method comprising:

3. 1. A method for isolating and amplifying a conversation between selected participants, comprising: receiving a plurality of spectral masks, each spectral mask in the plurality of spectral masks corresponding to a respective participant of a group of selected participants included in the conversation; generating a composite spectral mask by additively superposing the plurality of spectral masks; applying the synthetic spectral mask to audio captured by a microphone to remove audio that does not match the synthetic spectral mask and to amplify remaining audio that matches the synthetic spectral mask; Including, generating the synthetic spectral mask, sharing the spectral mask personalized for the participant and the plurality of spectral masks corresponding to the selected group of participants among the mobile devices corresponding to the participant and the selected group of participants, such that each mobile device generates its own synthetic spectral mask for filtering audio signals coming into each mobile device; combining the spectral mask personalized for the participant shared among the mobile devices with the plurality of spectral masks corresponding to the selected group of participants included in the conversation; Including, The method further comprising: filtering an incoming audio signal using the synthetic spectral mask to allow only the conversation between the selected group of participants and the participant to remain in the audio signal; transmitting the audio signal including only the conversation between the selected group of participants and the participant to an audio output device; Further comprising: The method.

4. 4. The method of claim 1, further comprising transmitting the amplified remaining audio that matches the synthetic spectral mask to audio output devices corresponding to participants in the conversation.

5. providing real-time subtitling of said conversation; and displaying the real-time subtitling of the conversation to the participants of the conversation on one of a mobile device screen or smart glasses with an audio output device attached to read the text of the conversation in addition to listening to the conversation; The method of any one of claims 1 to 4, further comprising:

6. The method of claim 4 or 5, wherein the audio output devices are located near the ears of the participants in the conversation.

7. sending voice samples of the participants of the conversation to a deep neural network server in a cloud environment to generate a spectral mask personalized for the participant; receiving the spectral mask personalized for the participant from the deep neural network server; The method of any one of claims 1 to 6, further comprising:

8. transmitting the audio signal including only the conversation between the selected group of participants and the participant to an audio output device; The method of claim 1 further comprising:

9. 4. The method of claim 1, further comprising sharing the spectral mask personalized for the participant and the plurality of spectral masks corresponding to the selected group of participants among mobile devices corresponding to the participant and the selected group of participants, such that each mobile device generates its own synthetic spectral mask for filtering audio signals coming into each mobile device.

10. The method of any one of claims 1 to 9, wherein the spectral masks personalized for the participants of the conversation are subtracted from the synthetic spectral mask based on the preferences of the participants.

11. The method of any one of claims 1 to 10, wherein the selected group of participants indicates who is permitted to take part in the conversation.

12. The method according to any one of claims 1 to 11, wherein the method is performed by a mobile device.

13. 13. The method of claim 12, wherein the mobile device wirelessly receives the plurality of spectral masks from mobile devices corresponding to the selected group of participants included in the conversation.

14. A method according to any one of the preceding claims, wherein the spectral mask is a mathematically defined filter applied to the audio.

15. 15. The method of any one of claims 1 to 14, wherein the personalized spectral mask is a predictive time series model that predicts the next likely frequency signature of a participant's voice for each moment as the voice is processed.

16. The method of any one of claims 1 to 15, wherein the personalized spectral mask predicts a participant's speech sequence of words during a conversation.

17. 17. The method of any one of claims 1 to 16, wherein the synthetic spectral mask uses logical OR in the frequency domain to create a filter that allows only the voice of the selected participant in the outgoing audio signal and not other irrelevant voices.

18. 1. A mobile device for isolating and amplifying a conversation between selected participants, comprising: A bus system; a storage device connected to said bus system, said storage device storing program instructions; A processor connected to the bus system, the processor executing the method according to any one of claims 1 to 17; The mobile device comprising:

19. A computer program for isolating and amplifying a conversation between selected participants, said computer program causing a mobile device to carry out the method according to any one of claims 1 to 18.

20. 20. A computer readable storage medium having stored thereon the computer program of claim 19.

Images