Participant-tuned filtering using deep neural network dynamic spectral masks for conversation isolation and safety in noisy environments

Abstract

Isolating and amplifying a conversation between selected participants is provided. A plurality of spectral masks are received. Each spectral mask of the plurality of spectral masks corresponds to a respective participant of a selected group of participants included in the conversation. A composite spectral mask is generated by additive superposition of the plurality of spectral masks. The composite spectral mask is applied to sound captured by a microphone to filter out sounds that do not match the composite spectral mask and to amplify remaining sounds that match the composite spectral mask.

Description

Technical Field

[0001] The present disclosure relates generally to spectral masks and, more particularly, to participant tuning filtering of conversations using deep neural network dynamic spectral masks for participant conversation isolation and safety in noisy environments. Background Art

[0002] In telecommunications, a spectral mask is a mathematically defined filter applied to sounds. Spectral masks are generally intended to reduce sounds that do not match the spectral mask. These unwanted sounds are usually attenuated by allowing only those sounds that match the spectral mask to pass through. Summary of the invention

[0003] According to one illustrative embodiment, a method for isolating and amplifying a conversation between selected participants is provided. A plurality of spectral masks are received. Each spectral mask of the plurality of spectral masks corresponds to a respective participant of a selected group of participants included in the conversation. A composite spectral mask is generated by additive superposition of the plurality of spectral masks. The composite spectral mask is applied to sound captured by a microphone to filter out sounds that do not match the composite spectral mask and to amplify remaining sounds that match the composite spectral mask. According to other illustrative embodiments, a mobile device and a computer program product for isolating and amplifying a conversation between selected participants are provided.

[0004] Different illustrative embodiments also transmit the amplified residual sound matched to the composite spectral mask to an audio output device corresponding to a participant in the conversation. Further, the audio output device is adjacent to the ear of the participant in the conversation.

[0005] In addition, different illustrative embodiments send speech samples of participants of the conversation to a deep neural network server of the cloud environment for generating a spectral mask personalized for the participant, and receive the spectral mask personalized for the participant from the deep neural network server. Different illustrative embodiments combine the spectral mask personalized for the participant with multiple spectral masks corresponding to a selected group of participants included in the conversation to form the composite spectral mask, use the composite spectral mask to filter incoming audio signals to allow only the conversation between the selected group of participants and the participant to remain in the audio signal, and transmit the audio signal including only the conversation between the selected group of participants and the participant to the audio output device. In addition, different illustrative embodiments share the spectral mask personalized for the participant and the multiple spectral masks corresponding to the selected group of participants between mobile devices corresponding to the participant and the selected group of participants, so that each mobile device generates its own composite spectral mask for filtering incoming audio signals to each mobile device.

[0006] Viewed from a first aspect, the present invention provides a method for isolating and amplifying a conversation between selected participants, the method comprising: receiving a plurality of spectral masks, each of the plurality of spectral masks corresponding to a respective participant in a selected group of participants included in the conversation; generating a composite spectral mask by additive superposition of the plurality of spectral masks; and applying the composite spectral mask to sounds captured by a microphone to filter out sounds that do not match the composite spectral mask and amplify remaining sounds that match the composite spectral mask.

[0007] Preferably, the present invention provides a method further comprising: transmitting the amplified residual sound matched to the composite spectral mask to an audio output device corresponding to a participant in the conversation.

[0008] Preferably, the present invention provides a method, further comprising: performing real-time subtitles of the conversation; and displaying the real-time subtitles of the conversation to participants of the conversation on one of a mobile device screen or smart glasses to which the audio output device is attached, for reading the text of the conversation and listening to the conversation.

[0009] Preferably, the present invention provides a method wherein the audio output device is adjacent to the ears of participants in the conversation.

[0010] Preferably, the present invention provides a method, further comprising: sending voice samples of the participants of the conversation to a deep neural network server in a cloud environment for generating a spectrum mask personalized for the participants; and receiving the spectrum mask personalized for the participants from the deep neural network server.

[0011] Preferably, the present invention provides a method, further comprising: combining a spectral mask personalized for a participant with a plurality of spectral masks corresponding to a selected group of participants included in a conversation to form a composite spectral mask; filtering an incoming audio signal using the composite spectral mask to allow only the selected group of participants and the conversation between the participants to remain in the audio signal; and transmitting the audio signal including only the selected group of participants and the conversation between the participants to the audio output device.

[0012] Preferably, the present invention provides a method, further comprising: sharing the spectrum mask personalized for the participant and the multiple spectrum masks corresponding to the selected group of participants between mobile devices corresponding to the participant and the selected group of participants, so that each mobile device generates its own composite spectrum mask for filtering incoming audio signals to each mobile device.

[0013] Preferably, the present invention provides a method wherein spectral masks personalized to participants of a conversation are excluded from the composite spectral mask based on the preferences of the participants.

[0014] Preferably, the present invention provides a method wherein a selected group of participants indicates who is authorized to participate in the conversation.

[0015] Preferably, the present invention provides a method, wherein the method is performed by a mobile device.

[0016] Preferably, the present invention provides a method wherein said mobile device wirelessly receives said plurality of spectral masks from mobile devices corresponding to a selected group of participants included in a conversation.

[0017] From a second aspect, the present invention provides a mobile device for isolating and amplifying a conversation between selected participants, the mobile device comprising: a bus system; a storage device connected to the bus system, wherein the storage device stores program instructions; and a processor connected to the bus system, wherein the processor executes the program instructions to: receive a plurality of spectrum masks, each of the plurality of spectrum masks corresponding to a corresponding participant among a selected group of participants included in the conversation; generate a composite spectrum mask by additive superposition of the plurality of spectrum masks; and apply the composite spectrum mask to sounds captured by a microphone to filter out sounds that do not match the composite spectrum mask and amplify remaining sounds that match the composite spectrum mask.

[0018] Preferably, the present invention provides a mobile device wherein the processor further executes program instructions to: transmit the amplified residual sound matching the composite spectral mask to an audio output device corresponding to a participant in the conversation.

[0019] Preferably, the present invention provides a mobile device, wherein the processor further executes the program instructions to: perform real-time captioning of the conversation; and display the real-time captioning of the conversation to participants of the conversation on one of a mobile device screen or smart glasses to which the audio output device is attached, for reading the text of the conversation and listening to the conversation.

[0020] Preferably, the present invention provides a mobile device wherein the audio output device is proximate to the ears of participants in the conversation.

[0021] Preferably, the present invention provides a mobile device, wherein the processor also executes program instructions to: send voice samples of participants in a conversation to a deep neural network server in a cloud environment for generating a spectrum mask personalized for the participant; and receive the spectrum mask personalized for the participant from the deep neural network server.

[0022] Preferably, the present invention provides a mobile device, wherein the processor further executes the program instructions to: combine the spectral mask personalized for the participant with the multiple spectral masks corresponding to a selected group of participants included in the conversation to form the composite spectral mask; use the composite spectral mask to filter the incoming audio signal to allow only the conversation between the selected group of participants and the participant to remain in the audio signal; and transmit the audio signal including only the conversation between the selected group of participants and the participant to the audio output device.

[0023] Preferably, the present invention provides a mobile device, wherein the processor further executes the program instructions to: share the spectrum mask personalized for the participant and the multiple spectrum masks corresponding to the selected group of participants between mobile devices corresponding to the participant and the selected group of participants, so that each mobile device generates its own composite spectrum mask for filtering incoming audio signals to each mobile device.

[0024] From a third aspect, the present invention provides a computer program product for isolating and amplifying a conversation between selected participants, the computer program product comprising a computer-readable storage medium having program instructions embodied therein, the program instructions being executable by a mobile device to cause the mobile device to perform a method comprising: receiving a plurality of spectrum masks, each of the plurality of spectrum masks corresponding to a corresponding participant of a selected group of participants included in the conversation; generating a composite spectrum mask by additive superposition of the plurality of spectrum masks; and applying the composite spectrum mask to sounds captured by a microphone to filter out sounds that do not match the composite spectrum mask and amplify remaining sounds that match the composite spectrum mask.

[0025] Preferably, the present invention provides a computer program product further comprising transmitting the amplified residual sound matching the composite spectral mask to an audio output device corresponding to a participant in the conversation.

[0026] Preferably, the present invention provides a computer program product, further comprising: executing real-time subtitles of the conversation; and displaying the real-time subtitles of the conversation to participants of the conversation on one of a mobile device screen or smart glasses to which the audio output device is attached, for reading the text of the conversation and listening to the conversation.

[0027] Preferably, the present invention provides a computer program product wherein the audio output device is proximate to the ears of participants in the conversation.

[0028] Preferably, the present invention provides a computer program product, which further includes: sending voice samples of the participants of the conversation to a deep neural network server in a cloud environment to generate a spectrum mask personalized for the participants; and receiving the spectrum mask personalized for the participants from the deep neural network server.

[0029] Preferably, the present invention provides a computer program product, further comprising: combining the spectral mask personalized for the participant with the multiple spectral masks corresponding to a selected group of participants included in the conversation to form the composite spectral mask; filtering the incoming audio signal using the composite spectral mask to allow only the selected group of participants and the conversation between the participants to remain in the audio signal; and transmitting the audio signal including only the selected group of participants and the conversation between the participants to the audio output device.

[0030] Preferably, the present invention provides a computer program product, further comprising: sharing the spectrum mask personalized for the participant and the multiple spectrum masks corresponding to the selected group of participants between mobile devices corresponding to the participant and the selected group of participants, so that each mobile device generates its own composite spectrum mask for filtering incoming audio signals to each mobile device.

[0031] Thus, different illustrative embodiments provide clear and secure conversations between a selected group of participants and participants by using a shared spectral mask corresponding to the selected group of participants from a mobile device to generate a composite spectral mask for filtering out extraneous sounds so that only the participant's voice remains in the audio signal output to the audio output device. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a pictorial representation of a network of data processing systems in which the illustrative embodiments may be implemented;

[0033] Figure 2 is a diagram of a data processing system in which the illustrative embodiments may be implemented;

[0034] Figure 3 is a diagram showing a cloud computing environment in which illustrative embodiments may be implemented;

[0035] Figure 4 is a diagram showing an example of abstraction layers of a cloud computing environment according to an illustrative embodiment;

[0036] Figure 5 is a diagram showing an example of an audio filtering system according to an illustrative embodiment;

[0037] Figure 6 is a diagram illustrating an example of a spectral mask combination process for audio filtering applications according to an illustrative embodiment;

[0038] Figure 7 is a diagram illustrating an example of a noisy environment communication process according to an exemplary embodiment;

[0039] Figure 8 is a diagram illustrating an example of an audio filtering process according to an illustrative embodiment;

[0040] Fig. 9 is a diagram illustrating an example of an alternative audio filtering process according to an illustrative embodiment;

[0041] Fig.10 is a flow chart illustrating a process for receiving a personalized spectral mask according to an illustrative embodiment;

[0042] Figure 11A-11B is a flow chart illustrating a process for filtering an incoming audio signal according to an illustrative embodiment; and

[0043] Fig.12 is a flow chart illustrating a process for isolating and amplifying a conversation between selected participants in accordance with an illustrative embodiment. DETAILED DESCRIPTION

[0044] The present invention may be a system, method and / or computer program product of any possible degree of technical detail integration. The computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon for causing a processor to execute various aspects of the present invention.

[0045] Computer readable storage medium can be a tangible device that can retain and store instructions for use by instruction execution devices. Computer readable storage medium can be, for example, but not limited to, electronic storage device, magnetic storage device, optical storage device, electromagnetic storage device, semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage medium includes the following: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device such as punch card or a protruding structure in a groove with instructions recorded thereon, and any suitable combination of the above. Computer readable storage medium as used herein should not be interpreted as a temporary signal itself, such as radio waves or other free propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (e.g., light pulses passing through fiber optic cables) or electrical signals transmitted by wires.

[0046] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or downloaded to an external computer or external storage device. The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0047] The computer-readable program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, configuration data of integrated circuits, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and process programming languages ​​(such as "C" programming languages ​​or similar programming languages). The computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network (including a local area network (LAN) or a wide area network (WAN)), or can be connected to an external computer (for example, using an Internet service provider through the Internet). In certain embodiments, the electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA) can perform computer-readable program instructions by utilizing the state information of the computer-readable program instructions to personalize the electronic circuit so as to perform various aspects of the present invention.

[0048] The present invention will be described below with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present invention. It should be understood that each box of the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer readable program instructions.

[0049] These computer-readable program instructions can be provided to a processor of a computer or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable storage medium in which the instructions are stored includes a manufactured product containing instructions for implementing aspects of the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0050] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in or in multiple boxes in the flowchart and / or block diagram.

[0051] The flow chart and block diagram in the accompanying drawings show the architecture, function and operation of the possible implementation of the system, method and computer program product according to different embodiments of the present invention. To this end, each box in the flow chart or block diagram may represent a module, segment or part of an instruction, which includes one or more executable instructions for realizing a specified logical function. In some alternative implementations, the function marked in the box may not occur in the order marked in the figure. For example, the two boxes shown in succession can actually be completed as a step, and are executed simultaneously, substantially simultaneously, in a partially or completely overlapping manner in time, or the boxes can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action or performs a combination of dedicated hardware and computer instructions.

[0052] Referring now to the drawings, and in particular to Figure 1-Figure 5 , provides a diagram of a data processing environment in which an illustrative embodiment may be implemented. It should be understood that Figure 1-Figure 5 This is meant to be an example only and is not intended to assert or imply any limitation with regard to the environments in which different embodiments may be implemented.Many modifications to the depicted environments may be made.

[0053] Figure 1 A pictorial representation of a network of data processing systems in which the illustrative embodiments may be implemented is shown. Network data processing system 100 is a network of computers, data processing systems, and other devices in which the illustrative embodiments may be implemented. Network data processing system 100 contains network 102, which is a medium for providing communication links between computers, data processing systems, and other devices connected together within network data processing system 100. Network 102 may include connections such as, for example, wired communication links, wireless communication links, fiber optic cables, etc.

[0054] In the depicted example, server 104 and server 106 are connected to network 102 along with storage device 108. Server 104 and server 106 may be, for example, server computers with high-speed connections to network 102. In addition, server 104 and server 106 provide personalized spectral mask generation services to client device users. Moreover, it should be noted that server 104 and server 106 may each represent multiple computing nodes in one or more cloud environments. Alternatively, server 104 and server 106 may each represent a server cluster in one or more data centers.

[0055] Clients 110, 112, and 114 are also connected to network 102. Clients 110, 112, and 114 are clients of server 104 and server 106. In this example, clients 110, 112, and 114 are shown as mobile communication devices having a wireless communication link to network 102, such as, for example, cellular phones, smart phones, etc. However, it should be noted that clients 110, 112, and 114 are merely examples and may represent other types of mobile communication devices having a wireless communication link to network 102, such as, for example, laptop computers, handheld computers, smart watches, gaming devices, etc. Users of clients 110, 112, and 114 may use clients 110, 112, and 114 to access the personalized spectrum mask generation service provided by servers 104 and server 106.

[0056] The storage device 108 is a network storage device capable of storing any type of data in a structured or unstructured format. In addition, the storage device 108 may represent multiple network storage devices. Further, the storage device 108 may store identifiers and network addresses of multiple different client devices, identifiers of multiple different users, voice samples of multiple different users, personalized spectrum masks of multiple different users, etc. In addition, the storage device 108 may store other types of data, such as authentication or credential data that may include, for example, usernames, passwords, and biometric templates associated with client device users.

[0057] In addition, it should be noted that network data processing system 100 may include any number of additional servers, clients, storage devices, and other devices not shown. Program code located in network data processing system 100 may be stored on a computer-readable storage medium and downloaded to a computer or other data processing device for use. For example, program code may be stored on a computer-readable storage medium on server 104 and downloaded to client 110 via network 102 for use on client 110.

[0058] In the depicted example, network data processing system 100 may be implemented as many different types of communications networks, such as a telecommunications network, the Internet, an intranet, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), or any combination thereof. Figure 1 This is intended only as an example and not as an architectural limitation for the different illustrative embodiments.

[0059] See now Figure 2 , depicts a diagram of a data processing system according to an illustrative embodiment. Data processing system 200 is a mobile communication device (such as Figure 1In the example of client 110 in FIG. 1 , computer readable program code or instructions implementing the processes of the illustrative embodiments may be located in the mobile communication device. In this example, data processing system 200 includes communication fabric 202 that provides communication between processor unit 204, memory 206, persistent storage device 208, communication unit 210, input / output (I / O) unit 212, and display 214.

[0060] Processor unit 204 serves to execute instructions for software applications and programs that may be loaded into memory 206. Processor unit 204 may be a set of one or more hardware processor devices or may be a multi-core processor, depending on the particular implementation.

[0061] Memory 206 and persistent storage 208 are examples of storage 216. A computer-readable storage device is any hardware capable of storing information such as, for example but not limited to, data, computer-readable program code in functional form, and / or other suitable information on a transient basis or a persistent basis. Further, a computer-readable storage device excludes a propagation medium. In these examples, memory 206 may be, for example, a random access memory (RAM) or any other suitable volatile or non-volatile storage device, such as flash memory. Persistent storage 208 may take various forms, depending on the particular implementation. For example, persistent storage 208 may include one or more devices. For example, persistent storage 208 may be a disk drive, a solid-state drive, or some combination of the above.

[0062] In this example, persistent storage device 208 stores audio filter application 218. However, it should be noted that even though audio filter application 218 is shown as residing in persistent storage device 208, in alternative illustrative embodiments, audio filter application 218 may be a combination of hardware and software components that are separate from persistent storage device 208. In another alternative illustrative embodiment, a first set of components of audio filter application 218 may be located in data processing system 200, and a second set of components of audio filter application 218 may be located in a second data processing system, such as, for example, Figure 1 Server 104 in.

[0063] Audio filtering application 218 controls the process of combining spectral masks corresponding to participants of a conversation by additive superposition to isolate and amplify only the audio signal (i.e., speech) of each participant of the conversation to improve the clarity and safety of the conversation. User 220 represents an identifier corresponding to a user of data processing system 200. It should be noted that user 220 is also connected via a network (such as, Figure 1 A participant in a conversation with one or more other mobile communication device users in a network 102).

[0064] Voice sample 222 represents a biometric sample of the voice of user 220. In other words, voice sample 222 is a recording of the voice of user 220. User 220 sends voice sample 222 from data processing system 200 to a spectral mask generation computer, such as Figure 1 Server 104 in.

[0065] The spectrum mask generation computer uses a deep learning neural network to generate a spectrum mask 224 that is personalized for the user 220 based on the voice sample 222. Deep learning (also known as deep structured learning or hierarchical learning) is a part of machine learning based on artificial neural networks. Machine learning can be supervised, semi-supervised, or unsupervised. Deep learning architectures (such as deep neural networks, recursive neural networks, and convolutional neural networks) have been applied to fields including speech recognition, audio recognition, natural language processing, machine translation, etc. The deep neural network includes multiple layers between the input layer and the output layer. The deep neural network moves through multiple layers and calculates the probability of each output. The spectrum mask 224 filters the incoming audio signal so that only the signal that matches the spectrum mask 224 (i.e., almost exclusively, the voice signal corresponding to the user 220) is retained. In other words, the spectrum mask 224 filters out all other audio signals that do not match the spectrum mask 224.

[0066] The set of spectrum masks 226 represents a set of one or more previously shared and stored spectrum masks corresponding to other users of mobile communication devices that include the audio filtering application of the exemplary embodiment. The audio filtering application 218 can display the set of spectrum masks 226 to the user 220 in the display 214 just before or at the beginning of a conversation with other mobile communication device users for selection of some, all, or none of the set of spectrum masks 226. The selected set of conversation participants 228 represents one or more other mobile communication device users that the user 220 has selected to participate in the conversation with the user 220 through the data processing system 200 and each of their respective mobile communication devices.

[0067] If one or more spectral masks of the selected conversation participant (i.e., the selected group of conversation participants 228) are not listed in the spectral mask set 226, the audio filtering application 218 may request a download or automatically initiate an over-the-air download of those spectral masks corresponding to the selected conversation participant that are not included in the spectral mask set 226. Thereafter, the audio filtering application 218 generates a composite spectral mask 230 that is a combination of all spectral masks corresponding to the user 220 and the selected group of conversation participants 228. Alternatively, the audio filtering application 218 may generate the composite spectral mask 230 using only the shared spectral masks corresponding to the selected group of conversation participants 228 and not corresponding to the user 220 based on the preference of the user 220.

[0068] The audio filtering application 218 generates a composite spectral mask 230 using the additive superposition of the spectral masks corresponding to the user 220 and the selected set of conversation participants 228. The audio filtering application 218 utilizes the composite spectral mask 230 to isolate and amplify only the audio signals of each participant of the conversation to form an audio signal 232. In other words, the audio signal 232 consists only of the conversation participant's voice and minimal other extraneous audio signals (such as background noise). Thus, the audio filtering application 218 transforms the incoming audio signals into a different state or thing (i.e., audio signal 232) that almost exclusively contains the conversation participant's voice by applying the composite spectral mask 230 to all incoming audio signals during the conversation.

[0069] Audio filtering application 218 sends audio signal 232 to audio output device 234. Audio output device 234 represents an identifier of an audio output device, such as a headset, earphones, or headphones connected to data processing system 200, either wired or wirelessly. Audio output device 234 is located in, on, or near one or both ears of user 220 so that audio signal 232 can be heard clearly and safely.

[0070] In this example, communication unit 210 provides communications with other computers, data processing systems, and devices via a network. Communication unit 210 may provide communications using both physical and wireless communication links. Physical communication links may utilize, for example, wire, cable, universal serial bus, or any other physical technology to establish a physical communication link for data processing system 200. Wireless communication links may utilize, for example, short wave, high frequency, ultra high frequency, microwave, wireless fidelity (Wi-Fi), Technology, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), second generation (2G), third generation (3G), fourth generation (4G), 4G Long Term Evolution (LTE), Advanced LTE, fifth generation (5G), or any other wireless communication technology or standard to establish a wireless communication link for the data processing system 200.

[0071] Input / output unit 212 allows for input and output of data with other devices that may be connected to data processing system 200. For example, input / output unit 212 may provide a connection for user input via a keypad, keyboard, mouse, microphone, and / or some other suitable input device. Display 214 provides a mechanism for displaying information to a user and may include touch screen capabilities that allow a user to make on-screen selections or enter data through, for example, a user interface.

[0072] Instructions for the operating system, applications, and / or programs may be located in storage device 216, which communicates with processor unit 204 via communication structure 202. In this illustrative example, the instructions are in functional form on persistent storage device 208. These instructions may be loaded into memory 206 for execution by processor unit 204. The processes of different embodiments may be performed by processor unit 204 using computer-implemented instructions, which may be located in a memory, such as memory 206. These program instructions are referred to as program code, computer-usable program code, or computer-readable program code that may be read and executed by a processor in processor unit 204. In different implementations, the program instructions may be implemented on different physical computer-readable storage devices, such as memory 206 or persistent storage device 208.

[0073] Program code 236 is located in functional form on selectively removable computer readable media 238 and can be loaded or transferred to data processing system 200 for execution by processor unit 204. Program code 236 and computer readable media 238 form computer program product 240. In one example, computer readable media 238 may be computer readable storage media 242 or computer readable signal media 244. Computer readable storage media 242 may include, for example, an optical or magnetic disk that is inserted or placed into a drive or other device that is part of persistent storage device 208 for transfer to a storage device, such as a hard drive that is part of persistent storage device 208. Computer readable storage media 242 may also take the form of a persistent storage device, such as a hard drive, thumb drive, or flash memory that is connected to data processing system 200. In some instances, computer readable storage media 242 may not be removable from data processing system 200.

[0074] Alternatively, program code 236 can be transferred to data processing system 200 using computer readable signal media 244. Computer readable signal media 244 can be, for example, a propagated data signal containing program code 236. For example, computer readable signal media 244 can be an electromagnetic signal, an optical signal, and / or a signal of any other suitable type. These signals can be transmitted through a communication link (e.g., a wireless communication link, a fiber optic cable, a coaxial cable, an electric wire, and / or a communication link of any other suitable type). In other words, in the illustrative example, the communication link and / or connection can be physical or wireless. Computer readable media can also take the form of non-tangible media, such as a communication link or wireless transmission containing program code.

[0075] In some illustrative embodiments, program code 236 may be downloaded from another device or data processing system via a network to persistent storage device 208 through computer readable signal media 244 for use within data processing system 200. For example, program code stored in a computer readable storage medium in a data processing system may be downloaded from the data processing system to data processing system 200 over a network. The data processing system providing program code 236 may be a server computer, a client computer, or some other device capable of storing and transmitting program code 236.

[0076] The different components shown for data processing system 200 are not meant to provide architectural limitations to the manner in which the different embodiments may be implemented. The different illustrative embodiments may be implemented in a data processing system including components in addition to or in place of those shown for data processing system 200. Figure 2 Other components shown in the figure may be different from the illustrative examples shown. Different embodiments may be implemented using any hardware device or system capable of executing program code. As an example, data processing system 200 may include organic components integrated with inorganic components and / or may be composed entirely of organic components that do not include humans. For example, the storage device may be composed of organic semiconductors.

[0077] As another example, a computer-readable storage device in data processing system 200 is any hardware device that can store data. Memory 206, persistent storage 208, and computer-readable storage media 242 are examples of physical storage devices in a tangible form.

[0078] In another example, a bus system can be used to implement the communication structure 202, and can include one or more buses, such as a system bus or an input / output bus. Of course, the bus system can be implemented using any suitable type of architecture that provides data transmission between different components or devices attached to the bus system. In addition, the communication unit may include one or more devices for sending and receiving data, such as a modem or a network adapter. Further, the memory can be, for example, the memory 206 or a high-speed cache such as found in an interface and memory controller hub that may be present in the communication structure 202.

[0079] It should be understood that although the present disclosure includes a detailed description of cloud computing, implementation of the teachings cited herein is not limited to a cloud computing environment. Rather, the illustrative embodiments can be implemented in conjunction with any other type of computing environment now known or later developed. Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources, such as, for example, networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services, which can be quickly provisioned and released with minimal management effort or interaction with the provider of the service. The cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0080] These features may include, for example, on-demand self-service, extensive network access, resource pools, rapid elasticity, and measured services. On-demand self-service allows cloud consumers to unilaterally and automatically provision computing capabilities, such as server time and network storage, on demand without the need for human interaction with the provider of the service. Extensive network access provides the ability to be available over the network and accessed through standard mechanisms that facilitate the use of heterogeneous thin client platforms or thick client platforms (such as mobile phones, laptops, and personal digital assistants). Resource pools allow the provider's computing resources to be pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated as needed. There is a sense of location independence because consumers generally do not have control or knowledge of the exact location of the resources provided, but may be able to specify a location at a higher level of abstraction, such as a country, state, or data center. Rapid elasticity provides the ability to be quickly and elastically provisioned (automatically in some cases) to quickly scale down and quickly release to quickly scale up. For consumers, the capacity available for provisioning generally appears to be unlimited and can be purchased in any quantity at any time. Measured services allow cloud systems to automatically control and optimize resource usage by leveraging metering capabilities at some level of abstraction appropriate to the type of service, such as, for example, storage, processing, bandwidth, and active user accounts. Resource usage can be monitored, controlled, and reported, providing transparency to providers and consumers of the utilized services.

[0081] Service models may include, for example, software as a service (SaaS), platform as a service (PaaS), and infrastructure as a service (IaaS). Software as a service is the ability to use applications that providers run on cloud infrastructure provided to consumers. Applications can be accessed from different client devices through thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure including networks, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings. Platform as a service is the ability to deploy applications created or acquired by consumers using programming languages ​​and tools supported by providers onto cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure including networks, servers, operating systems, or storage, but have control over deployed applications and possible application hosting environment configurations. Infrastructure as a service is the ability to provide consumers with processing, storage, networks, and other basic computing resources that consumers can deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but have control over operating systems, storage, deployed applications, and possibly limited control over selected networking components (such as, for example, host firewalls).

[0082] Deployment models may include, for example, private clouds, community clouds, public clouds, and hybrid clouds. A private cloud is a cloud infrastructure that is operated only for an organization. A private cloud may be managed by an organization or a third party and may exist on-premises or off-premises. A community cloud is a cloud infrastructure shared by several organizations and supports a specific community that shares issues such as missions, security requirements, policies, and compliance considerations. A community cloud may be managed by an organization or a third party and may exist on-premises or off-premises. A public cloud is a cloud infrastructure available to the public or a large industry group and is owned by an organization that sells cloud services. A hybrid cloud is a cloud infrastructure consisting of two or more clouds (such as, for example, private clouds, community clouds, and public clouds) that remain as unique entities but are bound together by standardized or proprietary technologies that enable data and application portability (such as, for example, cloud bursting for load balancing between clouds).

[0083] Cloud computing environments are service-oriented and focus on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is the infrastructure that includes a network of interconnected nodes.

[0084] See now Figure 3, depicts a diagram illustrating a cloud computing environment in which illustrative embodiments may be implemented. In this illustrative example, cloud computing environment 300 includes a set of one or more cloud computing nodes 310 with which local computing devices used by cloud consumers, such as smart phones 320A, desktop computers 320B, laptop computers 320C, and / or automobile computer systems 320N, may communicate. Cloud computing nodes 310 may be, for example, Figure 1 The local computing devices 320A-320N may be, for example, servers 104 and 106. Figure 1 Clients 110-114 in.

[0085] The cloud computing nodes 310 may communicate with each other and may be physically or virtually grouped into one or more networks, such as a private cloud, community cloud, public cloud, or hybrid cloud, or a combination thereof, as described above. This allows the cloud computing environment 300 to provide infrastructure, platform, and / or software as a service for which cloud consumers do not need to maintain resources on local computing devices, such as local computing devices 320A-320N. It should be understood that the types of local computing devices 320A-320N are intended to be illustrative only, and that the cloud computing nodes 310 and cloud computing environment 300 may communicate with any type of computerized device over any type of network and / or network addressable connection using, for example, a web browser.

[0086] See now Figure 4 , depicts a diagram illustrating abstract model layers according to an illustrative embodiment. The set of functional abstraction layers shown in this illustrative example may be provided by a cloud computing environment, such as Figure 3 The cloud computing environment 300 in FIG. Figure 4 The components, layers, and functions shown in are intended to be illustrative only, and embodiments of the present invention are not limited thereto. As described, the following layers and corresponding functions are provided.

[0087] The abstraction layer of cloud computing environment 400 includes hardware and software layer 402, virtualization layer 404, management layer 406, and workload layer 408. Hardware and software layer 402 includes hardware and software components of the cloud computing environment. Hardware components may include, for example, host 410, server 412 based on RISC (reduced instruction set computer) architecture, server 414, blade server 416, storage device 418, and network and networking components 420. In some illustrative embodiments, software components may include, for example, network application server software 422 and database software 424.

[0088] Virtualization layer 404 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 426; virtual storage 428; virtual networks 430, including virtual private networks; virtual applications and operating systems 432; and virtual clients 434.

[0089] In one example, the management layer 406 may provide the functionality described below. Resource provisioning 436 provides dynamic procurement of computing resources and other resources for performing tasks within a cloud computing environment. Metering and pricing 438 provides cost tracking when resources are utilized within a cloud computing environment, and bills or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User portal 440 provides access to the cloud computing environment for consumers and system administrators. Service level management 442 provides cloud computing resource allocation and management so that the required service levels are met. Service level agreement (SLA) planning and fulfillment 444 provides pre-scheduling and procurement of cloud computing resources in anticipation of future needs according to the SLA.

[0090] The workload layer 408 provides examples of functions that can utilize a cloud computing environment. Example workloads and functions that can be provided by the workload layer 408 can include mapping and navigation 446, software development and lifecycle management 448, virtual classroom education delivery 450, data analysis processing 452, transaction processing 454, and audio signal filtering 456.

[0091] The problem with having a conversation in a noisy environment is that it is difficult for participants in the conversation to clearly and safely hear what the other participants are saying. Some current solutions use experimental neural networks to amplify conversations in noisy environments, but none of these current solutions use already available hardware and spectral mask technology.

[0092] Recent developments in deep learning techniques have made it possible to create customized individual spectral masks that embody detailed features of an individual's speech. These deep learning techniques make it possible to create highly specific and effective filters to isolate an individual's speech from background noise.

[0093] The illustrative embodiments use hardware components that already exist, for example, in mobile phones and wired or wireless earbuds to isolate and amplify conversations between selected individuals in a noisy environment. The illustrative embodiments utilize dynamically generated composite spectral masks that are tuned in advance to the various participants in the conversation.

[0094] The illustrative embodiments utilize a pre-tilted deep neural network dynamic spectral mask to enable clear and safe conversations between participants in a noisy environment. The illustrative embodiments provide a dynamic spectral mask deep neural network as a cloud service that is bound to an audio filtering application resident on a mobile phone. A user of the mobile phone can record a voice sample of one or more individuals, upload the voice samples to the cloud service, and have the dynamic spectral mask deep neural network tuned to the voice of each individual. Alternatively, the voice of the individual can be recorded during the conversation and the individual's voice can be isolated using existing speaker diary techniques or captured separately from a voicemail message. It should be noted that the illustrative embodiments prefer longer voice samples, and the quality of individual spectral masks can depend on the voice sample length and bandwidth of the codec used to capture the voice sample. Therefore, when the illustrative embodiments capture the voice sample in length with multiple voice samples of each phoneme included in the speaker's language, in a silent environment, and with a wide bandwidth encoder, the quality of the individual spectral masks can be improved.

[0095] Once the illustrative embodiment generates a personalized dynamic spectrum mask for an individual using a dynamic spectrum mask deep neural network, the illustrative embodiment downloads the spectrum mask to the individual's mobile device. In addition, the illustrative embodiment can delete the individual's data on the cloud environment after downloading the spectrum mask to the mobile device.

[0096] Prior to or at the start of a conversation, individuals can share their spectral masks between mobile devices corresponding to participants in the conversation so that each individual wishing to participate in the conversation will have the spectral masks of all participants stored on their respective mobile devices. Mobile devices can share spectral masks via, for example, Bluetooth, Wi-Fi, near field communication, email, short message service, etc.

[0097] When the participants are ready to talk using the audio filtering application of the illustrative embodiment on their respective mobile devices, the participants place, for example, a headset on their ears or a receiver in their ears and activate the audio filtering application. Further, the participants indicate to the audio filtering application who is authorized to participate in the conversation. The participants can facilitate this selection via a contact list and Bluetooth, Wi-Fi, a small messaging server, email, etc. Alternatively, the participants can select individuals in the user interface of the audio filtering application.

[0098] Next, the audio filtering application generates a composite or composite spectral mask by additively superimposing the spectrograms of each participant in the conversation. Alternatively, the audio filtering application may generate a composite spectral mask from each participant except the user of the particular mobile device based on the user's preference as to whether the audio filtering application should amplify the user's own voice. The audio filtering application then applies this composite spectral mask to the sound incoming through the microphone of the mobile device, filtering out all sounds that do not match the composite spectrogram, and amplifying the remaining sounds that match the composite spectrogram through the earpiece or headset. Thus, the illustrative embodiments effectively filter out any other person's (e.g., non-participant's) voice and nearly all other extraneous noise during the conversation.

[0099] In an alternative illustrative embodiment, a single audio filtering application may be used with multiple wireless earbuds, headphones, and / or headsets, such that one mobile device may serve several participants. In another alternative illustrative embodiment, the audio filtering application may include the ability to perform real-time captioning of the conversation (optionally utilizing speaker diarisation for individual speaker identification) such that the audio filtering application may display real-time captioning of the conversation to the participant on a text output device (such as, for example, a mobile device screen or smart glasses with an attached earpiece) for reading the text of the conversation rather than listening to the conversation or for reading and listening simultaneously. In addition, the illustrative embodiment also includes automatic gain control to normalize the selected speech volume. In addition, the illustrative embodiment may optionally train the individual's spectral mask to include not only normal speech but also whispered speech.

[0100] For conversations containing sensitive content, participants can protect their information exchange by deliberately moving into a noisy environment and using the audio filtering application of the illustrative embodiment to make it very difficult for other individuals to eavesdrop on the conversation. Optionally, the audio filtering application can use the external speakers of the mobile device to deliberately output out-of-phase, unrecognizable phonemes or voice fragments that the audio filtering application actively filters out in order to reduce the possibility of eavesdropping or make eavesdropping more difficult. In an illustrative alternative embodiment, law enforcement can use the audio filtering application to pick out conversations between "persons of interest" in a noisy environment with the help of a remotely attached microphone or a more dedicated recording device (such as, for example, a laser microphone).

[0101] Therefore, the illustrative embodiments provide one or more technical solutions to overcome the technical problems of providing clear and safe conversations between mobile communication device users in noisy environments. Therefore, these one or more technical solutions provide technical effects and practical applications in the field of mobile device communications.

[0102] See now Figure 5, depicts a diagram showing an example of an audio filtering system according to an illustrative embodiment. The audio filtering system 500 can be in a network of a data processing system (e.g., Figure 1 A network data processing system 100 in a cloud computing environment (such as Figure 3 The audio filtering system 500 is implemented in a cloud computing environment 300 in FIG. The audio filtering system 500 is a system of hardware and software components for participant conversation isolation, amplification, and secure participant-tuned filtering using a deep neural network dynamic spectral mask in a noisy environment.

[0103] In this example, the audio filtering system 500 includes a spectral mask generation server 502, a mobile device 504, a mobile device 506, and a mobile device 508. However, it should be noted that the audio filtering system 500 may include any number of servers, mobile devices, and other devices not shown. The spectral mask generation server 502, the mobile device 504, the mobile device 506, and the mobile device 508 may be, for example, Figure 1 Server 104, client 110, client 112, and client 114 in.

[0104] User 510 represents a user of mobile device 504. Mobile device 504 may be, for example, a smart phone or other mobile communication device. User 510 inputs a voice sample 512 into mobile device 504. Mobile device 504 records voice sample 512 and uses an audio filtering application 514 (such as Figure 2 510) to send the speech sample 512 to the spectral mask generation server 502, which in this example resides in the cloud environment 515. The spectral mask generation server 502 utilizes the deep learning neural network component 516 and the graphics processing unit 518 to generate a personalized dynamic spectral mask 520 corresponding to the user 510 based on the speech sample 512. The personalized dynamic spectral mask 520 is a predictive time series model that predicts the next possible frequency signature of the user 510's speech moment by moment (possibly approximately once every millisecond) as the sound is processed. In other words, the personalized dynamic spectral mask 520 predicts the sequence of words that the user 510 will use during the conversation. The spectral mask generation server 502 sends the personalized dynamic spectral mask 520 to the mobile device 504.

[0105] The separate individual stored spectrum masks 522 represent the personal dynamic spectrum mask 520 corresponding to the user 510, as well as other spectrum masks corresponding to other mobile communication device users that were previously shared with the mobile device 504. In this example, the mobile device 506 and the mobile device 508 correspond to other users that the user 510 has selected as participants in the conversation. Therefore, at 524, the mobile device 506 and the mobile device 508 wirelessly share their respective spectrum masks (spectrum mask 526 and spectrum mask 528) with the mobile device 504 via, for example, Bluetooth technology. After receiving the spectrum mask 526 and the spectrum mask 528, the audio filtering application 514 combines the personal dynamic spectrum mask 520 with the spectrum mask 526 and the spectrum mask 528 to form a composite spectrum mask for filtering the microphone feeds of the mobile devices 504, 506, and 508 to allow only the speech of the user 510 and the users of the mobile devices 506 and 508 to be heard.

[0106] See now Figure 6 , depicts a diagram showing an example of an audio filtering application spectral mask combination process in accordance with an illustrative embodiment. The audio filtering application spectral mask combination process 600 may be implemented in a mobile device, such as Figure 1 Client 110, Figure 2 The data processing system 200, Figure 3 Smartphone 320A or Figure 5 5. In this example, the audio filtering application spectral mask combining process 600 combines the participant 1 spectral mask 602 and the participant 2 spectral mask 604. However, it should be noted that the audio filtering application spectral mask combining process 600 can combine any number of spectral masks corresponding to any number of selected participants in the conversation.

[0107] At the beginning of the conversation between Participant 1 and Participant 2, an audio filter is applied (e.g. Figure 2 Audio filtering applications in 218 or Figure 5 The audio filtering application 514 in FIG. 5 combines the participant 1 spectral mask 602 and the participant 2 spectral mask 604 into a composite spectral mask 606. The composite spectral mask 606 uses logical ORing in the frequency domain to create a filter that allows only the speech of participant 1 and participant 2 and no other extraneous sounds in the output audio signal.

[0108] See now Figure 7, depicts a diagram showing an example of a noisy environment communication process according to an illustrative embodiment. The network environment communication process 700 includes a mobile device 702 and a mobile device 704. The audio filtering application of the illustrative embodiment running on the mobile device 702 and the mobile device 704 applies a composite spectrum mask 706 and a composite spectrum mask 708 to the incoming audio signal of the conversation between the participant 710 and the participant 712, respectively, so that the participant 710 and the participant 712 can clearly and safely hear each other and themselves without including other sounds from the noisy environment.

[0109] See now Figure 8 , depicts a diagram showing an example of an audio filtering process according to an illustrative embodiment. The audio filtering process 800 includes a mobile device 802 and an audio output device 804, which is wired or wirelessly coupled to the mobile device 802. The audio output device 804 can be, for example, earbuds, smart glasses with earpieces, etc. It should be noted that instead of or in addition to audio, alternative illustrative embodiments can convert the audio signal of the conversation into a text format and output the text on the smart glasses or mobile device screen.

[0110] Mobile device 802 stores spectral mask 806, which corresponds to conversation participant 808, who is a user of mobile device 802. Also, it should be noted that audio output device 804 is worn by conversation participant 808 near or in one or both ears of conversation participant 808.

[0111] At 810, mobile device 802 receives all sounds from the noisy environment, including microphone feeds from mobile devices corresponding to conversation participant 812 and conversation participant 814. At 816, mobile device 802 filters the incoming sounds using a composite spectral mask that is a combination of spectral masks selected by conversation participant 808, including spectral mask 806 and spectral masks corresponding to conversation participants 812 and 814. At 818, mobile device 802 sends the filtered audio signal to audio output device 804 via Bluetooth or wired for conversation participant 808 to clearly and safely hear and / or see the conversation.

[0112] See now Fig. 9 , depicts a diagram showing an example of an alternative audio filtering process in accordance with an illustrative embodiment. The alternative audio filtering process 900 includes a mobile device 902 and an audio output device 904 coupled to the mobile device 902 by wire or wirelessly.

[0113] The mobile device 902 stores a spectrum mask 906 that corresponds to a conversation participant 908 who is a user of the mobile device 902. Also, an audio output device 904 is worn by the conversation participant 908 near or in one or both ears of the conversation participant 908.

[0114] At 910, the mobile device 902 receives all sounds including the voices of the conversation participants 914 and 916 from the noisy environment 912 via the laser monitoring device 918. The laser monitoring device 918 can be, for example, a laser microphone. At 920, the mobile device 802 filters the incoming sounds using a composite spectrum mask, which is a combination of spectrum masks selected by the conversation participants 908. The composite spectrum mask includes the spectrum mask 806 and the spectrum masks corresponding to the conversation participants 914 and 916. At 922, the mobile device 902 sends the filtered audio signal to the audio output device 904 via Bluetooth or wired for the conversation participants 908 to clearly and safely hear and / or see the conversation.

[0115] See now Fig.10 , a flowchart of a process for receiving a personalized spectral mask is shown in accordance with an illustrative embodiment. Fig.10 The process shown in can be implemented in a mobile device, such as Figure 1 Client 110, Figure 2 The data processing system 200, Figure 3 Smartphone 320A or Figure 5 A mobile device 504 is provided in the embodiment of the present invention.

[0116] The process begins when the mobile device receives an input to record a voice sample of a user of the mobile device (step 1002). The mobile device records the user's voice sample (step 1004). The mobile device sends the user's voice sample to the dynamic spectrum mask deep neural network server of the cloud environment via the network for generating a spectrum mask personalized for the user (step 1006).

[0117] Subsequently, the mobile device receives a spectrum mask personalized for the user from the dynamic spectrum mask deep neural network server via the network (step 1008). The mobile device stores the spectrum mask personalized for the user in a storage device of the mobile device (step 1010). Thereafter, the process terminates.

[0118] See now Figure 11A-11B , a flow chart showing a process for filtering an incoming audio signal in accordance with an illustrative embodiment. Figure 11A-11B The process shown in can be implemented in a mobile device, such as Figure 1 Client 110, Figure 2 The data processing system 200, Figure 3Smartphone 320A or Figure 5 A mobile device 504 is provided in the embodiment of the present invention.

[0119] The process begins when the mobile device receives input from a user of the mobile device to activate an audio filtering application on the mobile device (step 1102). In response to activating the audio filtering application on the mobile device in step 1102, the mobile device connects to other mobile devices that also have activated audio filtering applications within wireless range (such as, for example, within Bluetooth range) (step 1104). Thereafter, the mobile device displays a list of spectral masks corresponding to users of other mobile devices that have activated audio filtering applications within wireless range (step 1106).

[0120] The mobile device receives, from a user of the mobile device, a selection of one or more spectral masks in a spectral mask list corresponding to one or more other mobile device users to be included in the conversation (step 1108). When one or more of the one or more spectral masks are not already stored on a storage device of the mobile device, the mobile device downloads one or more of the one or more spectral masks selected in the spectral mask list (step 1110). The mobile device combines, using an audio filtering application, the one or more spectral masks selected in the spectral mask list with spectral masks corresponding to users stored on the storage device to form a composite spectral mask (step 1112).

[0121] The mobile device activates a microphone of the mobile device to capture an incoming audio signal including a conversation between one or more other mobile device users and the user (step 1114). The mobile device filters the incoming audio signal using a composite spectral mask to allow only the conversation between the one or more other mobile device users and the user to remain in the audio signal (step 1116). The mobile device transmits the audio signal including only the conversation between the one or more other mobile device users and the user to one of an earpiece, a headset, or a headphone connected to the mobile device (step 1118).

[0122] The mobile device determines whether an input is received to deactivate the audio filtering application (step 1120). If the mobile device determines that an input is not received to deactivate the audio filtering application (a no output of step 1120), then the process returns to step 1116, where the mobile device continues to filter the incoming audio signal using the composite spectral mask. If the mobile device determines that an input is received to deactivate the audio filtering application (a "yes" output of step 1120), then the process terminates thereafter.

[0123] See now Fig.12 , a flowchart showing a process for isolating and amplifying a conversation between selected participants in accordance with an illustrative embodiment. Fig.12The process shown in can be implemented in a mobile device, such as Figure 1 Client 110, Figure 2 The data processing system 200, Figure 3 Smartphone 320A or Figure 5 A mobile device 504 is provided in the embodiment of the present invention.

[0124] The process begins when a mobile device receives a plurality of spectral masks (step 1202). Each spectral mask in the plurality of spectral masks corresponds to a respective participant in a selected group of participants included in a conversation. The mobile device generates a composite spectral mask by additive superposition of the plurality of spectral masks (step 1204).

[0125] The mobile device applies the composite spectral mask to the sound captured by the microphone of the mobile device, thereby filtering out the sound that does not match the composite spectral mask and amplifying the remaining sound that matches the composite spectral mask (step 1206). The mobile device transmits the amplified remaining sound that matches the composite spectral mask to an audio output device in or on one or both ears of a participant in the conversation corresponding to the mobile device (step 1208). Thereafter, the process terminates.

[0126] Thus, the illustrative embodiments of the present invention provide a computer-implemented method, computer system, and computer program product for conversation participant tuning filtering using deep neural network dynamic spectral masks for participant conversation isolation, amplification, and safety in noisy environments. The description of various embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, the practical application, or technical improvement over technology found in the marketplace, or to enable one of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for isolating a conversation between selected participants, the method comprising: receiving a plurality of spectral masks, each spectral mask of the plurality of spectral masks corresponding to a respective participant of a selected set of participants included in the conversation, wherein a spectral mask is a mathematically defined filter applied to the sound; generating a composite spectral mask by additive superposition of the plurality of spectral masks, the plurality of spectral masks including a personalized spectral mask personalized for each respective participant included in the conversation, the personalized spectral mask being a predictive time series model that predicts a next frequency signature of the respective participant's speech as the sound is processed; as well as The composite spectral mask is applied to sounds captured by a microphone to filter out sounds that do not match the composite spectral mask, the composite spectral mask using a logical OR operation in the frequency domain, creating a filter that allows only the speech of a selected set of participants included in the conversation.

2. The method according to claim 1, further comprising: Remaining sounds matching the composite spectral mask are transmitted to audio output devices corresponding to participants of the conversation.

3. The method according to claim 2, further comprising: performing real-time captioning of said conversation; as well as The real-time captions of the conversation are displayed to participants of the conversation on one of a mobile device screen or smart glasses to which the audio output device is attached for reading the text of the conversation and listening to the conversation.

4. The method of claim 2, wherein the audio output device is adjacent to an ear of the participant in the conversation.

5. The method according to claim 2, further comprising: sending speech samples of the participants of the conversation to a deep neural network server in a cloud environment for generating a spectral mask personalized for the participants; as well as The spectral mask personalized for the participant is received from the deep neural network server.

6. The method according to claim 5, further comprising: combining the spectral mask personalized to the participant with the plurality of spectral masks corresponding to a selected group of participants included in the conversation to form the composite spectral mask; filtering an incoming audio signal using the composite spectral mask to allow only a selected group of participants and the conversation between the participants to remain in the audio signal; as well as The audio signal including only the conversation between a selected group of participants and the participants is transmitted to the audio output device.

7. The method according to claim 6, further comprising: The spectral mask personalized for the participant and the plurality of spectral masks corresponding to the selected group of participants are shared between mobile devices corresponding to the participant and the selected group of participants such that each mobile device generates its own composite spectral mask for filtering incoming audio signals to each mobile device.

8. The method of claim 5, wherein the spectral mask personalized to a participant of the conversation is excluded from the composite spectral mask based on the participant's preferences.

9. The method of claim 1, wherein the selected set of participants indicates who is authorized to participate in the conversation.

10. The method of claim 1, wherein the method is performed by a mobile device.

11. The method of claim 10, wherein the mobile device wirelessly receives the plurality of spectral masks from mobile devices corresponding to a selected group of participants included in the conversation.

12. A mobile device for isolating a conversation between selected participants, the mobile device comprising: Bus system; a storage device connected to the bus system, wherein the storage device stores program instructions; as well as A processor connected to the bus system, wherein the processor executes the program instructions to: receiving a plurality of spectral masks, each spectral mask of the plurality of spectral masks corresponding to a respective participant of a selected set of participants included in the conversation, wherein a spectral mask is a mathematically defined filter applied to the sound; generating a composite spectral mask by additive superposition of the plurality of spectral masks, the plurality of spectral masks including a personalized spectral mask personalized for each respective participant included in the conversation, the personalized spectral mask being a predictive time series model that predicts a next frequency signature of the respective participant's speech as the sound is processed; as well as The composite spectral mask is applied to sounds captured by a microphone to filter out sounds that do not match the composite spectral mask, the composite spectral mask using a logical OR operation in the frequency domain, creating a filter that allows only the speech of a selected set of participants included in the conversation.

13. The mobile device of claim 12, wherein the processor further executes the program instructions to: Remaining sounds matching the composite spectral mask are transmitted to audio output devices corresponding to participants of the conversation.

14. The mobile device of claim 13, wherein the processor further executes the program instructions to: performing real-time captioning of the conversation; and The real-time captions of the conversation are displayed to participants of the conversation on one of a mobile device screen or smart glasses to which the audio output device is attached for reading the text of the conversation and listening to the conversation.

15. The mobile device of claim 13, wherein the audio output device is adjacent to an ear of the participant in the conversation.

16. The mobile device of claim 13, wherein the processor further executes the program instructions to: Sending speech samples of the participants of the conversation to a deep neural network server in a cloud environment for generating a spectral mask personalized for the participants; and The spectral mask personalized for the participant is received from the deep neural network server.

17. The mobile device of claim 16, wherein the processor further executes the program instructions to: combining the spectral mask personalized to the participant with the plurality of spectral masks corresponding to a selected group of participants included in the conversation to form the composite spectral mask; filtering an incoming audio signal using the composite spectral mask to allow only a selected group of participants and the conversation between the participants to remain in the audio signal; and The audio signal including only the conversation between a selected group of participants and the participants is transmitted to the audio output device.

18. The mobile device of claim 17, wherein the processor further executes the program instructions to: The spectral mask personalized for the participant and the plurality of spectral masks corresponding to the selected group of participants are shared between mobile devices corresponding to the participant and the selected group of participants such that each mobile device generates its own composite spectral mask for filtering incoming audio signals to each mobile device.

19. A computer program product for isolating a conversation between selected participants, the computer program product comprising a computer-readable storage medium having program instructions embodied therein, the program instructions executable by a mobile device to cause the mobile device to perform a method comprising: receiving a plurality of spectral masks, each spectral mask of the plurality of spectral masks corresponding to a respective participant of a selected set of participants included in the conversation, wherein a spectral mask is a mathematically defined filter applied to the sound; generating a composite spectral mask by additive superposition of the plurality of spectral masks, the plurality of spectral masks including a personalized spectral mask personalized for each respective participant included in the conversation, the personalized spectral mask being a predictive time series model that predicts a next frequency signature of the respective participant's speech as the sound is processed; as well as The composite spectral mask is applied to sounds captured by a microphone to filter out sounds that do not match the composite spectral mask, the composite spectral mask using a logical OR operation in the frequency domain, creating a filter that allows only the speech of a selected set of participants included in the conversation.

20. The computer program product of claim 19, further comprising: Remaining sounds matching the composite spectral mask are transmitted to audio output devices corresponding to participants of the conversation.

21. The computer program product of claim 20, further comprising: performing real-time captioning of said conversation; as well as The real-time captions of the conversation are displayed to participants of the conversation on one of a mobile device screen or smart glasses to which the audio output device is attached for reading the text of the conversation and listening to the conversation.

22. The computer program product of claim 20, wherein the audio output device is proximate to an ear of the participant in the conversation.

23. The computer program product of claim 20, further comprising: sending speech samples of the participants of the conversation to a deep neural network server in a cloud environment for generating a spectral mask personalized for the participants; as well as The spectral mask personalized for the participant is received from the deep neural network server.

24. The computer program product of claim 23, further comprising: combining the spectral mask personalized to the participant with the plurality of spectral masks corresponding to a selected group of participants included in the conversation to form the composite spectral mask; filtering an incoming audio signal using the composite spectral mask to allow only a selected group of participants and the conversation between the participants to remain in the audio signal; as well as The audio signal including only a selected group of participants and the conversation between the participants is transmitted to the audio output device.

25. The computer program product of claim 24, further comprising: The spectral mask personalized for the participant and the plurality of spectral masks corresponding to the selected group of participants are shared between mobile devices corresponding to the participant and the selected group of participants such that each mobile device generates its own composite spectral mask for filtering incoming audio signals to each mobile device.

Images