Method and apparatus for separating target speech from multiple speakers

Abstract

Embodiments of the present application provide a method and apparatus for separating a target speech from a plurality of speakers. The method comprises receiving video data associated with the plurality of speakers; receiving audio data associated with each of at least one speaker; extracting video feature data from the received video data; and identifying the target speech from the plurality of speakers based on the received audio data and the extracted video feature data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computing, and more specifically, to a method and apparatus for separating target speech from multiple speakers, a non-transitory computer readable medium, and a computer device. BACKGROUND

[0002] Target speech separation extracts the speech of interest from an observed speech mixture. With the advent of the deep learning era, most existing supervised methods are based on spectrogram masking, in which the weights (mask) of the target speaker at each time-frequency (T-F) window of the mixed spectrogram are estimated. As a result, the product between the mixed spectrogram and the predicted mask is used as the target speech spectrogram. SUMMARY

[0003] Embodiments of the present application provide a method and apparatus for separating target speech from multiple speakers, a non-transitory computer readable medium, and a computer device, aiming to solve the problem that existing pure audio methods are often severely disturbed in complex acoustic environments, resulting in blurred acoustic target information.

[0004] According to an aspect, there is provided a method for separating target speech from multiple speakers, the method can include receiving video data associated with the multiple speakers; receiving audio data associated with each of the at least one speaker; extracting video feature data from the received video data; identifying the target speech from the multiple speakers based on the received audio data and the extracted video feature data.

[0005] According to another aspect, there is provided an apparatus for separating target speech from multiple speakers, the apparatus can include a first receiving module configured to receive video data associated with the multiple speakers; a second receiving module configured to receive audio data associated with each of the at least one speaker; an extracting module configured to extract video feature data from the received video data; an identifying module configured to identify the target speech from the multiple speakers based on the received audio data and the extracted video feature data.

[0006] According to yet another aspect, there is provided a non-transitory computer readable medium having stored thereon a computer program, which, when executed by at least one computer processor, causes the at least one computer processor to perform the method for separating target speech from multiple speakers.

[0007] According to still another aspect, a computer device is provided, comprising at least one processor and at least one memory having at least one program instruction stored therein, the at least one program instruction being loaded and executed by the at least one processor to implement the method for separating target speech from multiple speakers.

[0008] In the embodiments of the present application, the multi-modal framework can be used to extract the speech of the target speaker in the synchronous video and multi-channel audio damaged by noise and reverberation, so as to obtain a more complete and robust target acoustic representation. BRIEF DESCRIPTION OF DRAWINGS

[0009] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which various features of the application are illustrated. The various features are not drawn to scale in order to facilitate understanding of the application by those skilled in the art. In the drawings:

[0010] Figure 1 A networked computer environment according to at least one embodiment is shown;

[0011] Figure 2 is a block diagram of a system for separating target speech from multiple speakers according to at least one embodiment;

[0012] Figure 3 is an operational flow diagram showing operations performed by a program for separating target speech from multiple speakers according to at least one embodiment;

[0013] Figure 4 is a block diagram of a computer and server according to at least one embodiment; Figure 1

[0014] Figure 5 is a block diagram of an exemplary cloud computing environment including Figure 1 a computer system shown in FIG. 1; and

[0015] Figure 6 is a block diagram of functional layers of an exemplary cloud computing environment according to at least one embodiment. Figure 5 DETAILED DESCRIPTION

[0016] ​​The present application discloses specific embodiments of the claimed structure and method. However, it is to be understood that the disclosed embodiments are merely examples of the claimed structure and method, which can be embodied in various forms. However, these structures and methods can be embodied in many different forms and should not be construed as limited to the examples described in this application. Rather, these exemplary embodiments are provided so that this application will be thorough and complete, and will fully convey the scope to those skilled in the art. In the description, details of well-known features and techniques can be omitted to avoid unnecessarily obscuring the presented embodiments.

[0017] Embodiments relate generally to the field of computing, and more particularly, to speech recognition. The exemplary embodiments described below provide an apparatus (system), method, and computer program to separate a target speech from a plurality of speakers. Thus, by allowing the separation of a single speaker's speech from data containing speech of multiple speakers based on extracting audio and video features from the data, some embodiments have the ability to improve the field of computing.

[0018] As previously mentioned, target speech separation extracts the speech of interest from an observed speech mixture. With the advent of the deep learning era, most existing supervised methods are based on spectrogram masks, where the weights (masks) of the target speaker at each time-frequency (T-F) window of the mixed spectrogram are estimated. As a result, the product between the mixed spectrogram and the predicted mask is used as the target speech spectrogram. However, these methods only use audio information (referred to as audio-only methods), which is often severely disturbed in complex acoustic environments (e.g., noise and reverberation). Since the acoustic target information can be ambiguous in challenging acoustic environments, other modalities can provide complementary and stable information to increase robustness. Therefore, it can be advantageous to use a general multi-modal framework that aims to extract the speech of the desired speaker in synchronous video and multi-channel audio corrupted by noise and reverberation.

[0019] Various aspects are now described with reference to the drawings. In general, the drawings described below are to be considered in the context of the written description. The drawings described below are not necessarily drawn to scale. Embodiments will be described and explained with additional specificity and detail to enable a complete understanding of the claimed structures and methods. The described embodiments can be combined.

[0020] The example embodiments described below provide a system, method, and computer program that extract and integrate multi-modal disentanglement cues from multi-channel speech mixtures, target speaker's lip movements, and speech recordings to improve performance. Furthermore, to effectively explore and exploit the high correlation among modalities, a multi-modal fusion method based on factorized attention can be used for target speech separation. Specifically, a multi-stream architecture can take as input the multi-channel mixture, mouth images cropped from the synchronous video, and the target speaker's enrollment utterance, and can extract the target speech while suppressing all other interfering signals. By leveraging microphone array-based signal processing techniques, the audio stream can fully exploit the spectral and spatial characteristics of the multi-channel speech signal to obtain a more complete and robust target-related acoustic representation. The video stream can capture the spatio-temporal dynamics of the lip movements and produce lip embeddings. The speaker embedding stream can map the clean reference audio of the target speaker to a feature vector containing speaker-specific information. An aggregation method based on factorized attention can be used to fuse the high-level semantic information of the multiple modalities at the embedding level.

[0021] Reference will now be made to Figure 1 which is a functional block diagram of a networked computer environment showing a target speech separation system 100 (hereinafter "the system") for separating a target speech from multiple speakers. It should be appreciated that Figure 1 The illustration of only one implementation is provided for purposes of clarity and example, and is not meant to implicate any limitation on the scope of environment in which different embodiments can be implemented. Various modifications to the described environments can be made based on design and implementation requirements.

[0022] The system 100 can include a computer 102 and a server computer 114. The computer 102 can communicate with the server computer 114 via a communication network 110 (hereinafter "the network"). The computer 102 can include a processor 104 and a software program 108 stored in a data storage device 106 and capable of interfacing with a user and communicating with the server computer 114. As will be discussed below Figure 4 As discussed below, the computer 102 can include internal components 800A and external components 900A, respectively, and the server computer 114 can include internal components 800B and external components 900B, respectively. The computer 102 can be, for example, a mobile device, a phone, a personal digital assistant, a netbook, a laptop computer, a tablet computer, a desktop computer, or any type of computing device capable of running a program, accessing a network, and accessing a database.

[0023] The server computer 114 can also operate in a cloud computing service model, such as Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (laaS), as discussed below with reference to Figure 5 and Figure 6 The server computer 114 can also be located in a cloud computing deployment model, such as a private cloud, community cloud, public cloud, or hybrid cloud.

[0024] The server computer 114 for separating target speech from multiple speakers can operate a target speech separation program 116 (hereinafter “program”) that can interact with the database 112. The target speech separation program method is explained in more detail below with reference to Figure 3 The target speech separation program method is explained in more detail. In one embodiment, the computer 102 can operate as an input device that includes a user interface, while the program 116 can operate primarily on the server computer 114. In an alternative embodiment, the program 116 can operate primarily on at least one computer 102, while the server computer 114 can be used to process and store data used by the program 116. It should be noted that the program 116 can be a standalone program, or can be integrated into a larger target speech separation program.

[0025] It should be noted, however, that in some cases, processing of the program 116 can be shared between the computer 102 and the server computer 114 in any ratio. In another embodiment, the program 116 can operate on more than one computer, server computer, or some combination of computers and server computers, for example, multiple computers 102 in communication with a single server computer 114 over the network 110. In another embodiment, for example, the program 116 can operate on multiple server computers 114 in communication with multiple client computers over the network 110. Alternatively, the program can operate on a web server in communication with a server and multiple client computers over a network.

[0026] The network 110 can include wired connections, wireless connections, fiber optic connections, or some combination thereof. In general, the network 110 can be any combination of connections and protocols that will support communications between the computer 102 and the server computer 114. The network 110 can include various types of networks, such as local area networks (LANs), wide area networks (WANs), such as the Internet, telecommunications networks, wireless networks, public switched networks, satellite networks, cellular networks (e.g., fifth generation (5G) networks, long term evolution (LTE) networks, third generation (3G) networks, code division multiple access (CDMA) networks, etc.), public land mobile networks (PLMN), metropolitan area networks (MANs), private networks, ad hoc networks, intranets, fiber optic networks, etc., and / or combinations of these or other types of networks.

[0027] Figure 1 The number and arrangement of devices and networks shown is provided as an example. In practice, there can be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or differently arranged devices and / or networks than those shown. Figure 1 There can be more, fewer, or a different set of devices and / or networks than those shown, which can be arranged differently than shown. Furthermore, Figure 1 Two or more of the devices shown can be implemented within a single device, or Figure 1 A single device shown can be implemented as multiple, distributed devices. Additionally or alternatively, a set of devices (e.g., at least one device) of system 100 can perform at least one function that is described as being performed by another set of devices of system 100.

[0028] Reference is now made to Figure 2 which shows a block diagram 200 of a multi-modal target speech separation system. The multi-modal target speech separation system can include a directional feature extractor 202, a lip embedding network 204, a speaker embedding network 206, and a multi-modal separation network 208. The multi-modal speech separation system can receive video data 210 and audio data 212 as input. The multi-modal separation network 208 can output target speech data 214.

[0029] The directional feature extractor 202 can take as input a noisy, multi-channel speech mixture and the direction of the target speaker in order to extract an acoustic embedding. A short-time Fourier transform (STFT) convolutional 1D layer can be used to map the multi-channel mixed waveform to a complex spectrogram. Based on this complex spectrogram, single-channel spectral features (i.e., log power spectrum (LPS)) and multi-channel spatial features (i.e., interaural phase difference (IPD)) can be extracted. In addition to the spectral and spatial features that are independent of the target speaker, directional features can be extracted based on the spatial direction of the target speaker. All features can be concatenated and fed into an audio block, which can include a stack of dilated convolutional layers with exponentially increasing dilation factors. The output of the audio block can include at least one acoustic embedding. At the output of the system, an iSTFT convolutional 1D layer can be used to convert the estimated target speaker complex spectrogram back to a waveform.

[0030] The lip-embeddings network 204 can extract frame-level lip-embeddings from the video data 210. The lip-embeddings network 204 can receive the cropped mouth images of the target speaker as input. The lip-embeddings network 204 can include a spatio-temporal convolutional layer and a ResNet of 18 layers. Supervised information can be formed from the audio domain, which can use the video data 210 to discover at least one cross-domain correlation between the target speech and the lip motion. The lip-embeddings network 204 can utilize at least one video block, each of which can contain several extended temporal convolutional layers with residual connections. The output of the video block can include at least one lip-embedding. Since the resolutions of the video stream and the audio stream can be different, the lip-embeddings can be upsampled by nearest-neighbor interpolation to synchronize with the audio stream.

[0031] The speaker-embeddings network 206 can process the enrollment audio of the target speaker, such as the audio data 212, and can generate utterance-level speaker-embeddings. The speaker-embeddings can be biasing signals that can inform the multi-modal separation network 208 to perform and enhance the separation of the target speaker. A pre-trained speaker model can be introduced to produce the speaker-embeddings to characterize the target speaker. The speaker model can be pre-trained on a speaker verification task using at least one convolutional layer followed by a fully connected layer. The input to the speaker model can be an enrollment utterance of the target speaker, such as saying their own name when joining a teleconference. The speaker-embeddings network 206 can output utterance-level speaker-embeddings.

[0032] The multi-modal separation network 208 can combine the acoustic, lip, and speaker-embeddings and feed them into at least one fusion block that can output a T-F mask for the target speaker. The T-F mask can be used to estimate the target speech waveform. As described above, the three kinds of target information can be derived from a set of media sources, including acoustic embeddings from multi-channel speech, lip-embeddings from video, and speaker-embeddings from an enrollment utterance of the target speaker. To learn an effective target speech extraction from multi-modal information, a factorization layer can be used to quickly adapt to acoustic contexts. In speech recognition, one factor can be a group of speakers or a specific acoustic environment. The factorization layer can use a different set of parameters to process each acoustic category, which can depend on external factors that represent the acoustic conditions. According to at least one embodiment, the acoustic embeddings can be factorized into a set of acoustic subspaces (e.g., a telephone subspace, a speaker subspace), and they can be selectively aggregated with information from the other modalities. The other modalities can also provide information related to the acoustic conditions, such as speech activity interpreted from the opening and closing of the mouth, and target speaker speech characteristics contained in the speaker-embeddings. Specifically, parallel linear transformations The acoustic embedding A is factorized into distinct acoustic subspaces, where H can be the number of subspaces, and the acoustic representation in the h-th subspace at time step t can be expressed as: Where P is the output dimension of the linear transformation. The lip embedding V can also be mapped from a D-dimensional space to an H-dimensional space, where each dimension h can contain bias information corresponding to the h-th acoustic subspace. The mapped lip embedding can be passed to a softmax layer, and an estimated posterior can be generated for each subspace at each time step, calculated as follows:

[0033] The fused audiovisual embedding (AVE) can be obtained by summing the weighted contributions of different acoustic subspaces:

[0034]

[0035] Where σ can be a sigmoid activation function. Acoustic embeddings and speaker embeddings can be similarly combined and computed as follows:

[0036]

[0037] Where Ws can be a weight matrix that can transform the speaker embedding S from the speaker space into the acoustic subspace.

[0038] Now for reference Figure 3 It shows an operation flowchart 300, which illustrates the operation of separating target speech from multiple speakers, performed by a program. This can be achieved using... Figure 1 and Figure 2 To describe Figure 3 As mentioned earlier, the target speech separation program 116 ( Figure 1 It can quickly and effectively utilize audio and video feature data to separate the speech of a target speaker from the speech of multiple other speakers.

[0039] At step 302, video data associated with the plurality of speakers is received. It is understood that any number of speakers can be captured by any number of cameras. In an embodiment, the target speech separation program 116 on server computer 114 ( Figure 1 ) can be accessed via communication network 110 ( Figure 1 From at least one computer 102 ( Figure 1 ) Receive video data 210 ( Figure 2 ).

[0040] At step 304, audio data associated with each of at least one of the speakers is received. This audio data may include registered utterances, such as speakers stating their names while participating in a conference call. In an embodiment, the target speech separation program 116 on server computer 114 ( Figure 1 ) can be accessed via communication network 110 ( Figure 1 From at least one computer 102 ( Figure 1 ) Receive audio data 212 ( Figure 2 Audio data 212 can be transmitted to the speaker embedding network 206. Figure 2 ).

[0041] At step 306, video feature data is extracted from the received video data. This video feature data may include the orientation associated with the camera capturing the at least one speaker, and lip movement image data associated with each of the at least one speaker. In an embodiment, the orientation feature extractor 202 ( Figure 2 ) can be obtained from video data 210 ( Figure 2 Orientation data is extracted from the lip embedding network 204. Figure 2 Lip movement data can be further extracted from video 210.

[0042] At step 308, the target speech is identified from the plurality of speakers based on the received audio data and the extracted video feature data. In an embodiment, a multimodal separation network 208 ( Figure 2 ) can be extracted from the orientation feature extractor 202 ( Figure 2 ) Receive extracted orientation data from lip embedding network 204 ( Figure 2 ) receives extracted lip movement data, and receives data from the speaker embedding network 206 ( Figure 2 ) audio data 212 ( Figure 2 The extracted features. The multimodal separation network 208 can separate the speech of the target speaker and output the target speech data 214. Figure 3 ).

[0043] It should be understood that Figure 4 The illustrations provided are merely illustrative of one implementation and do not imply any limitations on how different embodiments may be implemented. Various modifications may be made to the described environment based on design and implementation requirements.

[0044] Figure 1 This is according to an exemplary embodiment. Figure 4 The block diagram 400 depicts the internal and external components of the computer. It should be understood that... Figure 1The illustrations provided are merely illustrative of one implementation method and do not imply any limitation on the environments in which different embodiments may be implemented. Various modifications may be made to the described environment based on design and implementation requirements.

[0045] Computer 102 ( Figure 1 ) and server computer 114 ( Figure 4 ) can include Figure 1 The corresponding sets of internal components 800A, 800B and external components 900A, 900B are shown. Each set of internal components 800 includes at least one processor 820 connected to at least one bus 826, at least one computer-readable random access memory (RAM) 822 and at least one computer-readable read-only memory (ROM) 824 including at least one operating system 828, and at least one computer-readable tangible storage device 830.

[0046] Processor 820 is implemented in hardware, firmware, or a combination of hardware and software. Processor 820 is a central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), microprocessor, microcontroller, digital signal processor (DSP), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or other type of processing component. In some embodiments, processor 820 includes at least one processor capable of being programmed to perform functions. Bus 826 includes components that allow communication between internal components 800A and 800B.

[0047] At least one operating system 828, and a server computer 114 ( Figure 1 Software program 108 on ) Figure 1 ) and target speech separation program 116 ( Figure 4 All of these are stored on at least one corresponding computer-readable tangible storage device 830, for execution by at least one corresponding processor 820 via at least one corresponding RAM 822 (which typically includes a cache memory). Figure 1 In the illustrated embodiment, each computer-readable tangible storage device 830 is a disk storage device of an internal hard disk drive. Alternatively, each computer-readable tangible storage device 830 is a semiconductor storage device, such as ROM 824, erasable programmable read-only memory (EPROM), flash memory, optical disk, magneto-optical disk, solid-state disk, optical disc (CD), digital versatile optical disc (DVD), floppy disk, cassette tape, magnetic tape, and / or other types of non-volatile computer-readable tangible storage devices capable of storing computer programs and digital information.

[0048] Each set of internal components 800A, 800B also includes a read / write (R / W) drive or interface 832 to read from and write to at least one portable computer readable tangible storage device 936 such as a CD-ROM, DVD, memory stick, magnetic tape, magnetic disk, optical disk or semiconductor storage device. Software programs such as the software programs 108 Figure 1 ) and the target speech separation program 116 Figure 1 ) can be stored on at least one respective portable computer readable tangible storage device 936, read via the respective R / W drive or interface 832 and loaded into the respective hard drive.

[0049] Each set of internal components 800A, 800B also includes a network adapter or interface 836 such as a TCP / IP adapter card, wireless Wi-Fi interface card, or a 3G, 4G or 5G wireless interface card or other wired or wireless communication links. The software programs 108 Figure 1 ) and the target speech separation program 116 Figure 1 ) on the server computer 114 Figure 1 ) can be downloaded to the computer 102 Figure 5 ) and the server computer 114 from an external computer via a network (e.g., the Internet, a local area network or other networks, wide area networks) and the respective network adapter or interface 836. From the network adapter or interface 836, the software programs 108 and the target speech separation program 116 on the server computer 114 are loaded into the respective hard drive. The network can comprise copper wires, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers.

[0050] Each set of external components 900A, 900B can include a computer display monitor 920, a keyboard 930, and a computer mouse 934. External components 900A, 900B can also include touch screens, virtual keyboards, touch pads, pointing devices, and other human interface devices. Each set of internal components 800A, 800B also includes device drivers 840 to interface to computer display monitor 920, keyboard 930 and computer mouse 934. Device drivers 840, R / W drive or interface 832 and network adapter or interface 836 comprise hardware and software (stored in storage device 830 and / or ROM 824).

[0051] It should be understood that, although the present application includes detailed descriptions of cloud computing, embodiments of the present application are not limited to cloud computing environments. Rather, some embodiments can be implemented in any of a variety of other types of computing environments.

[0052] Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a provider of the service. This model can be composed of at least five characteristics, at least three service models, and at least four deployment models.

[0053] The characteristics are as follows:

[0054] On-demand self-service: cloud users can unilaterally provision computing capabilities, such as server time and network storage, as needed automatically without requiring human interaction with the service's provider.

[0055] Broad network access: capabilities are available over a network and accessed through standard mechanisms that can promote heterogeneity among the thin or thick client platforms (e.g., mobile phones, laptops, and PDAs) that can be used to access the capabilities.

[0056] Resource pooling: the provider's computing resources are pooled to serve multiple users, with different physical and virtual resources dynamically assigned and reassigned according to demand. Location independence means that the user typically has no control or knowledge over the exact location of the provided resources but can be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter).

[0057] Rapid elasticity: capabilities can be rapidly and elastically provisioned, in some cases automatically, to quickly scale out and rapidly released to quickly scale in. To the user, the capabilities available for provisioning often appear to be unlimited and can be purchased in any quantity at any time.

[0058] Measured service: cloud systems automatically control and optimize resource use by leveraging a metering capability at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency for both the provider and the user of the service.

[0059] The service models are as follows:

[0060] Software as a Service (SaaS): the functionality provided to the user is the use of the provider's applications, running on cloud infrastructure, through a thin client interface such as a web browser (e.g., web-based e-mail). The user does not manage or control the underlying cloud infrastructure including network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0061] Platform as a Service (PaaS): the functionality provided to the user is the ability to deploy onto the cloud infrastructure user-created or acquired applications created using programming languages and tools supported by the provider. The user does not manage or control the underlying cloud infrastructure including networks, servers, operating systems, or storage, but has control over the deployed applications and possibly application hosting environment configurations.

[0062] Infrastructure as a Service (laaS): the functionality provided to the user is the provision of processing, storage, networks, and other fundamental computing resources where the user is able to deploy and run arbitrary software, which can include operating systems and applications. The user does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, deployed applications, and possibly limited control of select networking components (e.g., host firewalls).

[0063] Deployment models are as follows:

[0064] Private cloud: the cloud infrastructure is operated solely for an organization. It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0065] Community cloud: the cloud infrastructure is shared by several organizations and supports mission-critical enterprise resources. It can be managed by the organizations or a third party and can exist on-premises or off-premises.

[0066] Public cloud: the cloud infrastructure is made available to general public or a large industry group and is owned by an organization selling cloud services.

[0067] Hybrid cloud: the cloud infrastructure is a composition of two or more types of cloud (private, community, or public) that remain unique entities but are bound together, creating the hybrid cloud, to provide data and application portability.

[0068] A cloud computing environment is service oriented, with a focus on statelessness, low coupling, modularity, and semantic interoperability. The core of cloud computing is an infrastructure of interconnected nodes, including resources, organized to deliver virtual hosting.

[0069] Referring to Figure 5 illustrates an exemplary cloud computing environment 500. As shown, cloud computing environment 500 includes one or more cloud computing nodes 10 with which a cloud user can communicate. Cloud computing nodes 10 can communicate with one another. They can be Figure 6 The types of computing devices 54A-N shown are intended to be illustrative only and that cloud computing nodes 10 and cloud computing environment 500 can communicate with any type of computerized devices over any type of network and / or network addressable connection (e.g., using a web browser).

[0070] Referring to Figure 5 illustrates a set of functional abstraction layers provided by cloud computing environment 500 Figure 6 It should be understood that ​ The components, layers, and functions shown are intended to be illustrative only and that embodiments are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0071] Hardware and software layer 60 includes hardware and software components. Examples of hardware components include: mainframes 61; RISC (Reduced Instruction Set Computer) architecture based servers 62; servers 63; blade servers 64; storage devices 65; and networks and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0072] Virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 71; virtual storage 72; virtual networks 73, including virtual private networks; virtual applications and operating systems 74; and virtual clients 75.

[0073] In one example, management layer 80 can provide the functions described below. Resource provisioning 81 provides dynamic procurement of computing resources and other resources that are utilized to perform tasks within the cloud computing environment. Metering and Pricing 82 provide cost tracking as resources are utilized within the cloud computing environment, and billing or invoicing for consumption of these resources. In one example, these resources can include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal 83 allows for access to and task preformation of the cloud computing environment for consumers and system administrators. Service level management 84 provides cloud computing resource allocation and management such that required service levels are met. Service Level Agreement (SLA) planning and fulfillment 85 provide pre-arrangement for, and procurement of, cloud computing resources for which a future requirement is anticipated in accordance with an SLA.

[0074] Workloads layer 90 provides examples of functionality for which the cloud computing environment can be utilized. Examples of workloads and functions which can be provided from this layer include: mapping and navigation 91, software development and lifecycle management 92, virtual classroom education delivery 93, data analytics processing 94, transaction processing 95, and targeted voice separation 96. Targeted voice separation 96 can separate a target voice from multiple speakers.

[0075] Embodiments of the present application also provide a device for separating a target voice from multiple speakers. The device comprises: a first receiving module configured to receive video data associated with the multiple speakers; a second receiving module configured to receive audio data associated with each of the at least one speaker; an extracting module configured to extract video feature data from the received video data; and an identifying module configured to identify the target voice from the multiple speakers based on the received audio data and the extracted video feature data.

[0076] Embodiments of the present application also provide a computer device, comprising at least one processor and at least one memory, the at least one memory having at least one program instruction stored therein, the at least one program instruction being loaded and executed by the at least one processor to implement the method for separating a target voice from multiple speakers as described above.

[0077] Some embodiments can relate to systems, methods, and / or computer readable media at any possible technical detail level of integration. In embodiments of the present application, computer readable media can include non-transitory computer-readable storage media (or medium) on which computer readable program instructions are stored, which instructions, when executed by at least one processor, cause the at least one processor to perform operations as described above for targeted voice separation.

[0078] A computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0079] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0080] Computer-readable program code / instructions used to perform operations can be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, configuration data for integrated circuits, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and procedural programming languages ​​(such as the "C" programming language) or similar programming languages. The computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any type of network (including local area network (LAN) or wide area network (WAN)) or can be connected to an external computer (e.g., via the Internet through an Internet service provider). In some embodiments, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can execute computer-readable program instructions by utilizing state information of computer-readable program instructions to personalize the electronic circuits and thereby perform aspects or operations.

[0081] These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions are executed by the processor of the computer or other programmable data processing apparatus to create means for implementing the function / action specified in at least one block of a flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored comprises an article of writing including instructions for implementing aspects of the function / action specified in at least one block of a flowchart and / or block diagram.

[0082] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device, thereby producing a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the function / action specified in at least one block of a flowchart and / or block diagram.

[0083] The diagrams of the flowcharts and block diagrams in the drawings show the architecture, functionality, and operation of possible implementations of systems, methods and computer-readable media according to various embodiments. In this regard, each block in the flowcharts or block diagrams can represent a module, segment, or portion of instructions, which comprises at least one executable instruction for implementing the specified logical function(s). The

[0084] It will be apparent that systems and / or methods described herein can be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and / or methods is not limiting of the implementations. Thus, the operation and behavior of the systems and / or methods were described herein without reference to specific software code — it being understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0085] No element, act, or instruction used herein should be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles “a” and “an” are intended to include one or more items, and can be used interchangeably with “one or more.” Furthermore, as used herein, the term “set” is intended to include at least one item (e.g., a related item, an unrelated item, a combination of related and unrelated items, or the like), and can be used interchangeably with “at least one.” Where only one item is intended, the term “one” or similar language is used. Also, as used herein, the terms “has,” “have,” “having,” or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based, at least in part, on” unless explicitly stated otherwise.

[0086] The description of the various aspects and embodiments has been presented for purposes of illustration, but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the possible implementations. The terminology used is for the purpose of describing particular embodiments only and is not intended to be limiting of possible implementations. While each of the foregoing embodiments has been described in some detail for clarity’s sake, it will be apparent that other embodiments are possible provided they are within the scope of the disclosure. Each of the foregoing embodiments can be implemented individually, in any combination, or in the alternative (or in a combination of the alternative). Although the foregoing claims list each dependent claim directly from only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the set of claims. Numerous modifications and adaptations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used is for the purpose of describing particular embodiments only and is not intended to be limiting of possible implementations.

Claims

1. A method of separating a target speech from a plurality of speakers, characterized by, The method comprises: receiving video data associated with the plurality of speakers; receiving audio data associated with each of the at least one speaker, the audio data comprising: first audio data synchronized with the video data, and second audio data comprising pre-recorded enrollment utterances associated with each of the at least one speaker; extracting, from the received video data, video feature data comprising direction data corresponding to the at least one speaker; extracting, from the direction data and the first audio data, acoustic embeddings; extracting, from the received video data, frame-level lip embeddings; extracting, from the second audio data, utterance-level speaker embeddings; generating, using the acoustic embeddings, the frame-level lip embeddings, and the speaker embeddings, a time-frequency mask; identifying, based on the time-frequency mask, the target speech from the plurality of speakers.

2. The method of claim 1, wherein, The extracted video feature data further comprises lip motion data corresponding to each of the at least one speaker.

3. The method of claim 2, wherein, The lip motion data comprises pruned images corresponding to a mouth of each of the at least one speaker.

4. The method according to any one of claims 1 to 3, characterized in that, The video feature data is extracted using a convolutional neural network.

5. An apparatus for separating a target speech from a plurality of speakers, characterized by, The apparatus comprises: a first receiving module configured to receive video data associated with the plurality of speakers; a second receiving module configured to receive audio data associated with each of the at least one speaker, the audio data comprising: first audio data synchronized with the video data, and second audio data comprising pre-recorded enrollment utterances associated with each of the at least one speaker; an extracting module configured to extract, from the received video data, video feature data comprising direction data corresponding to the at least one speaker; extract, from the direction data and the first audio data, acoustic embeddings; extract, from the received video data, frame-level lip embeddings; extract, from the second audio data, utterance-level speaker embeddings; and generate, using the acoustic embeddings, the frame-level lip embeddings, and the speaker embeddings, a time-frequency mask; an identifying module configured to identify, based on the time-frequency mask, the target speech from the plurality of speakers.

6. A non-transitory computer readable medium, characterized in that, A computer program stored thereon, which when executed by at least one computer processor, causes the at least one computer processor to perform the method of any one of claims 1-4.

7. A computer device, comprising: An apparatus comprising at least one processor and at least one memory storing at least one program of instruction, the at least one program of instruction loaded and executed by the at least one processor to implement the method of any one of claims 1-4.

Images