Mono and multi-channel source separation with lip motion enhancement
By capturing facial image features of the target speaker in a self-service kiosk and calculating motion vectors, and utilizing a fusion learning model and blind source separation technology, the challenge of audio signal separation in multi-queue self-service kiosks was solved, achieving more efficient audio signal separation and transaction processing.
Patent Information
- Application Number
- CN202110025122.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-01-09
- Filing Date
- 2021-01-08
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-01-08
AI Technical Summary
In high-traffic public transportation settings, the voice recognition technology of automated self-service kiosks faces the problem of sound source separation, especially when multiple queues are conducting transactions simultaneously, it is difficult to effectively separate the target speaker's audio signal from other noise and interference.
By capturing facial image features of the target speaker to calculate motion vectors, and utilizing a fusion learning model and blind source separation technology, mixed source audio signals are separated, enhancing the source separation effect of mono and multi-channel audio signals.
It improves the accuracy of audio signal separation of the target speaker in the self-service kiosk, reduces noise and interference, and improves transaction efficiency and accuracy.
Smart Images

Figure CN113096683B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of speech and graphics processing technology, and in particular to a lip movement enhanced mono and multi-channel source separation technology. Background Art
[0002] In data processing, source separation is a broad and open problem in which signals are received, observed, or otherwise input from multiple sources; in particular, blind source separation ("BSS") describes a source separation scenario in which the number and nature of the different signal sources are not known in advance. It is desirable to separate the signals from each other without information about their sources.
[0003] Depending on the settings where the signal is derived from, a signal source context may be obtained to help determine the source of the signal. For example, at automated public self-service kiosk machines, particularly at locations with high public transportation intake, such as transit stations, theaters, tourist attractions, border crossings, etc., multiple queues may be established at least during busy hours to increase throughput and reduce wait times for transaction processing, such as ticketing, document verification, etc. To further increase throughput and reduce wait times, voice recognition technology may be deployed at the self-service kiosk machines to enable self-service transactions to be performed without the involvement of a human cashier or the need for a user to manually operate a physical interface, thereby improving the efficiency of self-service transactions.
[0004] However, in these high-public-transit settings, transactions are often conducted in close proximity to one another due to the need to manage public space. Therefore, automated self-service kiosks in locations with heavy public transit can present sound source separation issues depending on the specific context; challenges posed by these specific contexts should be addressed in a context-sensitive manner. Summary of the invention
[0005] [0003a] The present application provides a method comprising: calculating a motion vector based on facial features of a target speaker's facial image captured during a focus period; and, at least based on the motion vector, separating audio corresponding to a component source from a mixed source mono audio signal captured during the focus period; wherein the separation of audio from an audio signal is performed by a fused learning model comprising a plurality of learning sub-models.
[0006] [0003c] The present application also provides a system, comprising: one or more processors; and, a memory communicatively connected to the one or more processors, the memory storing computer executable modules that can be executed by the one or more processors, the computer executable modules performing related operations when executed by the one or more processors, the computer executable modules comprising: a facial feature extraction module, the facial feature extraction module being configured to calculate a motion vector based on facial features of a target speaker's facial image captured within a focus period; and, a source separation module, the source separation module being configured to separate audio corresponding to a component source from a mixed source monophonic audio signal captured within the focus period based at least on the motion vector; wherein the source separation module is configured to separate audio from an audio signal by means of a fusion learning model comprising a plurality of learning sub-models.
[0007] [0003d] In addition, the present application also provides a computer-readable storage medium storing computer-readable instructions executable by one or more processors, wherein the computer-readable instructions, when executed by the one or more processors, cause the one or more processors to perform the following operations: calculating a motion vector based on facial features of a target speaker's facial image captured during a focus period; and, based at least on the motion vector, separating audio corresponding to a component source from a mixed source mono audio signal captured during the focus period; wherein the separation of the audio from the audio signal is performed by a fusion learning model comprising a plurality of learning sub-models. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] The detailed description is set forth below with reference to the accompanying drawings. In the drawings, the left-most digit(s) of a reference number identifies the drawing in which the reference number first appears. The use of the same reference numbers in different drawings indicates similar or identical items or features.
[0009] Figure 1 Schematic diagrams showing example still images, video frames and / or facial images captured by an image capture device according to example embodiments of the present disclosure.
[0010] Figure 2A A system architecture configured as a computing source separation system according to an example embodiment of the present disclosure is shown. Figure 2B Shown according to Figure 2A An example of a dedicated processor.
[0011] Figure 3A , Figure 3B and Figure 3C A flowchart of a preprocessing process for inputting an audio signal and a target speaker facial image into a learning model to perform mono-channel source separation or multi-channel source separation calculation according to an example embodiment of the present disclosure is shown.
[0012] Figure 4A and Figure 4B An example of sample pixels extracted from image data of a facial image corresponding to, for example, eye features, nose features, and mouth features is shown.
[0013] Figure 5 Details of a fusion learning model according to an example embodiment of the present disclosure are shown.
[0014] Figure 6 An architectural diagram of a cloud computing system hosting a learning model according to an example embodiment of the present disclosure is shown.
[0015] Figure 7 An example terminal device system for implementing the above-described process and method for implementing source separation techniques for learning models is shown.
[0016] Fig. 8A and 8B An example computing system for implementing the above-described processes and methods for implementing source separation techniques for learning models is shown. DETAILED DESCRIPTION
[0017] The systems and methods discussed herein are directed to implementing source separation techniques, and more particularly, to performing source separation on mixed-source mono and multi-channel audio signals that are enhanced by input lip movement information from captured image data.
[0018] A learning model according to an example embodiment of the present disclosure may be a defined computing algorithm that can be executed by one or more processors of a computing system to perform a task, the task including processing inputs with various parameters and outputting results. A learning model may be, for example, a hierarchical model, such as a deep neural network, which may have a fully connected structure, may have a feedforward structure, such as a convolutional neural network ("CNN"); may have a back-propagation structure, such as a recurrent neural network ("RNN"); may be further constructed to have layers with outputs in opposite directions, such as a bidirectional recurrent neural network ("BRNN"); may be further constructed to have feedback loops within a layer, such as a long short-term memory ("LSTM") neural network; may further combine these structures, such as a bidirectional long short-term memory ("BLSTM") neural network; and may have or may have other architectures suitable for specific task calculations. Tasks may include, for example, classification, clustering, matching, regression, and the like.
[0019] The tasks may provide outputs for performing functions, such as identifying entities in images and / or videos; tracking the motion of entities in video frames in real time; matching identified entities in images and / or videos with other images and / or videos; providing annotations or transcriptions of images, videos, and / or audio in real time; etc. The learning model may configure a computing system to perform computations for the tasks on any one or several types of inputs, such as those described above, where the inputs are typically compressed and features are extracted from them before the computing system performs computations on the extracted features.
[0020] The learning model can be trained to configure a computing system to compute a task with an image as input, such a task providing an output for a function including image classification or computer vision. With respect to the image input, features extracted from the image input (referred to herein as "image features") may include, for example, edge features, corner features, and blob features known to those skilled in the art.
[0021] The learning model can be trained to configure a computing system to compute a task with a facial image as input, such a task providing an output for a function including facial recognition or facial matching. With respect to the facial image input, features extracted from the facial image input (referred to herein as "facial features") may include, for example, eye features, nose features, and mouth features.
[0022] With respect to facial features, in order to understand the example embodiments of the present disclosure, it should be noted that concepts such as “eye features,” “nose features,” and “mouth features” are generally high-level descriptions of an intuition that image features corresponding to eyes, noses, mouths, and other such familiar parts of a human face are extracted somewhere in an input facial image (e.g., the types of image features described above). However, labels such as “eye features,” “nose features,” or “mouth features” are generally not understood by those skilled in the art because they imply or are limited to any particular rule, rationale, principle, criterion, standard, or other technique or solution for identifying features corresponding to eyes, noses, mouths, etc. in facial images. Instead, depending on the nature of the input facial image and the context in which the input facial image was captured, those skilled in the art may consider any technique for corresponding image features to eyes, noses, mouths, etc. to be effective.
[0023] The learning model can be trained to configure a computing system to compute tasks that take a video as input, such as tasks that provide outputs for functions such as video tracking or video annotation. With respect to the video input, features extracted from the video input (referred to herein as "video features") can include, for example, motion features known to those skilled in the art.
[0024] A learning model according to an example embodiment of the present disclosure may be represented by a prefix that represents the type of input employed by the learning model. For example, a BLSTM neural network that takes audio as input may be referred to as an audio BLSTM ("A-BLSTM"); a BLSTM neural network that takes video as input may be referred to as a video BLSTM ("V-BLSTM"); a BLSTM neural network that takes multiple such inputs may be referred to as a fused BLSTM ("F-BLSTM").
[0025] A cloud computing system may provide a collection of servers that host computing resources to provide distributed computing, parallel computing, increased availability of physical or virtual computing resources, and such benefits. A data center implemented in a cloud computing system may host a learning model to provide these benefits to computing applications that use the learning model.
[0026] The cloud computing system can be connected to various terminal devices that collect data to be input into the learning model to train the learning model, and / or are associated with various tasks for computing and the output of the results required to perform those tasks. The terminal devices can be connected to the cloud computing system through the edge nodes of the cloud computing system. The edge node can be any server that provides outbound connections from multiple connection points to other nodes of the cloud computing system, and thus can divide the logical edge of the network of the cloud computing system, without having to be a physical edge. In addition, the edge node can be an edge-based logical node that deploys the non-centralized computing resources of the cloud computing system, such as a cloud server (cloudlet), a fog node, etc.
[0027] The terminal device can calculate various types of tasks involving various types of input data. For example, the terminal device can be an image capture device, such as a camera installed at a self-service kiosk, which collects static images, videos, and facial image data that may originate from a user of the self-service kiosk. According to another example, the terminal device can be an audio capture device, such as a microphone installed on a self-service kiosk, which collects audio that may originate from a user of the self-service kiosk. The self-service kiosk as a whole can be another terminal device that can be connected to an edge node of a cloud computing system, which performs real-time facial tracking (which will be described in further detail later), facial recognition, or real-time voice recognition (which will be described in further detail later) of the user of the self-service kiosk, and sends these types of data in real time to the cloud computing system to perform various tasks related to real-time processing of self-service transactions.
[0028] At specific sites, businesses, or other locations with public traffic where self-service transactions are conducted, multiple self-service kiosks may be installed and operated to conduct self-service transactions and compute back-end tasks that support the self-service transactions. Public traffic through these locations may be diverted based on the location of the self-service kiosks; for example, the self-service kiosks may be arranged in a row adjacent to each other. Such a configuration may allow public traffic to be organized into parallel queues, with each self-service kiosk located at the head of the queue so that the kiosk performs transactions in parallel for the individuals or groups at the head of each queue. If speech recognition technology is deployed in these settings, the individuals at the head of each queue may speak simultaneously, causing the audio capture device at each kiosk to pick up speech signals from multiple queues.
[0029] According to example embodiments of the present disclosure, computing backend tasks to support self-service transactions may include, for example, classification of entities detected in a video by a video CNN ("V-CNN"); signal processing and filtering of recorded audio in, for example, the frequency domain; extracting frames from a video stream and embedding features; and the like.
[0030] According to an example embodiment of the present disclosure, a self-service kiosk as a terminal device can be configured to transmit captured input data (e.g., still images, videos, facial image data, and audio signals) to one or more remote computing hosts, such as a data center hosting high-capacity computing resources, via one or more networks to perform calculations for the above-mentioned back-end tasks using the captured input data. Since the file size of raw input data such as still images, videos, facial image data, and audio signals is large, the input data can be compressed or otherwise pre-processed on the terminal device before being transmitted over one or more networks, and / or the terminal device can be configured to perform edge processing of the back-end tasks described above using the input data. Edge processing may be limited in computing power and scope.
[0031] Tasks and related data can be transferred from terminal devices to one or more remote computing hosts via an interface hosted on a cloud computing system. Terminal devices such as self-service kiosks may be geographically distant from one or more remote computing hosts, resulting in network delays in the transmission of tasks and related data. Compared with remote computing hosts, terminal devices have relatively low computing resource specifications, including processing power, storage, and memory, which also leads to relative differences in computing power between them. Typically, various forms of pre-processing are performed on the data to account for network delays and differences in computing power to improve response time and efficiency in completing tasks.
[0032] For example, large amounts of data captured locally on edge devices may result in intolerable levels of latency if passed through a cloud computing interface for computation by a learning model executed on a remote computing host. In addition, images, videos, or audio in their original, uncompressed format are very inefficient for computations used for classification or embedding because they contain much more data (often redundant for computational purposes) than compressed images, videos, or audio. Therefore, training of learning models according to example embodiments of the present disclosure is typically performed using a training data set, which may be a large number of training data sets that are input or loaded onto a remote computing host via offline input without the need to transmit those data sets over a network. The training data set may include, for example, known, source-separated audio recordings from users of a self-service kiosk during a transaction as described above.
[0033] Furthermore, it is well known that data can be transformed into a frequency representation by a transform algorithm such as the Fourier transform, and audio data can be transformed from a time domain representation (i.e., wave data) to a frequency domain representation by a Fourier-related algorithm such as the Short Time Fourier Transform ("STFT"). The inverse transform ("ISTFT") of the STFT can be further operated to return the frequency domain representation of the audio data to the time domain representation. Typically, the STFT is utilized by dividing the time domain representation of the audio data into time windows of known duration; the frequency domain representation in each time window (hereinafter referred to as the "time-frequency representation") can then be visualized in a spectrogram showing the amplitude STFT of the audio signal. Moreover, in discrete-time applications of the STFT, the audio signal can be divided into discrete frames with some overlap in the time domain, which are individually preprocessed by the STFT.
[0034] After preprocessing by STFT, the time-frequency representation of audio data can be divided into its frequency bands and sub-bands, each of which represents a specific frequency of the audio signal. The frequency domain representation of audio data can be represented by a parameter k representing its specific frequency band or sub-band and a parameter t representing its specific frame.
[0035] Typically, since source separation is still an open problem, source separation can be performed using a time domain representation of an audio signal or a time-frequency representation of an audio signal. According to an exemplary embodiment of the present disclosure, an audio signal captured by a terminal device can be preprocessed by STFT and converted to its time-frequency representation. Such preprocessing can be performed as edge processing on a terminal device, or can be performed by a remote computing host after the audio data is transmitted to a remote computing host. The preprocessing of converting an audio signal into its time-frequency representation can facilitate further preprocessing tasks, such as noise and interference elimination, and can also facilitate source separation techniques as described below.
[0036] According to an example embodiment of the present disclosure, the captured audio signal may be a mono audio signal. In other words, the captured audio signal may be captured from a single microphone. Such an example embodiment of the present disclosure may implement a mono source or mono source separation technique when a mono audio signal is used as input data. A-CNN, A-RNN, A-LSTM, and A-BLSTM may be examples of learning models that may be used to calculate a mono audio signal as this target input data. In particular, the A-LSTM and A-BLSTM learning model structures may utilize feedback loops to retain "memory" and effectively perform source separation on long sequences of audio signals.
[0037] According to an example embodiment of the present disclosure, the captured audio signal may be a multi-channel audio signal. In other words, the captured audio signal may be captured from multiple microphones. For example, a single terminal device (e.g., a single self-service kiosk) may have multiple microphones installed therein to capture audio from the same source, respectively, or multiple microphones of multiple terminal devices placed adjacent to each other may capture audio from the same sound source, respectively. Such an example embodiment of the present disclosure may implement a multi-channel source separation technique when a multi-channel audio signal is used as input data.
[0038] According to the learning model of the computational audio signal of the example embodiment of the present disclosure, a mask to be applied to the time-frequency representation of the audio signal can be determined. The mask can be a matrix or vector in the frequency domain, and when it is applied to the time-frequency representation of the mixed source audio signal by operation (e.g., multiplication operation is performed on each frame), at least one component source mixed source audio signal is generated. In particular, according to the example embodiment of the present disclosure, the target mask can be a mask of a target component source that produces a mixed source audio signal, and the mixed source audio signal is a target speaker. The noise mask can be a mask that at least reduces the constituent audio of the mixed source audio signal that does not correspond to any source or does not correspond to the target source. The time-frequency representation of the audio signal to which the mask is applied can be transformed by an inverse STFT to reconstruct a time domain representation of the audio signal, which time domain representation produces at least one component source of the mixed source audio signal, or produces audio of the mixed source audio signal with the characteristics of reducing noise and interference.
[0039] Alternatively, a learning model of a computational audio signal according to an example embodiment of the present disclosure may apply a voice activity detection ("VAD") system to a time-frequency representation of the audio signal. Typically, the VAD may be a state machine that receives an input frame and outputs a 0 for determining that the frame is non-speech or a 1 for determining that the frame is speech. Typically, the VAD may be applied to a mono audio signal or a multi-channel audio signal, although applying the VAD to a multi-channel audio signal may enhance the performance of the VAD by utilizing comparisons between redundant audio signals of multiple channels. Frames output by the VAD that are determined to be speech may be transformed by an inverse STFT to reconstruct a time domain representation of the audio signal that produces frames that include speech, including the speech of a target speaker.
[0040] The output of the learned model from the computational audio signal can provide classification of time-frequency frames and their frequency bins by labeling them as matching individual speakers, including a target speaker as described above. The speakers can be known speakers or unknown speakers; in any case, speakers labeled as unknown in the output of the learned model can be distinguished from other unknown speakers.
[0041] Signal separation techniques as described above, where calculations are performed only on audio signals, can be further enhanced by providing context about speakers and their speech rhythms to the learning model. According to an example embodiment of the present disclosure, context can be provided by performing calculations on static images, videos, and / or facial image data provided by image capture devices such as cameras of terminal devices of self-service kiosks. Alternatively, static images, videos, and / or facial image data can be provided by image capture devices such as cameras of mobile devices carried by users of self-service kiosks. For example, a user can authenticate to a self-service terminal, log in to a self-service information service terminal, exchange information with a self-service terminal, or otherwise interact with a self-service terminal by, for example, initiating a communication protocol between a mobile device and a self-service terminal over a short distance. Through the operation of the short-range communication protocol, the self-service terminal can request the user to provide static images, videos, and / or facial image data captured by an image capture device of the mobile device.
[0042] According to an example embodiment of the present disclosure, still images, video and / or facial image data may provide context for source separation by extracting facial features from the still images, video and / or facial image data. For example, still images, video and / or facial image data captured by an image capture device in the context of a self-service kiosk transaction may capture a frontal front view of at least one user facing the self-service kiosk. Figure 1 Diagrams showing example still images, video frames, and / or facial images captured by an image capture device according to example embodiments of the present disclosure.
[0043] One or more faces may be detected by computing a learning model of a static image, video frame, and / or facial image based on a variety of facial recognition techniques known in the art; this is not discussed in detail herein. In order to understand the example embodiments of the present disclosure, it should be sufficient to understand that one or more faces may be detected from an image. Depending on the field of view of the image capture device, one or more faces may be detected from different perspectives from the perspective of the image capture device. For example, Figure 1 As shown, the viewing angle ranges from 30° for a face detected on at least the rightmost pixel of a static image, video frame, and / or facial image to 150° for a face detected on at least the leftmost pixel of a static image, video, where 90° is defined as an angle directly in front of the image capture device (assuming the field of view of the image capture device is approximately 120°). A portion of a static image, video frame, and / or facial image containing the face of the target speaker may be extracted as the facial image of the target speaker.
[0044] According to an exemplary embodiment of the present disclosure, for the purpose of specifying a target for source separation calculations, one of the faces detected from a static image, a video frame, and / or a facial image may be designated as a target speaker. In particular, a face detected within a plurality of angle ranges including a frontal viewing angle from an image capture device (e.g., an angle range including a 90° angle) may be designated as a target speaker. Alternatively, a face detected within an angle range closest to a frontal viewing angle from an image capture device may be designated as a target speaker. In addition, a detected face may be designated as a target speaker based on the orientation of the face from the image capture device. For example, a face designated as a target speaker may be a face facing the image capture device. Therefore, based on the angle of the target speaker from the image capture device, the source speaker may be positioned relative to the image capture device and relative to one or more microphones that capture each audio signal.
[0045] According to the multi-channel source example embodiment of the present disclosure, the configuration of multiple microphones relative to each other in space and the information of the position of each microphone relative to the target speaker can be utilized in the calculation of the source separation task through the learning model. In addition, the position of the target speaker can be tracked across multiple state images, video frames and / or facial images, or the position of the target speaker can be tracked across multiple frames of a video stream.
[0046] Through these and similar principles, the localization of the target speaker can be associated with one or more audio signals, thereby enhancing the calculation of source separation through the learning model. Below, a technique for further utilizing the localization of the target speaker in combination with facial feature extraction in a monophonic example embodiment and a multi-channel example embodiment of the present disclosure will be described, thereby providing further enhancement in addition to speaker localization.
[0047] Figure 2A A system architecture of a system 200 configured to perform computational source separation according to an example embodiment of the present disclosure is shown. Figure 2B Shown according to Figure 2A An example of a dedicated processor.
[0048] The system 200 according to an example embodiment of the present disclosure may include one or more general-purpose processors 202 and one or more special-purpose processors 204. The general-purpose processors 202 and the special-purpose processors 204 may be physical, or may be virtualized and / or distributed. The one or more general-purpose processors 202 and the special-purpose processors 204 may execute one or more instructions stored on a computer-readable storage medium as described below to enable the one or more general-purpose processors 202 or the special-purpose processors 204 to perform various functions. The special-purpose processor 204 may be a computing device having hardware or software elements that facilitate the calculation of neural network computing tasks, such as training and reasoning calculations. For example, the special-purpose processor 204 may be an accelerator, such as a neural network processing unit ("NPU"), a graphics processing unit ("GPU"), a tensor processing unit ("TPU"), multiple implementations using field programmable gate arrays ("FPGA") and application-specific integrated circuits ("ASIC"), etc. In order to facilitate the calculation of tasks such as training and reasoning, the special-purpose processor 204 may, for example, implement an engine that calculates mathematical operations such as matrix operations and vector operations.
[0049] The system 200 may also include a system memory 206 that is communicatively connected to the general-purpose processor 202 and the special-purpose processor 204 via a system bus 208. The system memory 206 may be physical, or may be virtualized and / or distributed. Depending on the exact configuration and type of the system 200, the system memory 206 may be volatile, such as RAM, or non-volatile, such as ROM, flash memory, a micro hard drive, a memory card, etc., or some combination thereof.
[0050] The system bus 208 can transfer data between the one or more general-purpose processors 202 and the system memory 206, between the one or more special-purpose processors 204 and the system memory 206, and between the one or more general-purpose processors 202 and the one or more special-purpose processors 204. In addition, the data bus 210 can transfer data between the one or more general-purpose processors 202 and the one or more special-purpose processors 204. The data bus 210, for example, can be a peripheral component interconnect express ("PCIe") connection, a coherent accelerator processor interface ("CAPI") connection, etc.
[0051] Figure 2BAn example of one or more special-purpose processors 204 including any number of cores 212 is shown. The processing power of the one or more special-purpose processors 204 can be distributed among multiple cores 212. Each core 212 can include a local memory 214, which can contain pre-initialized data, such as model weights, or contain data structures, such as constant buffers for batch normalization or quantization, to perform special calculations. Each core 212 can be further configured to execute one or more groups of computer-executable acceleration engine modules 216 pre-initialized on the local memory 218 of the core 212, which can be executed by one or more cores 212, including parallel execution. Through multiple cores 212, arithmetic operations such as matrix multiplication or matrix transposition, function operations such as convolution, special defined operations such as convolution layer pooling, interpolation or region of interest operations, data manipulation, etc. can be executed or accelerated. Each core 212 can further include an instruction sequencer 220, which receives and sorts instructions received from an instruction buffer 222. Some cores (eg, four) 212 may communicate via a data bus 224 (eg, unidirectional), such as a unidirectional ring bus. By sending executable commands via a command processor interface 226, a software driver controlling the operation of each core 212 may control the cores 212 and synchronize their operations.
[0052] Typically, according to an example embodiment of the present disclosure, a time-frequency representation of an audio signal is input into an input layer of a learning model. Therefore, since it is also desirable to minimize data transmission through the system bus 208 or the data bus 210, in order to maximize computational throughput, the audio signal may be processed into its time-frequency representation through STFT before being transmitted through the system bus 208 or the data bus 210. For example, the STFT processing may be performed on a general-purpose processor 202, and the time-frequency representation of the audio signal may be transmitted to one or more dedicated processors 204, on which the learning model calculation may be performed on the time-frequency representation of the audio signal by the processor 204. Optionally, the STFT processing may be performed as described below with reference to Figure 6 The STFT process is performed on the terminal device. The time-frequency representation of the audio signal can be transmitted to a remote computing host of the hosting system 200 through one or more networks, where the learning model calculation can be performed on the frequency representation of the audio signal through a dedicated processor 204. Subsequently, the output of the learning model (such as the mask time-frequency representation of the audio signal or the frame of the time-frequency representation of the audio signal output by the VAD) can be transformed by inverse STFT to reconstruct the time domain representation of the audio signal, and the time domain representation of the audio signal generates the speech spoken by the target speaker. Again, in order to maximize the computational throughput, the output of the learning model can be transmitted back to one or more general-purpose processors 202 for inverse STFT.
[0053] In addition, the static image, video frame and / or facial image or the target speaker facial image extracted therefrom according to the exemplary embodiment of the present disclosure can be transmitted to the dedicated processor 204 via the system bus 208 or the data bus 210, wherein the learning model calculation can be performed on the static image, video frame and / or facial image or the target speaker facial image extracted therefrom to perform facial recognition by the dedicated processor 204. The static image, video frame and / or facial image can be obtained as an uncompressed bitmap image represented in the spatial domain, which can usually be compressed by various encoding algorithms to derive a frequency domain representation of the uncompressed image, on which further preprocessing can be performed to facilitate facial recognition calculation. The static image, video frame and / or facial image or the target speaker facial image extracted therefrom can be compressed on a terminal device such as a self-service kiosk or a mobile device as described above to avoid transmitting the uncompressed static image, video frame and / or facial image, or the target speaker facial image extracted therefrom, to a remote computing host via one or more networks. The still image, video frame and / or facial image or the target speaker facial image extracted therefrom may be further pre-processed on its frequency domain representation, since generally not all image data is relevant to the computation when the image is computed by the learning model. As an example for illustrative purposes, in the processing of data on its frequency domain representation, it is generally known to utilize a high-pass or low-pass filter to reduce or remove low-frequency or high-frequency data, respectively.
[0054] Therefore, an example embodiment of the present disclosure proposes a method for performing calculations by transmitting a time-frequency representation of an audio signal and a frequency-domain representation of at least one target speaker's facial image to one or more dedicated processors.
[0055] Figure 3A , Figure 3B and Figure 3C Flowcharts 300A and 300B of preprocessing procedures for inputting an audio signal and a target speaker's facial image into a learning model to perform single-channel source separation or multi-channel source separation calculations according to an example embodiment of the present disclosure are shown.
[0056] In step 302 of processes 300A and 300B, the user interacts with one or more terminal devices, which include at least one audio capture device and at least one video capture device. For example, the user can perform a transaction in a self-service kiosk, which is equipped with at least one microphone and at least one camera. When interacting with one or more terminal devices, the user can speak various instructions, commands, requests, responses, information and such voice content to at least one audio capture device in one or more terminal devices (such as one or more microphones of the self-service kiosk, any number of microphones of any nearby self-service kiosk, or the microphone of the user's mobile device), and the mobile device interacts with the self-service kiosk by initiating a communication protocol between the mobile device and the self-service kiosk (such as through a short-range communication protocol). In addition, the user can speak the voice content as described above facing one or more image capture devices (such as one or more cameras of the self-service kiosk or the camera of the user's mobile device), and the mobile device interacts with the self-service kiosk by initiating a communication protocol between the mobile device and the self-service kiosk (such as through a short-range communication protocol).
[0057] At least one audio capture device can capture audio as an audio signal during the user interacting with one or more terminal devices. For the sake of brevity, the span of this interactive time is subsequently referred to as a "focus period". The captured audio signal may include the voice content spoken by the user as described above. The captured audio signal may further include the voice of people other than the user. The captured audio signal may further include ambient noise, interference, and sounds from other unrelated sources. According to an example embodiment of the present disclosure, the purpose of the source separation calculation may be to separate the voice content spoken by the user as described above from other such audio captured, and the user is subsequently designated as a "target speaker". The audio of the audio signal can be captured in raw audio format by at least one audio capture device, and can be converted to a compressed format by a transform operation such as STFT, and then the audio signal is stored in its time-frequency representation, and the STFT is used to convert the audio signal from a time domain representation to a frequency domain representation.
[0058] At least one image capture device can capture image data as a still image, video, and / or facial image during a focus period with one or more terminal devices. When the user speaks the voice content as described above, the captured image data may include the user's facial image and facial movements. The captured image data may also include the user's body, faces and bodies other than the user, unrelated objects in the background, and background settings of the surrounding area. Still images, videos, and / or facial images can be captured by at least one image capture device in raw image format, and can be converted to a compressed format by converting the captured image into a coding mode of a codec that represents the frequency domain.
[0059] According to an example embodiment of the present disclosure, the audio capture device and the image capture device can be connected to the computing device through a data bus connection, an optical data connection, or other connections suitable for the transmission of audio signals and image data, respectively, and the computing device can be used to perform audio preprocessing and image preprocessing, respectively, or can be connected to the computing device through a wired or wireless network connection. For example, the computing device can be a personal mobile device, a computing system cluster, a server of a cloud computing system such as an edge server, an embedded computing system of a terminal device such as a self-service kiosk, etc. According to the respective computing capabilities, the example of the computing device according to the example embodiment of the present disclosure can be further used to perform subsequent source separation calculations using an image data set. In an example embodiment of the present disclosure, if a computing device such as a terminal device is not powerful enough to perform source separation calculations, the computing device can also be connected to an additional computing device for performing source separation calculations, such as a remote computing host, through an appropriate connection such as a wired or wireless network connection. In any case, at least each step of process 300 can be performed by a remote computing host described herein.
[0060] However, generally, although the pre-processing steps of the process 300A or 300B before step 306 may be performed at the terminal device, or as described above, the pre-processing steps may be performed by one or more general-purpose processors 202 of the remote computing host, as described above, the processing 300A or 300B after the pre-processing (i.e., any steps after step 304) may be performed by one or more dedicated processors 204 of the remote computing host. It should be understood that after step 304, the captured audio signal and image data and / or the extracted target speaker facial image may be transferred from any other processor (e.g., one or more general-purpose processors 202) on which it was pre-processed to one or more dedicated processors 204; on the dedicated processor 204, features may be extracted from the captured audio signal and image data and / or the target speaker facial image, and calculations may be performed on the extracted features by executing one or more learning models according to the exemplary embodiments of the present disclosure.
[0061] At step 304 of process 300A or 300B, a short time Fourier transform ("STFT") is performed on the captured audio signal.
[0062] Typically, performing STFT on an audio signal causes the audio signal to be divided into discrete frames with some overlap in its time domain representation, so that STFT can be performed on each time domain frame to output a time-frequency representation of the audio signal amplitude of the time domain frame.
[0063] After preprocessing by STFT, the time-frequency representation of the audio signal can be divided into its frequency bands and sub-bands, each of which represents a specific frequency of the audio signal. The frequency domain representation of the audio data can be characterized by a parameter k representing its specific frequency band or sub-band, and by a parameter t representing its specific frame.
[0064] In process 300A, the time-frequency representation of the audio signal can provide a time-frequency representation of a monophonic audio signal. That is, the audio signal occurring during the period of interest is captured from only one point. In process 300B, the time-frequency representation of the audio signal can provide a time-frequency representation of a multi-channel audio signal. That is, the audio signal occurring during the period of interest has been captured from multiple points. In addition, it is also known to configure multiple audio capture devices that capture audio signals at each different point relative to each other in physical space.
[0065] At step 306 of process 300A or 300B, the facial recognition learning model performs multi-face recognition on the captured image data.
[0066] According to an exemplary embodiment of the present disclosure, the captured image data has been pre-processed, converted into its frequency domain representation, and encoded into a compressed format before step 306. In order to understand the exemplary embodiment of the present disclosure, it is not necessary to elaborate on its details.
[0067] The facial recognition learning model may be any trained learning model as known in the art having a parameter set for computing a multi-face recognition technique. The computation of a multi-face recognition task on captured image data according to an example embodiment of the present disclosure may provide for detection of one or more faces in one or more still images or video frames of the captured image data. Depending on the field of view of the image capture device, one or more faces may be detected from different perspectives from the perspective of the image capture device. For example, Figure 1As shown, the viewing angle ranges from 30° for a face detected on at least the rightmost pixel of a static image, video frame, and / or facial image to 150° for a face detected on at least the leftmost pixel of a static image, video frame, and / or facial image, with 90° being defined as directly facing the image capture device (assuming that the field of view of the image capture device is approximately 120°). One of the faces detected therefrom can be designated as a target speaker to designate a target for source separation calculations. In particular, faces detected within a plurality of angle ranges including a frontal viewing angle from the image capture device (e.g., an angle range including a 90° angle) can be designated as a target speaker. Alternatively, a face detected within an angle range closest to a frontal viewing angle from the image capture device can be designated as a target speaker. In addition, a detected face can be designated as a target speaker based on the orientation of the face from the image capture device. For example, a face designated as a target speaker can be a face facing the image capture device.
[0068] Of course, according to the needs of various practical applications, depending on the configuration of the image capture device, the detected face designated as the target speaker does not have to be a face detected in a frontal perspective, but can be, for example, a side perspective or any specific angle between the frontal perspective and the side perspective. In addition, the face designated as the target speaker can be a quarter profile facing the image capture device or a profile facing the image capture device instead of the front face. Such designation can be determined based on knowledge of how one or more terminal devices (e.g., image capture devices) are physically configured relative to the user. However, according to the example embodiments of the present disclosure related to the self-service kiosk as described above, it is expected that in at least about 85% of all cases, a face detected facing the image capture device in a frontal perspective may be correctly designated as the target speaker.
[0069] For purposes of example embodiments of the present disclosure, facial images detected from captured image data need not be identified, i.e., facial images need not be classified to match any tagged speakers in any training dataset or other dataset. The detected facial images need only be distinguished from other facial images in the captured image data. However, the facial images may be further classified as matching tagged speakers based on, for example, customer data previously acquired by an operator of a self-service kiosk as described above, thereby identifying an established customer conducting a transaction in order to authenticate the customer and facilitate real-time transactions.
[0070] At step 308 of process 300A or 300B, the facial recognition learning model selects a target speaker detected from the captured image data.
[0071] A static image or a portion of a video frame containing the detected face of the target speaker may be selected from the captured image data as the target speaker facial image. According to the multiple facial recognition learning models of the example embodiments of the present disclosure, the detected face may be tracked throughout the static image or video stream sequence captured throughout the focus period, thereby tracking the face of the target speaker over a period of time when the target speaker is known or inferred to be interacting with one or more terminal devices. In the subsequent steps of process 300A or 300B, when referring to "facial image", it may refer to one or more facial images of the target speaker in the entire video data captured throughout the focus period.
[0072] Steps 306 and 308 may be performed after step 302, but independently of the order relative to step 304, and one or both of steps 306 and 308 may be performed simultaneously with step 304, before step 304, or after step 304. Steps 306 and 308 may be performed on a processor (e.g., dedicated processor 204) different from the terminal device or one or more processors that perform step 302, or steps 306 and 308 may be performed in parallel on the same terminal device or one or more processors that perform step 302 and one or more different cores that perform step 302.
[0073] In step 310A of process 300A or step 310B of process 300B, facial features are extracted from one or more facial images of the selected target speaker, and a motion vector is calculated based on the extracted features.
[0074] Although both step 310A and step 310B may result in extracting facial features from one or more facial images, step 310A may be performed according to the monaural source separation exemplary embodiment of the present disclosure, and step 310B may be performed according to the multi-channel source separation exemplary embodiment of the present disclosure. Generally, according to steps 310A and 310B, the learning model trained on the labeled facial image dataset may generally locate certain labeled features of a human face to the facial image of the selected target speaker, and extract sample pixels from the image data of the facial image corresponding to each labeled feature. For example, the sample pixels may correspond to eye features, nose features, or mouth features labeled in the training dataset used to train the parameters of the learning model. Figure 4A and Figure 4B An example of sample pixels extracted from image data of a facial image corresponding to, for example, eye features, nose features, and mouth features is shown. In addition, according to an exemplary embodiment of the present disclosure, a training dataset may be manually labeled to specifically mark lip features, so that a learning model trained on the labeled facial image dataset may extract lip features with a certain degree of accuracy. The following will describe with reference to steps 310A and 310B. Figure 4A and 4B for further details.
[0075] According to step 310A, an LLD vector may be further extracted from the mouth features (or in particular the lip features) as follows, the LLD vector being a new type of motion vector describing the lip movement according to an example embodiment of the present disclosure. An optical flow vector ("OPFW vector") may also be extracted from the mouth features (or in particular the lip features) based on the misalignment of sampled features in the image data over time by techniques known in the art, the optical flow vector being a motion vector known in the art for tracking the movement of an object through an image. Among all the extracted features, sample pixels corresponding to the mouth features (or in particular the lip features when the training data set labels lip features in addition to the mouth features) are selected. Assume that the number of selected sample pixels is N. As in Figure 4A As shown in , N=20 (since sample pixels 49 to 68 correspond to lip features).
[0076] like Figure 4A As shown, a local coordinate system can be established. The (0, 0) coordinates of the x-axis and y-axis are established at the upper left end of a group of sample pixels corresponding to the mouth features (especially the lip features), and in the entire focus period, at each time t, each sample pixel can have a first displacement in the x-axis direction and a second displacement in the y-axis direction. For the i-th sample pixel among the N sample pixels, the component of the sample's x-axis displacement can be recorded as V xi , the y-axis displacement component of the pixel is recorded as V yi Therefore, the x-axis displacement of all the sampled pixels in the N sampled pixels corresponding to the mouth features (especially the lip features) can be recorded as V x0 , V x1 , …, V x(N-1) , the y-axis displacement is recorded as V y0 , V y1 , …, V y(N-1) express.
[0077] Based on these displacements, the normalized displacement vector ΔV for each sample pixel can be calculated as follows i (t), where H(t) and W(t) are the height (on the y-axis) and width (on the x-axis) of the target speaker's face in the captured image starting at time t, respectively. The displacement vector can therefore be normalized to account for the fact that facial features on the target speaker's face may appear visually foreshortened due to small differences in orientation over time.
[0078]
[0079] On all normalized displacement vectors of sample pixels corresponding to mouth features (especially lip features), the LLD motion vector describing the facial lip movement of the target speaker at time t can be derived as the following vector:
[0080] ΔV(t)=[ΔV0(t), ΔV1(t),..., ΔV N-1 (t)] T
[0081] Optionally, according to step 310B, according to the following Figure 3C The sub-steps shown may further extract lip motion vectors ("LMV") from the mouth features (especially lip features) and enhance them through the VAD output.
[0082] In step 3101B, sample pixels corresponding to mouth features (or, in particular, lip features, in the case where the training data set labels lip features in addition to mouth features) are selected (i.e., among all extracted features). Let the number of selected sample pixels be K. Figure 4B As shown in , for example, K=20 (because sample pixels 49 to 68 correspond to lip features).
[0083] When capturing an image of a user's mouth, it is expected that the upper lip and lower lip of the user can be captured to move approximately vertically symmetrically with each other. Therefore, it is expected that at least some of the sampled pixels corresponding to mouth features (especially lip features) can be separated into sampled pixels of the upper lip and sampled pixels of the lower lip, wherein pairs of sampled pixels from the upper lip and from the lower lip can be paired vertically symmetrically.
[0084] In step 3102B, a lip motion vector ("LMV") is calculated from the selected sample pixels.
[0085] like Figure 4B As shown, a local coordinate system can be established. The (0, 0) coordinates of the x-axis and the y-axis are established at the upper left end of a group of sample pixels corresponding to the mouth features (especially the lip features). The first motion vector P1 can represent the motion of the pixel corresponding to the upper lip and can be written as P1=[p 1,0 , p 1,1 , ..., p 1,K-2 ] T ; The second motion vector P2 may represent the pixel motion corresponding to the lower lip and may be written as P2 = [p 2,0 , p 2,1 , ..., p 2,K-2 ] T Each p may represent the two-dimensional coordinate of one of the sample pixels corresponding to a mouth feature (or in particular a lip feature), and p 1,k and p2,k Symmetric pairs of sample pixels can be represented.
[0086] Given these two vectors, the distance between them can be expressed as a vector d of Euclidean distances between each pair of sample pixels, each of which is expressed as:
[0087] d k =||p 1,k -p 2,k ||2
[0088] Therefore, the total vector d can be written as:
[0089] d=[d0,d1,…,d K-2 ] T
[0090] Therefore, the motion vector Δmt describing the lip movement in time period t (sent in the focus period T (t=0, 1, 2, ..., T-1)) can be calculated as:
[0091] Δm t =d t -d t-1
[0092] Furthermore, the aspect ratio at each time t can be calculated as:
[0093]
[0094] where d t (v) represents the vertical distance vector of d at half height (compensating the upper and lower lips by K / 2 of the sample pixels respectively), W t Represents the horizontal distance between the leftmost sample pixel and the rightmost sample pixel of the upper and lower lips. LMV at time t, LMV t It is represented as a vector that combines the motion vector and the aspect ratio, as follows:
[0095]
[0096] In step 3103B, the LMV is normalized over time.
[0097] For example, the LMV can be normalized based on the height (on the y-axis) and width (on the x-axis) of the target speaker's face in the captured image starting at time t. Let the height of the detected face be Fh and let the width of the detected face be Fw. The normalized LMV at time t is, It is expressed as follows:
[0098]
[0099] Then, the normalized LMV can be solved The L2 norm of the vector is determined to be (θ LMV )The Euclidean distance to the origin (0, 0) of the vector space is as follows:
[0100]
[0101] In step 3104B, a hidden Markov model ("HMM") outputs the probability of speech based on the normalized LMV.
[0102] Generally, the HMM can be a statistical learning model that observes states and calculates probabilities given a statistical distribution (e.g., a Gaussian distribution). The HMM can calculate the distribution of the normalized Euclidean distance of the LMV and predict the probability of speech accordingly. According to an exemplary embodiment of the present disclosure, the HMM can observe θ LMV,t-1 state, and calculate θ LMV,t The probability of having a specific state. According to an exemplary embodiment of the present disclosure, θ LMV,t The state of θ may indicate, for example, that the upper lip and the lower lip are open, or that the upper lip and the lower lip are closed. LMV,t The prediction of the state of can be related to the probability that speech is detected or not detected.
[0103] In step 3105B, based on the output of the HMM, the VAD outputs a detection result.
[0104] The HMM can further be a component of the VAD as described above. Since the expected VAD output value is 0 or 1, the probability output of the HMM can be mapped to the range of 0 to 1 as follows:
[0105] P vad,t =HMM(θ LMV,t-1 ), P vad,t ∈[0, 1]
[0106] Here, P vad,tv A value close to 1 indicates that VAD detects speech at time t. vad,t A value close to 0 indicates that the VAD did not detect speech at time t. However, it is expected that P vad,t is the probability that there is noise, and according to an example embodiment of the present disclosure, P vad,t can be further post-processed to reduce false positives and improve detection rate. vad,t Smoothing, return P' vad,t :
[0107] P′ vad,t =β·P′ vad,t-1 +(1-β)·P vad,t
[0108] In step 312A of process 300A or step 312B of process 300B, the source separation learning model separates audio corresponding to at least one component source from the captured audio signal based on at least the calculated motion vector.
[0109] As previously discussed, the goal of the calculation of the source separation learning model can be to separate the speech content spoken by the target speaker from other such captured audio, and the context about the target speaker and the target speaker's speaking time can be used to enhance the calculation by the source separation learning model. According to an example embodiment of the present disclosure, the motion vector calculated based on step 310A or 310B can be considered to represent the movement of the target user's lips during the focus period, assuming that the target speaker is speaking the speech content, so the information represented by the motion vector, and the results of additional calculations based on the motion vector (e.g., according to the output of the VAD of step 310B) can be used to enhance the calculation of the source separation learning model.
[0110] Although both step 312A and step 312B can result in output audio corresponding to at least one component source from the captured audio signal, it is desired that the component source be as consistent as possible with the target speaker, and step 312A may be performed according to the mono source separation example embodiment of the present disclosure, and step 312B may be performed according to the multi-channel source separation example embodiment of the present disclosure. Typically, according to steps 312A and 312B, the source separation learning model may be a learning model trained on an audio signal containing labeled speech content. However, the techniques performed by the learning model may vary, as described in more detail below.
[0111] According to step 312A, the source separation learning model outputs a target mask and a noise mask. As described above, the mask can be a matrix or vector in the frequency domain, and when it is applied to the time-frequency representation of the mixed source audio signal by an operation (e.g., a multiplication operation for each frame), the mask or vector will produce a component source of at least one captured audio signal. In particular, the target mask can be a mask of a target component source that produces a mixed source monophonic audio signal, and according to an example embodiment of the present disclosure, the mixed source monophonic audio signal is a target speaker. The noise mask can be a mask that at least reduces the constituent audio of the mixed source monophonic audio signal that does not correspond to any source or does not correspond to the target source. The time-frequency representation of the audio signal to which the mask is applied can then be transformed by an inverse STFT to reconstruct a time domain representation of the audio signal, which time domain representation produces the sound content spoken by the target speaker.
[0112] According to step 312A, the source separation learning model can be a fusion learning model that combines multiple learning sub-models, where the outputs of some learning models are connected to the inputs of other learning models. The architecture of this fusion learning model is Figure 3A It is briefly shown in Figure 5 Details of a fusion learning model 500 according to an example embodiment of the present disclosure are shown.
[0113] In general, a learning sub-model according to example embodiments of the present disclosure may have any number of layers.
[0114] like Figure 5 As shown, the fusion learning model 500 accepts at least three inputs: LLD vector 502 extracted according to step 310A; OPFW vector 504 extracted according to step 310A; and monophonic audio signal 506 (carrying noise and interference) captured as described above. In the following, B represents the batch size of any batch of these types of data; T represents the duration of the training sample from the training data set; and F represents the frequency band of the time-frequency representation of the audio signal. Therefore, the LLD vector 502 can be written as Γ (B) .
[0115] Γ (B) ∈R (B×T×(K-2))
[0116] Wherein, K is the number of sample pixels corresponding to the mouth features (particularly the lip features) as described above.
[0117] The OPFW vector 504 may be written as (B) .
[0118] O (B) ∈R (B×T×W×H×2)
[0119] where H and W are the height (on the y-axis) and width (on the x-axis) of the target speaker's face in a captured image, respectively.
[0120] The mono audio signal 506 can be written as Y (B) .
[0121] Furthermore, the clean audio signal derived from smoothing the noisy audio signal can be expressed as X (B) , and the noise and interference to be removed can be expressed as N (B) .
[0122] Y (B) , X (B) , N (B) ∈C (B×T×F)
[0123] As described above, step 312A may enable the source separation learning model to output a target mask and a noise mask which may be two-dimensional matrices, respectively. The target mask may be represented as The noise mask can be expressed as
[0124]
[0125] like Figure 5 As shown, first, o (B) 504 is input into V-CNN 508, and Y (B) 506 is input into A-BLSTM510.
[0126] As is generally known in the art, coarse-to-fine feature recognition typically utilizes a learning model such as V-CNN 508 to identify fine features from coarse features extracted from image data during initial scanning and image classification, which can be used for more fine-grained feature classification calculations. Thus, V-CNN 508 can calculate (B) 504 from O (B) 504. Identify a finer motion vector from among the motion vectors included in.
[0127] As is generally known in the art, the BLSTM learning model can utilize contextual information from a later time audio signal to perform calculations on an earlier audio signal, and vice versa, by processing the input signal forward and backward. Therefore, although the output of the A-BLSTM 510 is not connected to the final output of the fusion learning model, the A-BLSTM 510 can perform an output intermediate result calculation that partially but not completely implements the target mask and the noise mask.
[0128] The motion vector output by V-CNN 508 can be compared with Γ (B) 502 is fused to generate features including, for example, motion vectors output by V-CNN 508 and Γ (B) The first-order fused motion vector of the two features (ie, the features of the LDD lip motion vector) of 502. The resulting first-order fused motion vector may contain information about the motion of the target speaker's lips over time, with improved granularity in terms of motion.
[0129] like Figure 5 As shown, the first-order fused motion vector is then input into the V-BLSTM 512.
[0130] Similar to the A-BLSTM 510 processing described above, the V-BLSTM 512 can utilize information from later time motion vectors to perform calculations on earlier time motion vectors, and vice versa. Therefore, although the output of the V-BLSTM 512 is not connected to the final output of the fusion learning model, the V-BLSTM 512 can perform calculations that output intermediate results, such as a first-order fused motion vector, in which the motion features over a period of time can be more consistent based on the context between the motion features at an earlier time and the motion vectors at a later time.
[0131] like Figure 5 As shown, the audio signal output by A-BLSTM 510 can then be fused with the first-order fused motion vector to generate, for example, a motion vector feature output by V-CNN 508, Γ (B) The second-order fused motion and audio vector of the features of 502 (i.e., the features of the LDD lip motion vector) and the features of the audio signal output by A-BLSTM 510. The resulting second-order fused motion and audio vector may contain information about the movement of the target speaker's lips over time and the corresponding audio signal captured at the same time.
[0132] like Figure 5 As shown, next, the second-order fused motion and audio vectors are input into F-BLSTM 514.
[0133] Unlike the A-BLSTM 510 and V-BLSTM 512 processing described above, the F-BLSTM 514 can utilize information about the motion of the target speaker's lips over time to enhance the source separation calculation. The fusion of these two categories of information in the same feature vector (indexed by time) can enable the F-BLSTM 514 to receive the second-order fused motion and audio vector to access the target speaker's lip motion features and the mixed source audio features at the same time t, so that the F-BLSTM 514 can determine, for example, whether the target speaker's lips are likely or unlikely to speak at time t; This determination helps determine whether it is possible to find the target speaker's voice in the audio signal at the same time t, thereby facilitating source separation calculations according to example embodiments of the present disclosure.
[0134] The source separation calculation performed by the F-BLSTM 514 can be based on the output of minimizing the cost function J calculated by performing gradient descent. As is known in the art, the cost function can be an arbitrarily defined function based on the output of the learning model, where the output of the cost function represents the degree of error in the output. Therefore, for the F-BLSTM 514 that outputs the target mask and the noise mask, the cost function can be defined to produce a larger output for the target mask that produces a target speech content spoken by the target speaker with a lower accuracy, and can also be defined to produce a larger output for the morning mask that produces an audio signal with more noise and residual interference.
[0135] According to an exemplary embodiment of the present disclosure, the gradient descent calculation may seek to minimize the output of the cost function J of the target mask and the noise mask simultaneously. Therefore, the cost function J according to an exemplary embodiment of the present disclosure may be, for example, as follows:
[0136]
[0137] Among them, ∠·, |·| and ||·|| F Respectively represent the phase of the signal; signal amplitude; Frobenius norm in vector space (one-dimensional matrix norm), ⊙ represents the dot product operation between two-dimensional matrices. Applying the Frobenius norm to the cost function can normalize the cost function for each batch size B, training sample duration T, and frequency band F.
[0138] The target mask of the F-BLSTM output coefficients can be 516 and noise mask 518 can be applied to mono audio signal Y (B) 506, to separate the speech of the target speaker and smooth the noisy audio signal by the following operations:
[0139]
[0140]
[0141] According to Figure 3A Step 314A shown may feed back the difference between the error output by the cost function J from the output of step 312A and the error output by the cost function J from the labeled training data to the fusion learning model to adjust the parameters until the error is minimized by gradient descent.
[0142] Subsequently, after minimizing the cost function, the target mask and noise mask output by the fusion learning model can be applied to the monophonic audio signal, and the resulting product can then be transformed by the inverse STFT to reconstruct a time domain representation of the audio signal that produces the sound content spoken by the target speaker.
[0143] Alternatively, according to step 312B, the blind source separation ("BSS") learning model outputs an unmixing matrix. Similar to the mask described above, the unmixing matrix W(k, t) can be a matrix in the frequency domain that, when applied to the time-frequency representation of the mixed source multi-channel audio signal by an operation (a multiplication operation for each frame), produces a target component source of the mixed source multi-channel audio signal, which, according to an example embodiment of the present disclosure, is the target speaker. The time-frequency representation of the audio signal to which the mask is applied can then be transformed by an inverse STFT to reconstruct a time domain representation of the audio signal that produces the sound content spoken by the target speaker.
[0144] In other words:
[0145] Y(k, t) = W(k, t)X(k, t)
[0146] Where Y(k, t) is the post-source separated audio signal and X(k, t) is the multi-channel audio signal captured in time-frequency representation.
[0147] According to an example embodiment of the present disclosure, the VAD as described in step 3105B above can be used for supervised BSS calculation through a BSS learning model, where the output of the VAD is enhanced by an LMV-based HMM (hereinafter referred to as "lip movement VAD" for simplicity). The supervised BSS calculation based on the independent vector analysis ("IVA") technology may be a technology known to those skilled in the art. However, according to an example embodiment of the present disclosure, as described below, the equations for supervised calculation can be used based on the VAD output as supervisory data. Therefore, the supervised BSS calculation algorithm can be modified for the purpose of the example embodiment of the present disclosure so as to replace the VAD output as described above with the supervisory data.
[0148] For purposes of example embodiments of the present disclosure, X(k, t) may be described as:
[0149] X(k,t)=H(k,t)S(k,t)+N(k,t)
[0150] Wherein, M further refers to the number of channels of the multi-channel audio signal X(k, t); S(k, t) is composed of the N component source signals included in X(k, t); H(k, t) is an M x N mixing matrix, which, when multiplied by S(k, t), obtains a mixing source matrix representing the multi-source signal; N(k, t) is a noise signal also composed of M channels.
[0151] The mixing matrix should be understood as a mathematical structure, which can represent the inverse of the unmixing matrix W(k, t).
[0152] Furthermore, in the statement Y(k, t)=W(k, t)X(k, t), Y(k, t) may include one or more separate component signals, where the nth component signal is labeled Y n And the audio signal of the mth channel constituting the multi-channel audio signal X(k, t) is marked as X m (k, t). Given the estimated unmixing matrix W(k, t), W may have M columns and N rows, where based on m and n, the element W nm (k, t) may be located at the mth column and the nth row.
[0153] The supervised BSS calculation algorithm can be performed through the following iterative steps:
[0154] In the first iteration:
[0155] Y(k, t) = W(k, t)X(k, t)
[0156] In the second iteration:
[0157] ΔW nm (k, t) = (I nm -φ(k,t)Y n (k,t)Y m (k, t) * )W nm (k, t)
[0158] In the third iteration:
[0159] W(k,t+1)=W(k,t)+ηΔW(k,t)
[0160] The supervisory BSS calculation algorithm requires supervisory data φ(k, t). According to an exemplary embodiment of the present disclosure, the smoothed VAD output P derived above in step 3105B can be obtained by the following steps: vad,t Derive φ(k, t):
[0161]
[0162] γ is the contribution coefficient, which determines the extent to which the VAD output contributes to the supervisory BSS calculation. Assume that the nth component signal Y n (k, t) takes the target speaker as the source; then, the VAD output may be related to the BSS calculation, and γ may have a non-zero value. However, suppose the jth component signal Y j (k, t), where j≠n, has noise or interference as its source; then, the VAD output may be irrelevant to the supervisory BSS calculation and the value of γ may be zero.
[0163] Subsequently, after the supervised BSS computation iterations are completed, the unmixing matrix W can be applied to the multi-channel audio signal X to derive a separated component audio signal Y including a component audio signal Yn corresponding to the target speaker. Yn can then be transformed by an inverse STFT to reconstruct a time domain representation of the audio signal that produces the sound content spoken by the target speaker.
[0164] Speech recognition techniques known to those skilled in the art may be performed on the speech content spoken by the target speaker, and the recognition of the speech content may generate information that facilitates the execution of user transactions performed on a terminal device (e.g., a self-service kiosk). In order to understand the exemplary embodiments of the present disclosure, the details of speech recognition techniques known to those skilled in the art need not be described in detail here.
[0165] Figure 6An architectural diagram of a cloud computing system 600 hosting an interface to one or more remote computing hosts according to an example embodiment of the present disclosure is shown. As described above, while examples of computing devices according to example embodiments of the present disclosure may be operated to perform subsequent training and reasoning calculations using image data sets, in other cases, other computing devices communicating via a network connection may be operated to perform training and reasoning calculations. Thus, the figure illustrates some possible architectural embodiments of computing devices as described above.
[0166] The cloud computing system 600 may be implemented on a network 602 of physical or virtual server nodes 604(1), 604(2), ..., 604(N) (where any unspecified server node may be referred to as server node 604) connected via a physical or virtual network. In addition, the network 602 terminates at physical or virtual edge nodes 606(1), 606(2), ..., 606(N) (where any unspecified edge node may be referred to as edge node 606) located at the physical and / or logical edge of the network 602. The edge nodes 606(1) to 606(N) may be connected to any number of terminal devices 608(1), 608(2), ..., 608(N) (where any unspecified terminal device may be referred to as terminal device 608). The terminal device 608, for example as terminal device 608(1), can collect collected data from any number of sensors 610(1), 610(2), ..., 610(N), where the sensor can be, for example, an audio capture device operable to collect audio signals or an image capture device operable to collect image data (e.g., still images, video frames, etc.).
[0167] As described in an example embodiment of the present disclosure, a learning model 614 implemented on a remote computing host accessed through an interface of a cloud computing system 600 may be stored on a physical or virtual memory 616 of a data center 618 ("data center memory") and may be loaded into a physical or virtual memory 620 of a data center 618 ("data center memory") to enable one or more physical or virtual processors 622 of a data center 618 ("data center processor(s)") to perform computations using the learning model 614 to compute input data associated with a computational task as described herein. The one or more data center processors 622 may be a dedicated computing device that facilitates computational neural network computational tasks. For example, the one or more data center processors 622 may be one or more dedicated processors 204 as described above, including one or more accelerators such as a neural network processing unit ("NPU"), a graphics processing unit ("GPU"), a tensor processing unit ("TPU"), and the like.
[0168] According to an example embodiment of the present disclosure, the audio signal and / or still image, video and / or facial image captured at the terminal device 608 may be loaded into the physical or virtual memory 624 of the terminal device 608 ("terminal memory"), so that one or more physical or virtual processors 626 ("one or more terminal processors") of the terminal device 608 perform the pre-processing operations as described above. Alternatively, the audio signal and / or still image, video and / or facial image captured at the terminal device 608 may be transmitted to any one of the edge nodes 606 and loaded into the physical or virtual memory 628 ("edge memory") of the edge node 606, so that one or more physical or virtual processors 630 ("one or more edge processors") of the edge node 606 perform the pre-processing operations as described above. Optionally, the audio signal and / or still image, video and / or facial image captured at the terminal device 608 can be transmitted to any server node 604 and loaded into the terminal memory 624 of the cloud computing system 600 ("cloud memory") so that one or more terminal processors 626 of the cloud computing system 600 ("one or more cloud processors") perform the preprocessing operations as described above.
[0169] According to an embodiment of the present disclosure, audio signals and / or static images, videos and / or facial images may be transmitted to one or more remote computing hosts and utilized by one or more data center processors 622 to perform calculations based on the learning model 614. According to an example embodiment of the present disclosure, implementation process 300A, the learning model may be a fusion learning model in which multiple learning sub-models are combined, wherein the outputs of some learning models are connected to the inputs of other learning models. The learning sub-models may be executed by different processors of the data center processor 622, or may be executed by the same processor of the data center processor 622 on different cores or different threads, and each learning sub-model may perform calculations simultaneously with respect to other learning sub-models.
[0170] Figure 7 An exemplary terminal device system 700 for implementing the above-described processes and methods for implementing source separation techniques for learning models is shown.
[0171] The techniques and mechanisms described herein can be implemented by multiple instances of the terminal device system 700 and by any other computing device, system, and / or environment. As described above, the terminal device system 700 can be a terminal device or a computing system in which multiple terminal devices are installed: the terminal device can be an image capture device, such as a camera installed in a self-service kiosk, which collects static images, videos, and facial image data that may come from a user of the self-service kiosk. According to another example, the terminal device can be an audio capture device, such as a microphone installed on a self-service kiosk, which collects audio that may originate from a user of the self-service kiosk. The self-service station as a whole can be the terminal device system 700. Figure 7 The system 700 shown in is only an example of a system and is not intended to imply any limitation on the scope of use or functionality of any computing device used to perform the above-mentioned processes and / or flows. Other well-known computing devices, systems, environments and / or configurations suitable for this embodiment include, but are not limited to, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, game consoles, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, implementations using field programmable gate arrays ("FPGAs"), and application-specific integrated circuits ("ASICs"), etc.
[0172] The system 700 may include one or more processors 702 and a system memory 704 communicatively connected to the one or more processors 702. The one or more processors 702 and the system memory 704 may be physical, or may be virtual and / or distributed. The one or more processors 702 may execute one or more modules and / or processes to enable the one or more processors 702 to perform various functions. In an embodiment, the one or more processors 702 may include a central processing unit ("CPU"), a GPU, an NPU, a TPU, any combination thereof, or other processing units or components known in the art. In addition, each of the one or more processors 702 may have its own local memory, which may also store program modules, program data, and / or one or more operating systems.
[0173] System memory 704 may be volatile, such as RAM, or non-volatile, such as ROM, flash memory, micro hard drive, memory card, etc., or some combination thereof, depending on the exact configuration and type of system 700. System memory 704 may include one or more computer executable modules 706 that may be executed by one or more processors 702. Module 706 may be hosted on a network as a service for a data processing platform, which may be implemented on a separate system from system 500.
[0174] Module 706 may include, but is not limited to, an audio data capture module 708 , an image data capture module 710 , one or more pre-processing modules 712 (including a Fourier transform submodule 714 ), and a data transmission module 716 .
[0175] The audio data capture module 708 may be configured to capture audio as an audio signal during the user's interaction with one or more terminal devices, as described above with reference to Figure 3A and 3B described.
[0176] The image data capture module 710 may be configured to decode the compressed image into a frequency domain representation of the image data set, as described above with reference to references 3A and 3B.
[0177] The one or more pre-processing modules 712 may be configured to perform pre-processing operations on the captured audio signal, including at least a Fourier transform submodule 714 performing a STFT on the captured audio signal, as described above with reference to Figure 3A and 3B described.
[0178] The data transmission module 716 may be configured to transmit the captured audio signal in its time-frequency representation to one or more dedicated processors, as described above with reference to Figure 2A and 2B described.
[0179] The system 700 may additionally include an input / output ("I / O") interface 740 and a communication module 750 that allows the system 700 to communicate with other systems and devices over a network, such as a terminal device including a camera and a microphone as described above. The network may include the Internet, wired media such as a wired network or a direct wired connection, and wireless media such as acoustic, radio frequency ("RF"), infrared, and other wireless media.
[0180] Fig. 8A and 8B An example computing system 800 is shown for implementing the above-described processes and methods for implementing source separation techniques for learning models.
[0181] The techniques and mechanisms described herein can be implemented by multiple instances of system 800 and by any other computing device, system, and / or environment. System 800 can be a distributed system consisting of multiple physically networked computers or network servers, physical or virtual clusters, computing clouds, or other networked computing architectures known to those skilled in the art that provide physical or virtual computing resources. Examples include learning systems, such as those described above with reference to Figure 6The data center described. The system 800 shown in FIG8 is only an example of a system and is not intended to imply any limitation on the scope of use or functionality of any computing device used to perform the above-described processes and / or flows. Other well-known computing devices, systems, environments, and / or configurations suitable for the embodiments include, but are not limited to, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, game consoles, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, implementations using field programmable gate arrays (“FPGAs”), and application-specific integrated circuits (“ASICs”), etc.
[0182] The system 800 may include one or more processors 802 and a system memory 804 communicatively connected to the one or more processors 802. The one or more processors 802 and the system memory 804 may be physical, or may be virtual and / or distributed. The one or more processors 802 may execute one or more modules and / or processes to enable the one or more processors 802 to perform various functions. In an embodiment, the one or more processors 802 may include a central processing unit ("CPU"), a GPU, an NPU, a TPU, any combination thereof, or other processing units or components known in the art. In addition, each processor 802 may have its own local memory, which may also store program modules, program data, and / or one or more operating systems.
[0183] The system memory 804 may be volatile, such as RAM, or non-volatile, such as ROM, flash memory, micro hard drive, memory card, etc., or some combination thereof, depending on the exact configuration and type of system 800. The system memory 804 may include one or more computer executable modules 806 that may be executed by the one or more processors 802. The modules 806 may be hosted on a network as a service for a data processing platform, which may be implemented on a separate system from the system 800.
[0184] Module 806 may include, but is not limited to, a multi-face recognition module 808, a target speaker selection module 810, a facial feature extraction module 812A or 812B, a source separation module 814A or 814B, and optionally a feedback module 816A.
[0185] The multi-face recognition module 808 may be configured to perform multi-face recognition on the captured image data, as described above with reference to Figure 3A and 3B described.
[0186] The target speaker selection module 810 may be configured to select a target speaker detected from the captured image data, as described above with reference to Figure 3A and 3B described.
[0187] The facial feature extraction module 812A or 812B may be configured to extract facial features from one or more facial images of the selected target speaker and calculate motion vectors, as described above with reference to Figure 3A or as described in 3B. Figure 8B As shown, the facial feature extraction module 812B may also include:
[0188] The sample pixel selection submodule 8121B may be configured to select sample pixels corresponding to mouth features or lip features, as described above with reference to FIG. Figure 3B as stated;
[0189] The lip motion vector calculation submodule 8122B may be configured to calculate the LMV from the selected sample pixels, as described above with reference to Figure 3B as stated;
[0190] The LMV normalization submodule 8123B may be configured to normalize the LMV over time, as described above with reference to Figure 3B as stated;
[0191] The speech probability determination submodule 8124B may be configured to output the speech probability by calculation based on the normalized LMV through the HMM, as described above with reference to Figure 3B described. and
[0192] The detection result determination submodule 8125B may be configured to output the detection result based on the HMM output through VAD, as described above with reference to Figure 3B described.
[0193] The source separation module 814A or 814B may be configured to calculate the captured audio signal and further output audio corresponding to at least one component source from the captured audio signal based at least on the calculated motion vector, as described above with reference to Figure 3A and 3B described.
[0194] The feedback module 816A can be configured to feed back the difference between the error output by the cost function J output from the source separation module 812A and the error output by the cost function from the labeled training data output to the fusion learning model for parameter adjustment until the error is minimized by gradient descent, as described above with reference to Figure 2A described.
[0195] The system 800 may additionally include an input / output ("I / O") interface 840 and a communication module 850 that allows the system 800 to communicate with other systems and devices (e.g., the terminal device system 700 described above) over a network. The network may include the Internet, wired media such as a wired network or a direct wired connection, and wireless media such as acoustic, radio frequency ("RF"), infrared, and other wireless media.
[0196] Some or all of the operations of the above methods may be performed by executing computer-readable instructions stored on a computer-readable storage medium as described below. The term "computer-readable instructions" used in the specification and claims includes routines, applications, application modules, program modules, programs, components, data structures, algorithms, etc. Computer-readable instructions may be implemented on a variety of system configurations, including single-processor or multi-processor systems, minicomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based programmable consumer electronics, combinations thereof, and the like.
[0197] Computer-readable storage media may include volatile memory (e.g., random access memory ("RAM")) and / or non-volatile memory (e.g., read-only memory ("ROM"), flash memory, etc.). Computer-readable storage media may also include other removable and / or non-removable memory, including but not limited to flash memory, magnetic storage, optical storage, and / or tape storage, which may provide non-volatile storage of computer-readable instructions, data structures, program modules, etc.
[0198] Non-transitory computer-readable storage media are examples of computer-readable media. Computer-readable media include at least two types of computer-readable media, namely computer-readable storage media and communication media. Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented in any process or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer-readable storage media include, but are not limited to, phase change memory ("PRAM"), static random access memory ("SRAM"), dynamic random access memory ("DRAM"), other types of random access memory ("RAM"), read-only memory ("ROM"), electrically erasable programmable read-only memory ("EEPROM"), flash memory or other storage technology, compact disk read-only memory ("CD-ROM"), digital versatile disk ("DVD") or other optical storage devices, cassettes, tapes, disk storage devices or other magnetic storage devices, or any other non-transmission media that can be used to store information for access by a computing device. In contrast, communication media can be embodied as computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier or other transmission mechanism. As defined herein, computer-readable storage media do not include communications media.
[0199] Computer readable instructions stored on one or more non-transitory computer readable storage media, when executed by one or more processors, can perform the above referenced Figures 1 to 8B The operations described. Generally, computer-readable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement a process.
[0200] Through the above technical solutions, the present disclosure provides a source separation technology, and more specifically, performs source separation on mixed source mono and multi-channel audio signals enhanced by inputting lip movement information from captured image data. These technologies include: selecting a target speaker facial image from multiple facial images captured within a period of interest; calculating a motion vector based on facial features of the target speaker facial image; and separating the audio corresponding to the component source from the mixed source audio signal captured within the period of interest based at least on the motion vector. The mixed source audio signal can be captured from a mono or multi-channel audio capture device. The audio can be separated from the audio signal by a fusion learning model including multiple learning sub-models. The audio can be separated from the audio signal by a blind source separation ("BSS") learning model. Using real-time image data showing the movement of the target speaker's lips to enhance the learning model can provide important context for enhancing the accuracy of source separation. In addition, based on the distance of the audio capture device that can be used for blind source separation from the speaker, the knowledge of the audio capture device configuration in the multi-channel embodiment of the present disclosure can achieve facial delay.
[0201] Example of terms
[0202] A. A method comprising: selecting a target speaker facial image from a plurality of facial images captured over a focus period; calculating a motion vector based on facial features of the target speaker facial image; and separating audio corresponding to a component source from a mixed source audio signal captured during the focus period based at least on the motion vector.
[0203] B. A method according to paragraph A, wherein multiple facial images are detected in a captured image at different viewing angles from the perspective of an image capture device, and a facial image of a target speaker closest to a frontal viewing angle of the image capture device is detected within a certain angle range.
[0204] C. The method of paragraph A, wherein the facial features include mouth features extracted from the target speaker's facial image.
[0205] D. The method of paragraph C, wherein the mouth features include mouth features extracted from the target speaker's facial image by a learning model trained on a facial image dataset labeled for lip features.
[0206] E. The method of paragraph A, wherein the audio signal is captured from an audio capture device comprising a single audio channel.
[0207] F. The method of paragraph A, wherein the audio signal is captured from an audio capture device comprising multiple audio channels.
[0208] G. A method comprising: calculating a motion vector based on facial features of a target speaker's facial image captured over a focus period; separating audio corresponding to a component source from a mixed source mono audio signal captured during the focus period based at least on the motion vector; wherein the separation of the audio from the audio signal is performed by a fused learning model comprising a plurality of learning sub-models.
[0209] H. As described in paragraph G, wherein calculating the motion vector includes calculating at least an LLD vector and an optical flow vector based on facial features.
[0210] I. A method as described in paragraph G, wherein, at each moment in the focus period, the motion vector is further normalized based on the width and height of the target speaker's facial image at the corresponding moment.
[0211] J. The method described in paragraph G, wherein the fusion learning model includes at least one learning sub-model having an input layer receiving audio features as input, and at least one learning sub-model having an input layer receiving image features as input.
[0212] K. The method as described in paragraph J also includes fusing the audio features output by at least one learning sub-model of the fusion learning model with the image features output by at least one learning sub-model of the fusion learning model.
[0213] L. A method as described in paragraph K, wherein the fusion learning model also includes at least one learning sub-model, which has an input layer that receives a vector including fused audio features and image features as input.
[0214] M. The method as described in paragraph L, wherein the fusion learning model outputs a target mask and a noise mask by performing source separation based on fused audio features and video features.
[0215] N. A method comprising: calculating a motion vector based on facial features of a target speaker facial image captured over a focus period; and separating audio corresponding to a component source from a mixed-source multi-channel audio signal captured during the focus period based at least on the motion vector; wherein the separation of the audio from the audio signal is performed by a blind source separation ("BSS") learning model.
[0216] O. A method as described in paragraph N, wherein calculating the motion vector includes calculating at least a lip motion vector ("LMV") based on facial features.
[0217] P. A method as described in paragraph O, wherein calculating the motion vector also includes calculating the probability of speech based on the LMV through a hidden Markov model ("HMM").
[0218] Q. A method as described in paragraph P, wherein the fusion learning model includes at least one learning sub-model having an input layer that receives audio features as input, and a learning sub-model having at least one input layer that receives image features as input.
[0219] R. The method as described in paragraph Q also includes: based on the probability of speech, a voice activation detection ("VAD") system outputs a detection result.
[0220] S. As described in paragraph R, wherein the BSS learning model outputs an unmixing matrix through iterative calculation based on supervised data.
[0221] T. The method of paragraph S, wherein the supervisory data comprises detection results output by a VAD system.
[0222] U. A system, comprising: one or more processors; and a memory communicatively connected to the one or more processors, the memory storing computer executable modules executable by the one or more processors, the computer executable modules performing related operations when executed by the one or more processors, the computer executable modules comprising: a target speaker selection module configured to select a target speaker facial image from a plurality of facial images captured within a focus period; a facial feature extraction module configured to calculate a motion vector based on facial features of the target speaker facial image; and a source separation module configured to separate audio corresponding to a component source from a mixed source audio signal captured within the focus period based at least on the motion vector.
[0223] V. A system as described in paragraph U, wherein the multiple facial images are detected in the captured image at different viewing angles from the perspective of the image capture device, and the target speaker selection module is configured to select a target speaker facial image detected within an angle range closest to a frontal angle from the image capture device.
[0224] W. A system as described in paragraph U, wherein the facial features include mouth features extracted from a facial image of the target speaker.
[0225] X. A system as described in paragraph W, wherein the mouth features include mouth features extracted from a target speaker facial image by a learning model trained on a facial image dataset labeled with lip features.
[0226] Y. The system of paragraph U, wherein the audio signal is captured from an audio capture device comprising a single audio channel.
[0227] Z. The system of paragraph U, wherein the audio signal is captured from an audio capture device comprising multiple audio channels.
[0228] AA. A computer-readable storage medium storing computer-readable instructions executable by one or more processors, wherein the computer-readable instructions, when executed by the one or more processors, cause the one or more processors to perform the following operations, including: selecting a target speaker facial image from a plurality of facial images captured within a focus period; calculating a motion vector based on facial features of the target speaker facial image; and separating audio corresponding to a component source from a mixed source audio signal captured within the focus period based at least on the motion vector.
[0229] BB. The computer-readable storage medium of paragraph AA, wherein the plurality of facial images are detected in the captured image at different viewing angles from the perspective of the image capture device, and the facial image of the target speaker is detected within an angle range of a closest frontal viewing angle from the image capture device.
[0230] CC. The computer-readable storage medium of paragraph AA, wherein the facial features include mouth features extracted from the target speaker facial image.
[0231] DD. A computer-readable storage medium as described in paragraph CC, wherein the mouth features include mouth features extracted from the target speaker facial image by a learning model trained on a facial image dataset with lip feature labels.
[0232] EE. The computer-readable storage medium of paragraph AA, wherein the audio signal is captured from an audio capture device comprising a single audio channel.
[0233] FF. The computer-readable storage medium of paragraph AA, wherein the audio signal is captured from an audio capture device comprising multiple audio channels.
[0234] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.
Claims
1. A method for separating monophonic and multi-channel sound sources with lip movement enhancement, comprising: Calculating a motion vector based on facial features of a target speaker facial image corresponding to a pre-specified target speaker captured during a focus period, wherein the motion vector comprises at least an LLD vector and an optical flow vector, the LLD vector being determined by a normalized displacement vector of a plurality of sample pixels in the target speaker facial image; and separating audio corresponding to component sources from a mixed-source mono audio signal captured during a period of interest based at least on the LLD vectors and the optical flow vectors; The separation of audio from the audio signal is performed by a fusion learning model including multiple learning sub-models.
2. The method according to claim 1, wherein: At each time instant in the focus period, the motion vector is normalized based on the width and height of the target speaker's facial image at the corresponding time instant.
3. The method according to claim 1, wherein the fusion learning model includes at least one learning sub-model having an input layer receiving audio features as input, and at least one learning sub-model having an input layer receiving image features as input.
4. The method according to claim 3 further comprises fusing audio features output by at least one learning sub-model of the fusion learning model with image features output by at least one learning sub-model of the fusion learning model.
5. The method according to claim 4, wherein: The fusion learning model also includes at least one learning sub-model having an input layer, which receives as input a vector including fused audio features and image features. 6 . The method according to claim 5 , wherein the fusion learning model outputs a target mask and a noise mask by performing source separation based on the fused audio features and video features.
7. A mono and multi-channel sound source separation system with lip movement enhancement, comprising: one or more processors; and A memory communicatively connected to the one or more processors, the memory storing computer executable modules executable by the one or more processors, the computer executable modules performing related operations when executed by the one or more processors, the computer executable modules comprising: A facial feature extraction module, the facial feature extraction module being configured to calculate a motion vector based on facial features of a target speaker facial image corresponding to a pre-specified target speaker captured within a focus period, wherein the motion vector comprises at least an LLD vector and an optical flow vector, the LLD vector being determined by a normalized displacement vector of a plurality of sample pixels in the target speaker facial image; and a source separation module configured to separate audio corresponding to component sources from a mixed-source mono audio signal captured during the period of interest based at least on the LLD vector and the optical flow vector; The source separation module is configured to separate the audio from the audio signal through a fusion learning model including multiple learning sub-models.
8. The system according to claim 7, wherein: The facial feature extraction module is further configured to: at each moment of the focus period, normalize the motion vector based on the width and height of the target speaker's facial image at the corresponding moment.
9. The system of claim 7, wherein the fusion learning model comprises at least one learning sub-model having an input layer that receives audio features as input, and at least one learning sub-model having an input layer that receives image features as input.
10. The system according to claim 9, wherein: The source separation module is also configured to fuse the audio features output by at least one learning sub-model of the fusion learning model with the image features output by at least one learning sub-model of the fusion learning model.
11. The system according to claim 10, wherein: The fusion learning model also includes at least one learning sub-model having an input layer, which receives as input a vector including fused audio features and image features.
12. The system according to claim 11, wherein: The source separation module is also configured to output a target mask and a noise mask by performing a fusion learning model of source separation based on the fused audio features and video features.
13. A computer-readable storage medium storing computer-readable instructions executable by one or more processors, the computer-readable instructions, when executed by the one or more processors, causing the one or more processors to perform the following operations: The motion vector is calculated based on facial features of a target speaker facial image corresponding to a pre-specified target speaker captured during a focus period, wherein The motion vector comprises at least an LLD vector and an optical flow vector, wherein the LLD vector is determined by a normalized displacement vector of a plurality of sample pixels in the target speaker's facial image; and separating audio corresponding to component sources from a mixed-source mono audio signal captured during a period of interest based at least on the LLD vectors and the optical flow vectors; The separation of audio from the audio signal is performed by a fusion learning model including multiple learning sub-models.
14. The computer-readable storage medium of claim 13, wherein: The operation further includes: at each moment of the focus period, normalizing the motion vector based on the width and height of the target speaker's facial image at the corresponding moment.
15. The computer-readable storage medium of claim 13, wherein: The fusion learning model includes at least one learning sub-model having an input layer, which receives audio features as input, and at least one learning sub-model having an input layer, which receives image features as input.
16. The computer-readable storage medium of claim 15, wherein: The operation also includes: fusing audio features output by at least one learning sub-model of the fusion learning model with image features output by at least one learning sub-model of the fusion learning model.
17. The computer-readable storage medium of claim 16, wherein: The fusion learning model also includes at least one learning sub-model having an input layer, which receives as input a vector including fused audio features and image features.
