Real-time audio-visual speech enhancement method and system based on target speaker pixel information, terminal and medium

Abstract

The invention discloses a real-time audio-visual speech enhancement method and system based on target speaker pixel information, a terminal and a medium, and the method comprises the steps: collecting a video stream of a target speaker and mixed speech in a current scene, and determining a lip image sequence corresponding to the target speaker; inputting the lip image sequence into a pre-trained visual element recognition network, and outputting an estimated visual element sequence; and inputting the estimated vision element sequence and the mixed voice into a pre-trained target speaker extraction network together, and outputting pure voice of the target speaker. According to the method, audio-visual speech enhancement is realized through two stages of visual element recognition and target speaker extraction, the dependence on large-scale strict synchronous audio-visual data is reduced, and training data is easier to obtain; and in addition, the vision elements are used as cross-modal intermediate representation, so that extra complexity caused by directly processing high-dimensional video features can be avoided, the system calculation amount is reduced, and the real-time deployment feasibility is improved.

Description

A real-time audiovisual speech enhancement method, system, terminal, and medium based on target speaker visual pixel information Technical Field

[0001] This invention relates to the field of speech processing technology, and in particular to a real-time audiovisual speech enhancement method, system, terminal, and medium based on target speaker visual pixel information. Background Technology

[0002] Existing speech enhancement technologies have achieved good results in suppressing environmental noise, but in scenarios where multiple people are speaking at the same time and the target speech and the interfering voices highly overlap, traditional single-modal methods still have difficulty in stably distinguishing between the target speaker and the non-target speaker.

[0003] To improve speaker discrimination capabilities, existing technologies typically incorporate prior knowledge of the target speaker's identity, such as voiceprint embedding. However, these methods generally suffer from two problems: first, voiceprint features are easily affected by changes in the speaker's physiological state, recording equipment, and transmission channel, posing a risk of mismatch; second, when the target speaker and the interfering speaker have similar timbre, relying solely on voiceprint information is insufficient for differentiation.

[0004] In recent years, audiovisual speech enhancement technologies have attempted to incorporate lip visual information to assist speech separation. Current mainstream solutions mostly adopt an end-to-end audiovisual joint modeling approach, directly inputting noisy audio and video sequences and outputting enhanced speech. While effective, this approach typically relies on large-scale, strictly synchronized, and fully labeled audiovisual data, resulting in high training costs and limited generalization. Furthermore, the large number of model parameters and computational demands hinders low-latency deployment on laptops, mobile terminals, or other edge devices.

[0005] Therefore, existing technologies still have shortcomings. Summary of the Invention

[0006] To address the aforementioned deficiencies in existing technologies, this invention provides a real-time audiovisual speech enhancement method, system, terminal, and medium based on target speaker visual pixel information. The technical solution adopted by this invention is as follows: Firstly, this invention provides a real-time audiovisual speech enhancement method based on target speaker visual pixel information. The method includes: acquiring a video stream of a target speaker and mixed speech in the current scene, and determining a lip image sequence corresponding to the target speaker based on the video stream, wherein the mixed speech includes the target speaker's clean speech, interfering human voices, and background noise; inputting the lip image sequence into a pre-trained visual pixel recognition network, and outputting an estimated visual pixel sequence; inputting the estimated visual pixel sequence and the mixed speech together into a pre-trained target speaker extraction network, and outputting the target speaker's clean speech.

[0007] In one implementation, determining the lip image sequence corresponding to the target speaker based on the video stream includes: locating the lip region of the target speaker based on the video stream; and based on the lip region, cropping the lip image of the target speaker according to the time axis and performing preprocessing to obtain a continuous lip image sequence.

[0008] In one implementation, the training process of the visual pixel recognition network includes: acquiring an audiovisual database, obtaining lip image samples, speech samples, and text transcription samples based on the audiovisual database; converting the text transcription samples into a first phoneme sequence based on a phoneme-to-speech conversion module, and obtaining the correspondence between phonemes and the time axis based on a forced alignment module; converting the first phoneme sequence into a first frame aligned visual pixel sequence based on a preset phoneme-visme mapping rule and the correspondence between phonemes and the time axis; inputting the lip image samples into a basic visual pixel recognition network, and training the basic visual pixel recognition network using the first frame aligned visual pixel sequence as a supervisory label, thereby obtaining a trained visual pixel recognition network.

[0009] In one implementation, the estimated speckle sequence and the mixed speech are input together into a pre-trained target speaker extraction network to output the clean speech of the target speaker, including: performing time alignment, encoding, or embedding representation on the estimated speckle sequence to form speckle conditions; and inputting the speckle conditions and the mixed speech together into the pre-trained target speaker extraction network to output the clean speech of the target speaker.

[0010] In one implementation, the training process of the target speaker extraction network includes: acquiring a speech database and a noise database, wherein the speech database includes clean speech samples of the training object and corresponding text transcription samples, and the noise database includes background noise samples; converting the text transcription samples of the clean speech samples into a second phoneme sequence, and based on a preset phoneme-visual mapping rule and the correspondence between phonemes and the time axis, converting the second phoneme sequence into a second frame-aligned visual sequence; mixing the clean speech samples, the background noise samples, and preset interfering human voice samples according to a set signal-to-noise ratio, and performing acoustic simulation to generate a mixed speech sample; inputting the mixed speech sample and the second frame-aligned visual sequence together into the basic target speaker extraction network, and training with the clean speech samples as the supervised target to obtain the trained target speaker extraction network.

[0011] In one implementation, the training process of the target speaker extraction network further includes applying data perturbation during the generation of the second frame aligned pixel sequence, the data perturbation including any one or more of the following: random pixel replacement, pixel insertion, pixel deletion, pixel masking, or time jitter.

[0012] In one implementation, the training process of the target speaker extraction network further includes: using an acoustic simulation strategy to perform data augmentation on the speech database and noise database to generate diverse training samples; the acoustic simulation strategy includes any one or more of the following: changing the number of speakers interfering with the speech, changing the type of background noise, changing the signal-to-noise ratio, and changing the reverberation intensity.

[0013] Secondly, embodiments of the present invention also provide a real-time audiovisual speech enhancement system based on target speaker visual pixel information. The system is used to implement the steps of the real-time audiovisual speech enhancement method based on target speaker visual pixel information described in any of the above solutions. The system includes: an image and speech acquisition module, used to acquire a video stream of the target speaker and mixed speech in the current scene, and determine a lip image sequence corresponding to the target speaker based on the video stream, wherein the mixed speech includes the target speaker's clean speech, interfering human voices, and background noise; a visual pixel sequence estimation module, used to input the lip image sequence into a pre-trained visual pixel recognition network and output an estimated visual pixel sequence; and a clean speech extraction module, used to input the estimated visual pixel sequence and the mixed speech together into a pre-trained target speaker extraction network and output the target speaker's clean speech.

[0014] Thirdly, embodiments of the present invention also provide a terminal, wherein the terminal includes a memory, a processor, and a real-time audiovisual speech enhancement program based on target speaker visual information stored in the memory and executable on the processor. When the processor executes the real-time audiovisual speech enhancement program based on target speaker visual information, it implements the steps of the real-time audiovisual speech enhancement method based on target speaker visual information of any of the above-mentioned schemes.

[0015] Fourthly, embodiments of the present invention also provide a computer-readable storage medium, wherein the computer-readable storage medium stores a real-time audiovisual speech enhancement program based on target speaker visual pixel information, and the real-time audiovisual speech enhancement program based on target speaker visual pixel information implements the steps of the real-time audiovisual speech enhancement method based on target speaker visual pixel information as described in any of the above schemes on the computer-readable storage medium.

[0016] Beneficial Effects: Compared with existing technologies, this invention provides a real-time audiovisual speech enhancement method based on target speaker visual pixel information. First, it acquires a video stream of the target speaker and mixed speech in the current scene. Based on the video stream, it determines the lip image sequence corresponding to the target speaker. The mixed speech includes the target speaker's clean speech, interfering human voices, and background noise. Then, the lip image sequence is input into a pre-trained visual pixel recognition network, which outputs an estimated visual pixel sequence. Finally, the estimated visual pixel sequence and the mixed speech are input together into a pre-trained target speaker extraction network, which outputs the target speaker's clean speech.

[0017] This invention achieves audiovisual speech enhancement through two stages: speckle recognition and target speaker extraction. This reduces the reliance on large-scale, strictly synchronized audiovisual data, making training data easier to obtain. Furthermore, by using specks—discrete, low-dimensional intermediate representations that are strongly correlated with the pronunciation content—as a cross-modal bridge, the direct reliance on high-dimensional video features is reduced. This allows the system to retain the discriminative ability of the visual modality while significantly reducing model complexity, thereby reducing the system's computational load and improving the feasibility of real-time deployment. Attached Figure Description

[0018] Figure 1 is a flowchart of a preferred embodiment of the real-time audiovisual speech enhancement method based on target speaker visual pixel information according to an embodiment of the present invention.

[0019] Figure 2 is a schematic diagram of the overall technical route of the real-time audiovisual speech enhancement method based on the target speaker's visual pixel information according to an embodiment of the present invention.

[0020] Figure 3 is a flowchart of the training process of the visual pixel recognition network in the real-time audiovisual speech enhancement method based on the visual pixel information of the target speaker according to an embodiment of the present invention.

[0021] Figure 4 is a flowchart of the training process of the target speaker extraction network in the real-time audiovisual speech enhancement method based on target speaker visual information according to an embodiment of the present invention.

[0022] Figure 5 is a flowchart of the online inference process of the real-time audiovisual speech enhancement method based on the target speaker's visual pixel information in an embodiment of the present invention.

[0023] Figure 6 is a comparison of the processing effects of the present invention and the traditional end-to-end method on the recorded samples.

[0024] Figure 7 is a block diagram of the real-time audiovisual speech enhancement system based on the target speaker's visual pixel information provided in an embodiment of the present invention.

[0025] Figure 8 is a schematic diagram of the terminal provided in an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0027] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content, operations, or steps, nor does it require execution in the described order. For example, some operations or steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0028] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0029] It should be understood that, in order to clearly describe the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. For example, "first control information" and "second control information" are only used to distinguish different control information and do not limit their order.

[0030] Those skilled in the art will understand that the words "first" and "second" do not limit the quantity or the order of execution, and that the words "first" and "second" do not necessarily imply that they are different.

[0031] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0032] This invention provides a real-time audiovisual speech enhancement method based on target speaker visual pixel information. While fully utilizing visual information to improve target speech extraction capabilities, it reduces reliance on large-scale synchronous audiovisual data and balances model lightweightness, real-time performance, and robustness in real-world scenarios. In specific applications, this embodiment first acquires a video stream of the target speaker and mixed speech in the current scene. Based on the video stream, a lip image sequence corresponding to the target speaker is determined. The mixed speech includes the target speaker's clean speech, interfering human voices, and background noise. Then, the lip image sequence is input into a pre-trained visual pixel recognition network, which outputs an estimated visual pixel sequence. Finally, the estimated visual pixel sequence and the mixed speech are input together into a pre-trained target speaker extraction network, which outputs the target speaker's clean speech.

[0033] This embodiment achieves audiovisual speech enhancement through two stages: speckle recognition and target speaker extraction. This reduces the reliance on large-scale, strictly synchronized audiovisual data, making training data easier to obtain. Furthermore, by using specks—discrete, low-dimensional intermediate representations that are strongly correlated with the pronunciation content—as a cross-modal bridge, the direct reliance on high-dimensional video features is reduced. This allows the system to retain the discriminative ability of the visual modality while significantly reducing model complexity, thereby reducing the system's computational load and improving the feasibility of real-time deployment.

[0034] Specifically, the real-time audiovisual speech enhancement method based on target speaker visual pixel information in this embodiment can be applied to a terminal, such as a computer or other intelligent device. As shown in Figure 1, the real-time audiovisual speech enhancement method based on target speaker visual pixel information in this embodiment includes the following steps: Step S100, acquiring the video stream of the target speaker and the mixed speech in the current scene, and determining the lip image sequence corresponding to the target speaker based on the video stream, wherein the mixed speech includes the target speaker's clean speech, interfering human voices, and background noise.

[0035] This embodiment first acquires a video stream of the target speaker and mixed speech in the current scene. Specifically, a camera is used to capture the target speaker in the current scene to obtain a video stream, and a microphone is used to acquire the sound signal in the current scene. The acquired sound signal includes the target speaker's pure speech, interfering human voices, and background noise, thus obtaining mixed speech. The purpose of this embodiment is to extract the target speaker's pure speech from the mixed speech, thereby achieving audiovisual speech enhancement.

[0036] As shown in Figure 2, this embodiment decouples the overall task into two relatively independent stages: "visual recognition" and "target speaker extraction." The first stage identifies the estimated visual sequence corresponding to the target speaker's lip image sequence. The second stage inputs the mixed speech and the estimated visual sequence output from the first stage into the target speaker extraction network to recover the target speaker's clean speech, thus obtaining clean speech. Based on this, this embodiment first locates the lip region of the target speaker based on the video stream, then extracts the lip images of the target speaker according to the time axis based on the lip region, and performs preprocessing to obtain a continuous lip image sequence. The preprocessing in this embodiment includes frame segmentation and normalization of the audio in the video stream to facilitate obtaining a temporally continuous lip image sequence.

[0037] Step S200: Input the lip image sequence into a pre-trained pixel recognition network and output an estimated pixel sequence.

[0038] After obtaining the lip image sequence, this embodiment inputs the lip image sequence into a pre-trained spectrogram recognition network, thereby extracting an estimated spectrogram sequence strongly correlated with the speech content from the lip image sequence based on the spectrogram recognition network. In this embodiment, a spectrogram is the visual representation of a phoneme, which can be understood as a discrete visual articulatory unit presented through the state of visible articulatory organs such as the lips and jaw. A phoneme is the smallest unit of speech divided according to the natural attributes of speech; from the perspective of phonological quality, it is the smallest unit or the smallest speech segment that constitutes a syllable. Therefore, a spectrogram refers to the state of visible articulatory organs such as the mouth, tongue, and jaw corresponding to a certain phoneme. Compared with the original lip pixel stream, the spectrogram representation is more compact and more suitable as conditional information for subsequent speech extraction networks. Therefore, this embodiment uses spectrograms as a cross-modal intermediate representation, which avoids the additional complexity brought by directly processing high-dimensional video features, thereby reducing the system's computational load and improving the feasibility of real-time deployment.

[0039] Specifically, in this embodiment, the pixel recognition network is first trained, as shown in Figure 3. The core idea is to use text transcription from the audiovisual database to generate frame-aligned pixel sequences, and use the frame-aligned pixel sequences as supervision labels. Then, the lip image sequence is used as input to train the pixel recognition network.

[0040] In practical applications, as shown in Figure 3, this embodiment first acquires an audiovisual database, and based on this database, obtains lip image samples, speech samples, and text transcription samples (as shown in Figure 3, "I think...good"). Then, based on the word-to-phoneme conversion module, the text transcription samples are converted into a first phoneme sequence, and based on the forced alignment module, the correspondence between the audio and the phoneme sequence is obtained, resulting in the correspondence between each phoneme and the time axis. This yields the start and end positions of the first phoneme sequence on the time axis, achieving frame-level temporal positioning of the phonemes, preparing for the subsequent generation of frame-aligned visual pixel labels. The word-to-phoneme conversion module in this embodiment can employ G2P phoneme conversion technology to convert phonemes in text form into speech representation. The forced alignment (Montreal Forced Aligner) module is a tool for accurately time-stamping audio files and their corresponding text, widely used in speech recognition, audio annotation, and speech synthesis.

[0041] Next, based on a preset phoneme-visual mapping rule, this embodiment can convert each phoneme in the first phoneme sequence into its corresponding visual. Then, by combining the correspondence between phonemes and the time axis, time alignment is performed to obtain the first frame aligned visual sequence, thus realizing the conversion from the first phoneme sequence to the first frame aligned visual sequence. This first frame aligned visual sequence reflects the visual label of each frame on the time axis, reflecting the state of the visible articulatory organs such as the mouth, tongue, and jaw in each frame. Finally, this embodiment inputs the lip image samples into a basic visual recognition network and uses the first frame aligned visual sequence as a supervised label to train the basic visual recognition network. The supervised loss between the estimated visual sequence output by the basic visual recognition network and the first frame aligned visual sequence is calculated. The network parameters are optimized through backpropagation of the loss to complete the training, resulting in a trained visual recognition network.

[0042] In one embodiment, the input video frame rate can be set to 25fps, and the pixel recognition network outputs the corresponding pixel label frame by frame, that is, outputs 25 pixel results per second. The pixel categories can be predefined according to the application scenario, for example, setting 14 core pixel categories and adding 1 blank label category to indicate silent or paused states.

[0043] Once the trained visual pixel recognition network is obtained, this embodiment can input the lip image sequence extracted from the video stream of the current target speaker into the visual pixel recognition network. Based on the visual pixel recognition network processing, an estimated visual pixel sequence aligned with time is automatically output.

[0044] Step S300: Input the estimated spectrogram sequence and the mixed speech into the pre-trained target speaker extraction network, and output the clean speech of the target speaker.

[0045] The objective of this stage is to use the estimated spectrogram sequence provided in the previous stage as content cues to extract the target speaker's clean speech from a mixed speech containing the target speaker's clean speech, interfering human voices, and background noise. Unlike traditional methods that rely on speakerprint references, this invention utilizes visual content information about "what the target person is saying" to help distinguish between target and non-target speech.

[0046] Specifically, this embodiment first trains the target speaker extraction network, as shown in Figure 4. This embodiment first acquires a speech database and a noise database. The speech database includes clean speech samples of the training object and corresponding text transcription samples, while the noise database includes background noise samples. This embodiment also introduces one or more interfering human voice samples. Then, the text transcription samples of the clean speech samples are converted into a second phoneme sequence. Based on a preset phoneme-visual mapping rule and the correspondence between phonemes and the time axis, the second phoneme sequence is converted into a second-frame aligned visual sequence. The method for obtaining the second-frame aligned visual sequence in this embodiment is the same as that for obtaining the first-frame aligned visual sequence, both using a phoneme conversion module and a forced alignment module. The resulting second-frame aligned visual sequence is aligned with the time axis of the clean speech.

[0047] Furthermore, since the pixel recognition in the previous stage cannot achieve complete accuracy in actual deployment, this embodiment also applies data perturbation during the generation of the second frame aligned pixel sequence. This perturbation includes any one or more of the following: random pixel replacement, pixel insertion, pixel deletion, pixel masking, or time jitter, to improve the system's tolerance to errors from the previous stage. By actively exposing errors from the previous stage during the training process in this stage, the target speaker extraction network in the second stage no longer assumes that the input pixel sequence is absolutely correct, but instead learns to recover stable and clean speech even when there are local deviations in the pixels. This design effectively alleviates the error accumulation problem commonly found in cascaded systems.

[0048] Next, in this embodiment, the clean speech sample, the background noise sample, and the preset interfering human voice sample are mixed according to a set signal-to-noise ratio, and acoustic effects such as room reverberation can be further superimposed to achieve acoustic simulation, generating a mixed speech sample. Finally, in this embodiment, the mixed speech sample and the second frame aligned pixel sequence are input into the basic target speaker extraction network, and the clean speech sample is used as the supervised target for training, resulting in a trained target speaker extraction network.

[0049] In other implementations, this embodiment may also employ acoustic simulation strategies to augment the speech and noise databases, generating diverse training samples. These acoustic simulation strategies include any one or more of the following: changing the number of speakers interfering with the speech, changing the type of background noise, changing the signal-to-noise ratio, and changing the reverberation intensity. Generating diverse training samples helps improve the robustness and generalization ability of the network model in real-world recording environments.

[0050] In practical applications, after the target speaker extraction network is trained, this embodiment can first perform time alignment, encoding, or embedding representation on the estimated speckle sequence output by the speckle recognition network to form speckle conditions. Then, the speckle conditions, together with the mixed speech collected in the current scene, are input into the pre-trained target speaker extraction network to output the clean speech of the target speaker. The output clean speech is the enhanced target speech, which can be used by subsequent playback, communication, storage, or speech recognition modules.

[0051] Furthermore, this invention also provides an online inference process for the model. After the networks in both stages are trained, the model parameters are fixed and the online inference stage begins. The core idea of ​​the inference stage is to perform only forward propagation: first, an estimated spectrogram sequence is obtained from the lip image sequence, and then the estimated spectrogram sequence guides the separation of mixed speech, thereby outputting the clean, standard speech of the target speaker in real time.

[0052] As shown in Figure 5, the snowflake markers indicate that the parameters of the spectrogram recognition network and the target speaker extraction network remain frozen during inference and are no longer updated. The input consists of a sequence of lip images of the target speaker and mixed speech from the current scene. The lip image sequence first enters the spectrogram recognition network to output an estimated spectrogram sequence. Subsequently, the estimated spectrogram sequence and the mixed speech are input together into the target speaker extraction network, ultimately outputting clean speech. Unlike schemes that rely on reference speech or voiceprint registration, this invention does not require additional reference speech samples from the target speaker during the inference stage. It can form effective guidance solely based on the lip movements of the target speaker in the video stream captured from the current scene, making it more suitable for applications such as real-time conversations, video conferencing, and on-site recording.

[0053] To verify the effectiveness of the present invention, a target speaker extraction test dataset was synthesized based on the FaceStar dataset (a commonly used face recognition dataset) in the embodiments, and the two-stage decoupling method of the present invention was compared with the traditional end-to-end baseline method. SI-SNR (Scale-Invariant Signal-to-Noise Ratio), STOI (Short-Time Intensity Difference), and PESQ (Perceptual Evaluation of Speech Quality) were used as evaluation indicators, and the results are shown in Table 1.

[0054] Table 1

[0055] As can be seen from Table 1, the two-stage decoupling method of the present invention outperforms the end-to-end baseline in all three indicators, indicating that the intermediate representation of visual pixels can effectively improve the target speech extraction performance, and at the same time verify the effectiveness of the decoupling training path.

[0056] Furthermore, after conducting practical inference tests using video and audio recorded on a laptop, it was found that the proposed solution exhibits better generalization in real-world recording scenarios compared to traditional end-to-end methods, as shown in Figure 6. Figure 6 demonstrates stronger suppression of non-target human voices and better separation in areas where target and interfering speech overlap. This indicates that the proposed solution is not only effective on synthetic test sets but also has good application potential under real-world recording conditions.

[0057] In summary, this invention reduces the reliance on large-scale, strictly synchronized audiovisual data by splitting the audiovisual speech enhancement task into two stages: speckle recognition and target speaker extraction, making training data easier to obtain. Furthermore, using specks as cross-modal intermediate representations avoids the additional complexity of directly processing high-dimensional video features, thereby reducing system computation and improving real-time deployment feasibility. In addition, by introducing speckle perturbation and acoustic simulation, the robustness of the model to previous-stage errors, noise variations, and mismatches with real-world scenarios is enhanced. The inference stage also eliminates the need for reference speech of the target speaker or additional voiceprint registration, resulting in a simpler application process.

[0058] Based on the above embodiments, the present invention also provides a real-time audiovisual speech enhancement system based on target speaker visual pixel information. The system is used to implement the steps of the above method embodiments. As shown in FIG7, the system of this embodiment includes: an image and speech acquisition module 10, a visual pixel sequence estimation module 20, and a clean speech extraction module 30. Specifically, the image and speech acquisition module 10 is used to acquire a video stream of the target speaker and mixed speech in the current scene, and determine the lip image sequence corresponding to the target speaker based on the video stream. The mixed speech includes the target speaker's clean speech, interfering human voices, and background noise. The visual pixel sequence estimation module 20 is used to input the lip image sequence into a pre-trained visual pixel recognition network and output an estimated visual pixel sequence. The clean speech extraction module 30 is used to input the estimated visual pixel sequence and the mixed speech together into a pre-trained target speaker extraction network and output the target speaker's clean speech.

[0059] The principles of each module in the real-time audiovisual speech enhancement system based on target speaker visual pixel information in this embodiment are the same as those of each step in the above method embodiment, and will not be elaborated further here.

[0060] Based on the above embodiments, the present invention also provides a terminal, the principle block diagram of which is shown in FIG8. The terminal may include one or more processors 100 (only one is shown in FIG8), a memory 101, and a computer program 102 stored in the memory 101 and executable on one or more processors 100. For example, a real-time audiovisual speech enhancement program based on target speaker visual pixel information. When one or more processors 100 execute the computer program 102, they can implement the various steps in the embodiments of the real-time audiovisual speech enhancement method based on target speaker visual pixel information. Alternatively, when one or more processors 100 execute the computer program 102, they can implement the functions of each module / unit in the embodiments of the real-time audiovisual speech enhancement system based on target speaker visual pixel information, which is not limited here.

[0061] In one embodiment, the processor 100 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0062] In one embodiment, memory 101 can be an internal storage unit of the terminal, such as a hard disk or RAM. Memory 101 can also be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SM), secure digital card (SD), flash card, etc. Furthermore, memory 101 can include both internal and external storage units. Memory 101 is used to store computer programs and other programs and data required by the terminal. Memory 101 can also be used to temporarily store data that has been output or will be output.

[0063] Those skilled in the art will understand that the principle block diagram shown in FIG8 is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. A specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0064] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), direct memory bus RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time audiovisual speech enhancement method based on target speaker visual pixel information, characterized in that, The method includes: acquiring a video stream of a target speaker and mixed speech in the current scene, and determining a lip image sequence corresponding to the target speaker based on the video stream, wherein the mixed speech includes the target speaker's clean speech, interfering human voice, and background noise; inputting the lip image sequence into a pre-trained spectrogram recognition network and outputting an estimated spectrogram sequence; inputting the estimated spectrogram sequence and the mixed speech together into a pre-trained target speaker extraction network and outputting the target speaker's clean speech.

2. The real-time audiovisual speech enhancement method based on target speaker visual pixel information according to claim 1, characterized in that, Determining the lip image sequence corresponding to the target speaker based on the video stream includes: locating the lip region of the target speaker based on the video stream; and based on the lip region, cropping the lip image of the target speaker according to the time axis and performing preprocessing to obtain a continuous lip image sequence.

3. The real-time audiovisual speech enhancement method based on target speaker visual pixel information according to claim 1, characterized in that, The training process of the visual pixel recognition network includes: acquiring an audiovisual database, obtaining lip image samples, speech samples, and text transcription samples based on the audiovisual database; converting the text transcription samples into a first phoneme sequence based on a phoneme-to-speech conversion module, and obtaining the correspondence between phonemes and the time axis based on a forced alignment module; converting the first phoneme sequence into a first frame aligned visual pixel sequence based on a preset phoneme-visme mapping rule and the correspondence between phonemes and the time axis; inputting the lip image samples into the basic visual pixel recognition network, and using the first frame aligned visual pixel sequence as a supervision label to train the basic visual pixel recognition network, thereby obtaining the trained visual pixel recognition network.

4. The real-time audiovisual speech enhancement method based on target speaker visual pixel information according to claim 1, characterized in that, The process of inputting the estimated spectrogram sequence and the mixed speech into a pre-trained target speaker extraction network and outputting the clean speech of the target speaker includes: performing time alignment, encoding, or embedding representation on the estimated spectrogram sequence to form spectrogram conditions; and inputting the spectrogram conditions and the mixed speech into a pre-trained target speaker extraction network and outputting the clean speech of the target speaker.

5. The real-time audiovisual speech enhancement method based on target speaker visual pixel information according to claim 1, characterized in that, The training process of the target speaker extraction network includes: acquiring a speech database and a noise database, wherein the speech database includes clean speech samples of the training object and corresponding text transcription samples, and the noise database includes background noise samples; converting the text transcription samples of the clean speech samples into a second phoneme sequence, and based on a preset phoneme-visual mapping rule and the correspondence between phonemes and the time axis, converting the second phoneme sequence into a second frame aligned visual sequence; mixing the clean speech samples, the background noise samples, and preset interfering human voice samples according to a set signal-to-noise ratio, and performing acoustic simulation to generate a mixed speech sample; inputting the mixed speech sample and the second frame aligned visual sequence together into the basic target speaker extraction network, and using the clean speech samples as the supervised target for training, to obtain the trained target speaker extraction network.

6. The real-time audiovisual speech enhancement method based on target speaker visual pixel information according to claim 5, characterized in that, The training process of the target speaker extraction network further includes applying data perturbation during the generation of the second frame aligned pixel sequence. The data perturbation includes any one or more of the following: random pixel replacement, pixel insertion, pixel deletion, pixel masking, or time jitter.

7. The real-time audiovisual speech enhancement method based on target speaker visual pixel information according to claim 5, characterized in that, The training process of the target speaker extraction network further includes: using an acoustic simulation strategy to perform data augmentation on the speech database and noise database to generate diverse training samples; the acoustic simulation strategy includes any one or more of the following: changing the number of speakers interfering with the speech, changing the type of background noise, changing the signal-to-noise ratio, and changing the reverberation intensity.

8. A real-time audiovisual speech enhancement system based on target speaker visual pixel information, characterized in that, The system is used to implement the steps of the real-time audiovisual speech enhancement method based on target speaker visual information as described in any one of claims 1-7. The system includes: an image and speech acquisition module, used to acquire a video stream of the target speaker and mixed speech in the current scene, and determine the lip image sequence corresponding to the target speaker based on the video stream, wherein the mixed speech includes the target speaker's clean speech, interfering human voice, and background noise; a visual sequence estimation module, used to input the lip image sequence into a pre-trained visual recognition network and output an estimated visual sequence; and a clean speech extraction module, used to input the estimated visual sequence and the mixed speech together into a pre-trained target speaker extraction network and output the target speaker's clean speech.

9. A terminal, characterized in that, The terminal includes a memory, a processor, and a real-time audiovisual speech enhancement program based on target speaker visual information stored in the memory and executable on the processor. When the processor executes the real-time audiovisual speech enhancement program based on target speaker visual information, it implements the steps of the real-time audiovisual speech enhancement method based on target speaker visual information as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a real-time audiovisual speech enhancement program based on target speaker visual pixel information, and the real-time audiovisual speech enhancement program based on target speaker visual pixel information implements the steps of the real-time audiovisual speech enhancement method based on target speaker visual pixel information as described in any one of claims 1-7 on the computer-readable storage medium.

Images