Multi-modal target speaker extraction method, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a multi-modal target speaker extraction method, electronic equipment and a storage medium. The method comprises the following steps: acquiring an audio / video with a speaker; inputting the audio and video into a video and audio feature extraction network based on a deep learning network; in a video and audio feature extraction network, semantic features of voices in audios and videos are extracted through sub-band coding so as to reduce the scale of a model, feature mapping is carried out on videos in the audios and videos through a shufflenet network, visual features are extracted, calculation overhead is reduced, the semantic features and the visual features are fused, and the video feature extraction efficiency is improved. Obtaining a multi-modal feature using the visual feature to constrain the semantic feature; and performing mask training on the speaker extraction model based on the neural cycle network by using the multi-modal features. According to the embodiment of the invention, the multi-modal speaker extraction method and the corresponding training method which are low in computing resource consumption and flexible in deployment are realized, and the method can adapt to real-time application scenes.

Description

Technical Field

[0001] The invention belongs to the field of intelligent speech, and in particular relates to a multimodal target speaker extraction method, electronic equipment and storage medium. Background Art

[0002] With the development of technology, TSE (Target Speaker Extraction) has been widely used in voice tasks. TSE can greatly improve speaker recognition and interaction efficiency. Specifically, TSE processes voice signals through deep learning, self-learning, adaptive filters, etc., to filter out interfering sound sources from complex multi-source environments and extract the voice of the designated target speaker. This has important application value for voice products and services.

[0003] TSE can be applied in the following areas: voice assistants can better respond to user commands and filter background noise; video conferencing software can clearly record and replay the speaker's voice; digital audio workstations can efficiently edit the designated voices at the recording site; medical voice recording assistants can effectively extract conversations between medical staff. In addition, TSE has also greatly improved the accuracy of voice search and recognition. It provides learning assistance for the hearing-impaired and safe driving capabilities for drivers. In voice enhancement and customer service systems, TSE also plays an important role.

[0004] The existing TSE technology collects multi-channel signals through a high-density microphone array, and then uses an unsupervised model based on variational autoencoders to separate sound sources. The staff will add voice tags later, filter and extract the specified speaking parts for editing. On the video conferencing software side, the voices of the participants are collected by integrating multiple audio input channels, and the voiceprint features of each participant are registered when the meeting is initialized. During the meeting, the TSE model uses the pre-registered voiceprint database and matches and tracks the voiceprints of the participants according to the timing and grammatical features of each input channel, thereby dynamically locking and extracting the speaker's voice stream. These products use a variety of technologies such as sound source separation, voiceprint recognition, and deep learning to achieve the functional requirements of dynamically locking and extracting the target speaker's voice from a complex multi-source environment in real time and effectively, thereby improving speech recognition and interactive experience.

[0005] In the process of implementing the present invention, the inventors found that there are at least the following problems in the related art: The target speaker extraction technology based on speech information in the existing technology has some defects when dealing with complex scenarios. Specifically, the TSE model relies on a large amount of labeled speech data for training, but the actual available data is limited, which will cause the problem of small sample learning. Most supervised learning models rely on labeled speech data for training and are constrained by public speech libraries, which causes the problem of TSE's poor performance in small sample situations. The computational efficiency of classical algorithms such as ICA or DUET is not high, and it is difficult to well support the deployment requirements of the end-side or edge-side. Specifically, it only relies on speech signals. When there are multiple speakers speaking alternately or simultaneously, it is difficult to accurately distinguish between different speaking intervals; at the same time, TSE models may not be able to quickly adapt to new speakers or changes in audio signals, which may limit their ability to handle new unknown speakers. Speakers' voice features (such as pitch, tone, and accent) may vary greatly. TSE models may have difficulty handling this variability, especially when encountering new unknown speakers, and their recognition ability is also weak. In addition, methods based on blind source separation are difficult to handle speech signals in highly mixed and dynamic environments. This method is often affected by permutation ambiguity, which means that the order of separated sources is arbitrary. In highly mixed environments, this ambiguity may lead to incorrect source assignment. Specifically, it is also limited in handling dynamically changing conference scenes because it is difficult to cope with scene changes in the absence of visual information. Moreover, most speech-based algorithms have high computational complexity and are difficult to run in real time on edge devices or other devices with simple configurations. These technical limitations affect its application in actual complex scenarios. Summary of the invention

[0006] In order to at least solve the problem that the target speaker extraction technology in the prior art has defects when processing complex scenes. In a first aspect, an embodiment of the present invention provides a multimodal target speaker extraction method, comprising: Get audio and video with speakers; Inputting the audio and video into an audio and video feature extraction network based on a deep learning network, wherein the audio and video feature extraction network includes: a shufflenet network in which subband coding, convolutional layers are replaced by point-by-point group convolution and channel shuffling; In the audio-visual feature extraction network, the subband coding is used to extract the semantic features of the speech in the audio and video to reduce the model size, the shufflenet network is used to perform feature mapping on the video in the audio and video to extract the visual features to reduce the computational overhead, and the semantic features are fused with the visual features to obtain a multimodal feature that uses the visual features to constrain the semantic features; The multimodal features are used to perform mask training on a speaker extraction model based on a neural recurrent network, and the training is performed until the reconstructed audio restored by the speaker extraction model is close to the speech in the audio and video with the speaker, so that the speaker extraction model can extract the audio corresponding to the target speaker in the audio and video with the speaker.

[0007] In a second aspect, an embodiment of the present invention provides a multimodal target speaker extraction system, comprising: An audio and video acquisition module, used to acquire audio and video with speakers; An input module, used for inputting the audio and video into an audio and video feature extraction network based on a deep learning network, wherein the audio and video feature extraction network includes: a shufflenet network in which subband coding, convolutional layers are replaced by point-by-point group convolution and channel shuffling; A multimodal feature fusion module is used to extract the semantic features of the speech in the audio and video using the subband coding in the audio and video feature extraction network to reduce the model size, perform feature mapping on the video in the audio and video using the shufflenet network to extract visual features to reduce the computational overhead, and fuse the semantic features with the visual features to obtain multimodal features that use visual features to constrain semantic features; The speaker extraction module is used to perform mask training on a speaker extraction model based on a neural recurrent network using the multimodal features, and train the speaker extraction model until the reconstructed audio restored by the speaker extraction model is close to the speech in the audio and video with the speaker, so that the speaker extraction model can extract the audio corresponding to the target speaker in the audio and video with the speaker.

[0008] According to a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the multimodal target speaker extraction method of any embodiment of the present invention.

[0009] In a fourth aspect, an embodiment of the present invention provides a storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the multimodal target speaker extraction method of any embodiment of the present invention are implemented.

[0010] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the steps of the multimodal target speaker extraction method of any embodiment of the present invention are implemented.

[0011] The beneficial effect of the embodiment of the present invention is that the method realizes a multimodal speaker extraction method and a corresponding training method with low computing resource consumption and flexible deployment. It can adapt well to inputs under different conditions, and the low-latency deployment capability meets the needs of real-time application scenarios. This greatly enhances the feasibility of the present application in actual services. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0013] Figure 1 is a flow chart of a multimodal target speaker extraction method provided by an embodiment of the present invention; Figure 2 is a visual preprocessing flow chart of a multimodal target speaker extraction method provided by an embodiment of the present invention; Figure 3 It is a speech restoration flow chart of a multimodal target speaker extraction method provided by an embodiment of the present invention; Figure 4 is a schematic diagram of experimental results of a multimodal target speaker extraction method provided by an embodiment of the present invention; Figure 5 It is a schematic diagram of the structure of a multimodal target speaker extraction system provided by an embodiment of the present invention; Figure 6 A schematic structural diagram of an embodiment of an electronic device for multimodal target speaker extraction provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] like Figure 1 FIG. 1 is a flow chart of a multimodal target speaker extraction method provided by an embodiment of the present invention, comprising the following steps: S11: Get audio and video with the speaker; S12: Inputting the audio and video into an audio and video feature extraction network based on a deep learning network, wherein the audio and video feature extraction network includes: a shufflenet network in which subband coding, convolutional layers are replaced by point-by-point group convolution and channel shuffling; S13: In the audio-visual feature extraction network, the subband coding is used to extract the semantic features of the speech in the audio and video to reduce the model size, the shufflenet network is used to perform feature mapping on the video in the audio and video to extract visual features to reduce the computational overhead, and the semantic features are fused with the visual features to obtain multimodal features that use visual features to constrain semantic features; S14: Mask training is performed on a speaker extraction model based on a neural recurrent network using the multimodal features until the reconstructed audio restored by the speaker extraction model is close to the speech in the audio and video with the speaker, so that the speaker extraction model can extract the audio corresponding to the target speaker in the audio and video with the speaker.

[0016] In this implementation, considering that the speaker identification task in the prior art is mainly based on the study of speech signals, and visual information is not fully utilized, this method considers whether visual information can be used to make up for the lack of speech information to improve the target speaker extraction effect. However, the combination of visual information and speech information will encounter the following difficulties: 1. Speech feature extraction is relatively easy, but visual feature extraction is more difficult.

[0017] 2. Speech and vision are two different modalities. How to effectively integrate them has always been a difficulty in carrying out cross-modal tasks.

[0018] 3. Direct use of vision in mobile and edge scenarios will also bring higher computing overhead. Usually, mobile devices and devices in edge scenarios have relatively weak configurations and poor computing capabilities, and cannot afford this computing overhead.

[0019] 4. Existing speech datasets also lack corresponding speech and video samples, making it difficult to verify the visual enhancement effect. Considering that the use of visual information also needs to deal with privacy issues, it is difficult to implement it in commercial products.

[0020] In general, in the essence of target speaker extraction technology, voice is often regarded as the core feature, and visual functions are not easily valued. In terms of overall system design, it is difficult to think of how to coordinate vision and voice. The speaker separation technology that relies on voice features in the existing technology has certain limitations when dealing with complex multi-person interaction scenarios, and it is difficult to achieve accurate speaker extraction. Through thinking and simulating these technologies, this method found that they have problems in the following aspects: 1. In a meeting scenario where multiple people are discussing alternately, voice alone cannot efficiently distinguish the voice segments of different speakers. 2. These methods are difficult to adapt to the dynamic changes of the scene, and the extraction effect is not good. 3. Based on a single voice feature, it is not enough to achieve accurate identification of new users.

[0021] In order to make up for the shortcomings of the target speaker extraction method based only on speech, this method thinks of using visual information as a supplement. Therefore, this method proposes a speaker separation method based on video multimodal information. The core idea of ​​this method is: to use vision to extract the speaker's mouth shape features and interactive behaviors, which can well assist and constrain speech information. This method uses a deep learning network to simultaneously extract speech and visual features, and deeply fuses them to obtain the separation results. Compared with the existing technology, this method can more accurately extract speaking intervals and new users through visual information, and process dynamic scenes more efficiently, so as to improve the effect and robustness of speaker separation.

[0022] For step S11, this method can be applied to multi-person interaction scenarios, such as two-person dialogue scenarios and multi-person interaction conference scenarios, and in the above scenarios, a newly added unknown speaker will also appear (for example, the voice features of the unknown speaker have not been recorded). This method can be configured in a terminal device with a microphone and a camera. In a multi-person interaction scenario, the microphone is used to collect audio, and the camera is used to collect video to obtain audio and video with the speaker. For terminal devices without cameras, such as smart speakers, the smart speaker can be used to collect audio, and the camera in the scene can be used to collect video, and the audio and video can be synchronized to obtain audio and video with the speaker. In the target speaker extraction task, the above method can be used to obtain audio and video with the speaker. Since this method can cope with complex multi-person interaction scenarios, it can cope with the dynamically changing conference scenes of newly added unknown speakers. At the same time, after training and optimization, the algorithm and model of this method can be applied to edge devices with low configuration and limited computing power in real time. If the target speaker extraction model is to be trained, the audio and video with speakers as training data can gather multiple speakers to talk to each other, and at the same time collect videos of the speakers during their speaking process, so as to obtain the audio and video with speakers for training.

[0023] As an implementation manner, the method further includes preprocessing the audio and video with the speaker, including: Performing data enhancement including rotation and cropping on the image frame of the video in the audio and video with the speaker; Using face detection to locate the speaker in the audio and video, and performing facial alignment processing on the speaker to obtain a facial image of the speaker; Cut out the mouth image from the facial image, and extract the grayscale value of the mouth image as an input parameter of the video in the audio and video; The speech in the audio or video with the speaker is framed and windowed.

[0024] In this embodiment, the method uses video preprocessing technology to extract the speech signal and each person's mouth movement image sequence. Figure 2 shown.

[0025] Data enhancement is performed on audio and video with speakers. Specifically, for the video in the audio and video, the image frames of the video are extracted, and the images are rotated, cropped, and other operations are performed to expand the image data set. Face detection and positioning are performed on the expanded image data set to determine the speaker (or speakers) in the image, and facial alignment is performed at the same time. The mouth area in the facial image is cropped, and the grayscale value of the mouth image is extracted, so that parameter data for target speaker extraction can be obtained. Similarly, speech also needs to be preprocessed. Specifically, the speech signal in the audio and video can be framed and windowed, and the short-time Fourier transform can be calculated to extract the frequency characteristics of the speech, so that parameter data for target speaker extraction can be obtained. Data enhancement and widening technologies are implemented on the collected audio and video. Specifically, flipping, cropping, and noise addition technologies are used on image data to compensate for data expansion. This effectively expands the adaptability of the trained model under different angles and lighting conditions, and also helps the extraction effect of target speaker extraction.

[0026] For step S12, the audio and video are input into the audio and video feature extraction network based on the deep learning network. The multimodal target speaker extraction of this method is divided into two parts, one part is the multimodal audio and video feature extraction network, and the other part is the multimodal target speaker extraction model.

[0027] Regarding the audio and video feature extraction network, this method uses a deep learning network to automatically extract semantic features from speech and vision respectively. Using feature fusion and deep learning methods, the deep fusion of speech and visual information is achieved. In terms of audio data, this method uses subband coding technology in the field of speech processing to extract audio features. Compared with directly using periodic signals, subband coding can better abstract semantic information and reduce the scale of training models. Considering the low latency requirements in edge scenarios, this method optimizes the model structure. Specifically, point-by-point group convolution and channel shuffling are introduced into the audio and video feature extraction network to reduce the number of calculations while completing information interaction. This structure replaces ordinary convolution and can greatly reduce computational overhead. At the same time, this method also optimizes the deployment of the model by means of quantization.

[0028] For step S13, subband coding is to convert the original signal from the time domain to the frequency domain, and then divide it into several sub-bands to extract the semantic features of the speech in the audio and video. The pitch and resonance peak of the sub-band speech are mainly concentrated in the low frequency band, so more bits can be allocated to represent its sample value, while fewer bits can be allocated to the high frequency band where friction sounds and similar friction noises appear, so that the speech data can be fully compressed, thereby reducing the model size. ShuffleNet uses a computationally efficient CNN architecture. In order to further apply this method to edge devices, ShuffleNet introduces point-by-point group convolution and channel shuffling to reduce the number of calculations while completing information interaction. This structure replaces ordinary convolution and can greatly reduce computational overhead. Among them, point-by-point convolution is a special convolution operation that uses a 1x1 convolution kernel to convolve the input feature map. Point-by-point convolution is mainly used to change the number of channels of the feature map without changing the spatial dimension of the feature map. Channel shuffling can complete the fusion of information between channels, further reducing the amount of calculation. In this way, visual features and semantic features can be obtained efficiently and conveniently, and visual features and semantic features can be fused to obtain multimodal features that use visual features to constrain semantic features. The speaker's mouth shape features and interactive behaviors are extracted using vision, which can well assist and constrain voice information.

[0029] For step S14, for target speaker extraction, after obtaining the multimodal features, they are input into a speaker extraction model based on a neural recurrent network, and the audio corresponding to the target speaker can be directly extracted.

[0030] In the training, the method specifically includes: optimizing the residual connection of the multimodal features by using a long short-term memory network, masking the multimodal features, and using the speech in the audio and video with the speaker as a time-frequency mask reference; The multimodal features of the mask are restored by inverse short-time Fourier transform to obtain reconstructed audio, and the training is performed until the reconstructed audio approaches the time-frequency mask reference.

[0031] In this implementation, visual features and semantic features are connected and fused, and then input into LSTM (Long Short-Term Memory) for sequence modeling, optimizing the residual connection of the model. The multimodal features are masked, and the speech in the audio and video with the speaker is used as the time-frequency mask reference, which is the training goal.

[0032] The speaker extraction model based on the neural recurrent network is used to predict and generate the probability of existence of each sub-band speech, such as the probability of whether there is speech in the lip shape in the video, and the probability of existence of each speech corresponding to the lip shape. Since the multimodal features are masked, the mask estimation of the full band is restored by the neural network, the mask is calculated with the spectrum in the microphone, and the speech waveform is reconstructed by ISTFT-inverse short-time Fourier transform, and the training is performed until the reconstructed audio approaches the time-frequency mask reference. By masking the speech, the speech information and lip movement information (lip movement information is obtained by extracting the face and mouth in the image frame in preprocessing) are matched, and the speaking interval and speaker results in the input video can be given.

[0033] As an implementation manner, the audio in the audio or video with a speaker includes: single-channel audio collected by a single microphone.

[0034] It can be seen from this implementation that the present application can be applied to a simple edge device, and the audio collected by the device is a single-channel audio collected by a single microphone. Even with single-channel audio, multi-modal target speaker extraction can be achieved.

[0035] As another implementation, the audio in the audio and video with the speaker includes: multi-channel audio collected by a microphone array; The method further comprises: determining a phase difference characteristic between audio signals of each microphone in the microphone array; The phase difference feature, the semantic feature and the visual feature are integrated to obtain a multimodal feature; The multimodal features are used to perform mask training for introducing spatial audio information into a speaker extraction model based on a neural recurrent network.

[0036] In this embodiment, the present application may also be applied to other smart devices, which may have a microphone array consisting of multiple microphones, taking two microphones as an example.

[0037] In addition to visual and single-channel audio information, this method also takes advantage of the hardware advantages of the device and adopts a dual-microphone array structure. Microphone arrays are set on both sides of the device, which can be used to simultaneously obtain the audio signals of each speaking participant. By analyzing the time difference or phase shift relationship between audio signals at different acquisition points, the phase information is input into the speaker extraction model after feature fusion. Compared with mono audio, the dual-microphone array combined with phase positioning technology takes advantage of spatial audio information. This further makes up for and strengthens the target speaker extraction based on vision and mono audio. The whole process is as follows Figure 3 Finally, the method was used to extract the target speaker in multimodal mode. The results are shown in Figure 4 As shown in the figure, sir is the source interference ratio and pesq is the perceived evaluation of voice quality. Compared with traditional voiceprints, in addition to eliminating the registration time and computing power overhead, the performance has also been improved to a certain extent. After adding the phase information of the dual microphones, the performance has also been significantly improved with the same computing power overhead.

[0038] In general, the steps of this method can be summarized as follows: Step 1: Data Preprocessing (1) Enhance the visual and speech data, such as rotating and cropping, to expand the dataset.

[0039] (2) Use face detection to locate the speaker and perform facial alignment.

[0040] (3) Crop the mouth area of ​​the acquired facial image.

[0041] (4) Extract the grayscale value of the mouth image to obtain visual feature input.

[0042] Step 2: Speech Preprocessing (1) Perform frame and window processing on the speech signal.

[0043] (2) Calculate the short-time Fourier transform and extract the frequency features.

[0044] (3) Calculate the phase difference at different sampling points to obtain the IPD feature.

[0045] Step 3: Multimodal feature extraction (1) The speech features are decomposed into sub-bands according to the probability distribution of speech information through a neural network.

[0046] (2) Input the visual features into the shufflenet network to extract the visual features.

[0047] Step 4: Feature fusion and modeling (1) Fusion of audio and visual features through concatenation.

[0048] (2) Input LSTM sequence modeling and residual connection optimization.

[0049] (3) After the existence probability on the generated sub-band is obtained, the mask estimate of the full band is restored through a neural network and a complex-valued TF mask reference is generated.

[0050] Step 5: Voice Reconstruction The mask is calculated with the spectrum in the mic and the speech waveform is reconstructed through ISTFT.

[0051] It can be seen from this implementation that this method realizes a multimodal speaker extraction method and a corresponding training method with low computing resource consumption and flexible deployment. It can adapt well to inputs under different conditions, and its low-latency deployment capability meets the needs of real-time application scenarios. This greatly enhances the feasibility of this application in actual services.

[0052] like Figure 5 FIG. 1 is a schematic diagram of the structure of a multimodal target speaker extraction system provided by an embodiment of the present invention. The system can execute the multimodal target speaker extraction method described in any of the above embodiments and is configured in a terminal.

[0053] A multimodal target speaker extraction system 10 provided in this embodiment includes: an audio and video acquisition module 11, an input module 12, a multimodal feature fusion module 13 and a speaker extraction module 14.

[0054] The audio and video acquisition module 11 is used to acquire audio and video with a speaker; the input module 12 is used to input the audio and video into an audio and video feature extraction network based on a deep learning network, wherein the audio and video feature extraction network includes: a shufflenet network in which subband coding, convolution layers are replaced by point-by-point group convolution and channel shuffling; the multimodal feature fusion module 13 is used to extract the semantic features of the speech in the audio and video in the audio and video feature extraction network using the subband coding to reduce the model scale, use the shufflenet network to perform feature mapping on the video in the audio and video, extract visual features to reduce the computational overhead, and fuse the semantic features with the visual features to obtain multimodal features that use visual features to constrain semantic features; the speaker extraction module 14 is used to perform mask training on a speaker extraction model based on a neural recurrent network using the multimodal features, and train until the reconstructed audio restored by the speaker extraction model is close to the speech in the audio and video with the speaker, so that the speaker extraction model extracts the audio corresponding to the target speaker in the audio and video with the speaker.

[0055] The embodiment of the present invention further provides a non-volatile computer storage medium, the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the multi-modal target speaker extraction method in any of the above method embodiments; As an implementation mode, the non-volatile computer storage medium of the present invention stores computer executable instructions, and the computer executable instructions are configured as follows: Get audio and video with speakers; Inputting the audio and video into an audio and video feature extraction network based on a deep learning network, wherein the audio and video feature extraction network includes: a shufflenet network in which subband coding, convolutional layers are replaced by point-by-point group convolution and channel shuffling; In the audio-visual feature extraction network, the subband coding is used to extract the semantic features of the speech in the audio and video to reduce the model size, the shufflenet network is used to perform feature mapping on the video in the audio and video to extract the visual features to reduce the computational overhead, and the semantic features are fused with the visual features to obtain a multimodal feature that uses the visual features to constrain the semantic features; The multimodal features are used to perform mask training on a speaker extraction model based on a neural recurrent network, and the training is performed until the reconstructed audio restored by the speaker extraction model is close to the speech in the audio and video with the speaker, so that the speaker extraction model can extract the audio corresponding to the target speaker in the audio and video with the speaker.

[0056] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by the processor, the multimodal target speaker extraction method in any of the above method embodiments is executed.

[0057] Figure 6 FIG. 1 is a schematic diagram of the hardware structure of an electronic device of a multimodal target speaker extraction method provided by another embodiment of the present application. Figure 6 As shown, the device includes: One or more processors 610 and memory 620, Figure 6 A processor 610 is taken as an example. The device of the multimodal target speaker extraction method may further include: an input device 630 and an output device 640.

[0058] The processor 610, the memory 620, the input device 630 and the output device 640 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0059] The memory 620, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the multimodal target speaker extraction method in the embodiment of the present application. The processor 610 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 620, that is, the multimodal target speaker extraction method in the above method embodiment is implemented.

[0060] The memory 620 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data, etc. In addition, the memory 620 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include a memory remotely arranged relative to the processor 610, and these remote memories may be connected to the mobile device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0061] The input device 630 can receive input digital or character information. The output device 640 can include a display device such as a display screen.

[0062] The one or more modules are stored in the memory 620 , and when executed by the one or more processors 610 , perform the multimodal target speaker extraction method in any of the above method embodiments.

[0063] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0064] The non-volatile computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the non-volatile computer-readable storage medium may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0065] An embodiment of the present invention also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the multimodal target speaker extraction method of any embodiment of the present invention.

[0066] The electronic device of the embodiment of the present application exists in various forms, including but not limited to: (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.

[0067] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access features. These terminals include: PDAs, MIDs, and UMPC devices, such as tablet computers.

[0068] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0069] (4) Other electronic devices with data processing functions.

[0070] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include" and "comprise" include not only those elements, but also other elements not explicitly listed, or also include elements inherent to such processes, methods, articles or equipment. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or equipment that includes the elements.

[0071] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0072] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal target speaker extraction method, comprising: Get audio and video with speakers; Inputting the audio and video into an audio and video feature extraction network based on a deep learning network, wherein the audio and video feature extraction network includes: a shufflenet network in which subband coding, convolutional layers are replaced by point-by-point group convolution and channel shuffling; In the audio-visual feature extraction network, the subband coding is used to extract the semantic features of the speech in the audio and video to reduce the model size, the shufflenet network is used to perform feature mapping on the video in the audio and video to extract the visual features to reduce the computational overhead, and the semantic features are fused with the visual features to obtain a multimodal feature that uses the visual features to constrain the semantic features; The multimodal features are used to perform mask training on a speaker extraction model based on a neural recurrent network, and the training is performed until the reconstructed audio restored by the speaker extraction model is close to the speech in the audio and video with the speaker, so that the speaker extraction model can extract the audio corresponding to the target speaker in the audio and video with the speaker.

2. The method according to claim 1, wherein: The audio in the audio and video with a speaker includes: single-channel audio collected by a single microphone.

3. The method according to claim 1, wherein: The audio in the audio and video with the speaker includes: multi-channel audio collected by a microphone array; The method further comprises: determining a phase difference characteristic between audio signals of each microphone in the microphone array; The phase difference feature, the semantic feature and the visual feature are integrated to obtain a multimodal feature; The multimodal features are used to perform mask training for introducing spatial audio information into a speaker extraction model based on a neural recurrent network.

4. The method according to claim 1, wherein: The method of using the multimodal features to perform mask training on a speaker extraction model based on a neural recurrent network until the reconstructed audio restored by the speaker extraction model is close to the speech in the audio and video with the speaker includes: Performing residual connection optimization on the multimodal features using a long short-term memory network, and masking the multimodal features, using the speech in the audio and video with the speaker as a time-frequency mask reference; The multimodal features of the mask are restored by inverse short-time Fourier transform to obtain reconstructed audio, and the training is performed until the reconstructed audio approaches the time-frequency mask reference.

5. The method according to claim 1, wherein: The method further includes preprocessing the audio and video with the speaker, including: Performing data enhancement including rotation and cropping on the image frame of the video in the audio and video with the speaker; Using face detection to locate the speaker in the audio and video, and performing facial alignment processing on the speaker to obtain a facial image of the speaker; Cut out the mouth image from the facial image, and extract the grayscale value of the mouth image as an input parameter of the video in the audio and video; The speech in the audio or video with the speaker is framed and windowed.

6. A multimodal target speaker extraction system, comprising: An audio and video acquisition module, used to acquire audio and video with speakers; An input module, used for inputting the audio and video into an audio and video feature extraction network based on a deep learning network, wherein the audio and video feature extraction network includes: a shufflenet network in which subband coding, convolutional layers are replaced by point-by-point group convolution and channel shuffling; A multimodal feature fusion module is used to extract the semantic features of the speech in the audio and video using the subband coding in the audio and video feature extraction network to reduce the model size, perform feature mapping on the video in the audio and video using the shufflenet network to extract visual features to reduce the computational overhead, and fuse the semantic features with the visual features to obtain multimodal features that use visual features to constrain semantic features; The speaker extraction module is used to perform mask training on a speaker extraction model based on a neural recurrent network using the multimodal features, and train the speaker extraction model until the reconstructed audio restored by the speaker extraction model is close to the speech in the audio and video with the speaker, so that the speaker extraction model can extract the audio corresponding to the target speaker in the audio and video with the speaker.

7. The system according to claim 6, wherein: The audio in the audio and video with the speaker includes: multi-channel audio collected by a microphone array; The multimodal feature fusion module is also used for: Determining a phase difference characteristic between audio signals of each microphone in the microphone array; The phase difference feature, the semantic feature and the visual feature are fused to obtain a multimodal feature.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

9. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method described in any one of claims 1 to 5.

10. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 5 are implemented.

Images