Multi-modal speech processing method, device, medium and computer program product

Abstract

The invention relates to the technical field of speech recognition, and particularly provides a multi-mode speech processing method and device, a medium and a computer program product. The multi-mode voice processing method comprises the following steps: acquiring to-be-processed audio and video data, and performing data separation processing on the to-be-processed audio and video data to obtain original audio data and original video data; performing face detection processing on the original video data to obtain initial video data; performing audio detection processing on the original audio data to obtain initial audio data; and performing consistency detection processing on the initial video data and the initial audio data, and outputting target audio data and target video data under the condition that a consistency verification condition is met. The face in the video is recognized through face detection, noise and sound of the target speaker are recognized and separated through audio detection, and synchronization and matching of audio and video data in time and content are ensured through consistency detection, so that development of a multi-mode voice separation task is supported.

Description

Technical Field

[0001] The present disclosure relates to the field of speech recognition technology, and in particular to a multimodal speech processing method, device, medium and computer program product. Background Art

[0002] Currently, speech separation technology is widely used in many fields. For example, speech separation technology is added to the speech recognition system to separate the voice of the target speaker from other interfering sounds, thereby improving the robustness of the speech recognition system.

[0003] In the related art, the automatic speech recovery solution only performs speech separation processing on audio, and fails to recover audio and video multimodal data. Multimodal speech separation can often achieve better separation results than single-modal speech separation through lip sound consistency. Therefore, the existing automatic speech recovery solution cannot meet the needs of multimodal speech separation tasks, which greatly limits the application scope and effect of speech separation technology. Summary of the invention

[0004] In view of this, exemplary embodiments of the present disclosure provide a multimodal speech processing method, device, medium, and computer program product to solve the problems existing in the related art.

[0005] According to one aspect of an exemplary embodiment of the present disclosure, there is provided a multimodal speech processing method, the method comprising:

[0006] Acquire the audio and video data to be processed, and perform data separation processing on the audio and video data to be processed to obtain original audio data and original video data;

[0007] Performing face detection processing on the original video data to obtain initial video data;

[0008] Performing audio detection processing on the original audio data to obtain initial audio data;

[0009] The initial video data and the initial audio data are subjected to consistency check processing, and when consistency check conditions are met, target audio data and target video data are output.

[0010] Another aspect of the exemplary embodiments of the present disclosure provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described in the exemplary embodiments of the present disclosure.

[0011] According to another aspect of the exemplary embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the method described in the exemplary embodiments of the present disclosure is implemented.

[0012] According to another aspect of the exemplary embodiments of the present disclosure, a computer program product is provided, including a computer program / instruction. When the computer program / instruction is executed by a processor, the method described in the exemplary embodiments of the present disclosure is implemented.

[0013] As will be described in detail below, according to a multimodal speech processing method of an embodiment of the present disclosure, by acquiring audio and video data to be processed, and performing data separation processing on the audio and video data to be processed, original audio data and original video data are obtained; face detection processing is performed on the original video data to obtain initial video data; audio detection processing is performed on the original audio data to obtain initial audio data; consistency detection processing is performed on the initial video data and the initial audio data, and when the consistency verification conditions are met, target audio data and target video data are output. Therefore, the multimodal speech processing method provided by the present disclosure recognizes faces in the video through face detection, recognizes and separates noise and the voice of the target speaker through audio detection, and ensures synchronization and matching of audio and video data in time and content through consistency detection, thereby supporting the implementation of multimodal speech separation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0015] Figure 1 A flowchart of a multimodal speech processing method provided by an exemplary embodiment of the present disclosure;

[0016] Figure 2 This is a schematic diagram of a face frame and face key point detection result provided by an exemplary embodiment of the present disclosure;

[0017] Figure 3 A schematic diagram of a non-stationary noise detection model training process provided by an exemplary embodiment of the present disclosure;

[0018] Figure 4 A schematic diagram of a speaker number detection model training process provided by an exemplary embodiment of the present disclosure;

[0019] Figure 5 A schematic diagram of a speech noise reduction model training process provided as an example in the present disclosure;

[0020] Figure 6 A schematic diagram of the training process of an audio and video consistency detection model provided by an exemplary embodiment of the present disclosure;

[0021] Figure 7 A flowchart of a multimodal speech processing method provided by an exemplary embodiment of the present disclosure;

[0022] Figure 8 A schematic block diagram of functional modules of a multimodal speech processing device provided by an exemplary embodiment of the present disclosure;

[0023] Fig. 9 A structural block diagram of an electronic device provided as an example in the present disclosure;

[0024] Fig.10 A schematic diagram of a computer program product provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0026] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0027] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0028] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0029] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0030] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0031] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0032] As an optional but non-limiting implementation, in response to receiving an active request from the user, the method of sending a prompt message to the user may be, for example, a pop-up window, in which the prompt message may be presented in text. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device. It is understandable that the above notification and the process of obtaining user authorization are only illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0033] Currently, speech separation technology is widely used in many fields. For example, speech separation technology is added to the speech recognition system to separate the voice of the target speaker from other interfering sounds, thereby improving the robustness of the speech recognition system.

[0034] In the related art, the automatic speech recovery solution only performs speech separation processing on audio, and fails to recover audio and video multimodal data. Multimodal speech separation can often achieve better separation results than single-modal speech separation through lip sound consistency. Therefore, the existing automatic speech recovery solution cannot meet the needs of multimodal speech separation tasks, which greatly limits the application scope and effect of speech separation technology.

[0035] Therefore, in order to solve the above problems, the exemplary embodiment of the present disclosure provides a multimodal speech processing method, which separates the audio and video in the audio and video to be processed by ffmpeg (Fast Forward MPEG (Moving Picture Experts Group)), determines the audio segment and the video segment to be detected by voice activity detection; performs face detection on the video segment to be detected, cuts the face area of ​​the video that meets the condition that the number of faces = 1 and the face is complete, and retains it, and discards it if it does not meet the condition; performs impact noise detection on the audio segment to be detected corresponding to the retained video, detects that the audio does not contain impact noise and retains it, and detects that the audio contains impact noise and discards it; at the same time, performs speaker number detection on the audio segment to be detected corresponding to the retained video, detects that the audio with only single-person voice and no interference voice is retained, and detects that the audio with non-single-person voice is discarded. Perform speech enhancement on the retained audio, remove background noise and reverberation, perform audio and video consistency detection on the enhanced voice segment and the corresponding video segment, and if the output bias frame number = 0 and the confidence is greater than the set value, the audio and video are retained as multimodal speech separation training corpus.

[0036] For example, Figure 1 A flowchart of a multimodal speech processing method provided by an exemplary embodiment of the present disclosure is shown in FIG. Figure 1 As shown, the following steps may be specifically included:

[0037] Step S110: pre-processing the original audio and video data.

[0038] The original audio and video data to be processed is obtained, and the original audio and video data is preprocessed.

[0039] The first step of preprocessing is to separate the audio and video signals in the original audio and video data to obtain the original audio data and the original video data.

[0040] Exemplarily, the audio and video signals can be separated by ffmpeg, which is a widely used multimedia framework that can process audio and video data in various formats and supports conversion and separation between audio and video.

[0041] Furthermore, the second step of preprocessing is to use voice activity detection (VAD) technology to identify the valid speech part in the original audio data. VAD can distinguish between background noise and human voice, and can accurately identify the original audio segment that needs further analysis. In practical applications, the length of the original audio segment can be controlled within 2s to 20s, which can ensure the integrity of the voice information and facilitate processing and analysis.

[0042] Based on this, by preprocessing the original audio and video data, the original audio data and the original video data can be separated, and the effective voice part in the original audio data can be detected to provide data for subsequent audio detection and video detection.

[0043] Step S120: performing face detection on the original video data by using a face detection module.

[0044] The face detection module is used to detect whether there are faces in the original video data, the number of faces, and whether the faces are fully visible throughout the entire process.

[0045] For example, MTCNN (Multi-task Cascaded Convolutional Networks) can be used as a face detection model to perform face detection on raw video data. MTCNN is an efficient face detection tool that can detect the position and number of faces in video frames and provide detailed information on facial key points, including important features such as eyes, nose, and mouth. Key points can be used to characterize the direction and expression state of the face.

[0046] During the processing, each video frame in the original video data is input into the MTCNN model in turn, and the model outputs the face frame and face key point information.

[0047] In order to ensure the consistency and availability of the data, it is also necessary to check whether the face in the video clip continues to appear. Specifically, it can be verified whether the face frame is continuously equal to 1 (i.e., a face is always detected) throughout the video clip, and whether the five main facial key points (eyes, nose, mouth, etc.) are always visible. This step can help exclude video clips with incomplete facial information due to factors such as mask occlusion and head rotation.

[0048] If the face frame is continuously equal to 1 and the five faces are not continuously visible, the corresponding video frame is retained.

[0049] If the face frame is not continuously equal to 1 or the five facial key points are not continuously visible, the corresponding video frame is removed.

[0050] For example, Figure 2 This is a schematic diagram of a face frame and face key point detection result provided by an exemplary embodiment of the present disclosure, such as Figure 2 As shown in the figure, the borders and black dots in the picture represent the face frames and the positions of the key points of the face detected by the model in some sample video frames. The number of face frames represents the number of faces detected in the video, the position of the face frame is the position of the face, and the five key points of the face represent the positions of the eyes, mouth and nose of the detected face.

[0051] In addition, in order to simplify the processing steps and reduce the computational burden, the face detection model does not involve face recognition, that is, it does not consider whether the faces in the video are the same person. If different people's faces appear in the video data, the audio and video inconsistency problem caused by it will be identified and processed in the subsequent audio and video consistency detection module.

[0052] Based on this, the face detection module can remove video frames in the original video data where the face frame is not continuously equal to 1 or the five face key points are not continuously visible, further reducing the amount of calculation for subsequent processing and reducing the computational burden.

[0053] Step S130: performing noise detection on the original audio data through a non-stationary noise detection module.

[0054] In current speech processing technologies, both single-channel noise reduction modules based on deep learning neural networks and single-channel noise reduction modules based on speech signal processing technology have shown excellent performance in suppressing stationary noises such as wind noise. However, for non-stationary noises such as whistles or percussion sounds, the noise reduction effect is often not ideal. Due to its unpredictable nature, non-stationary noise brings additional challenges to accurate noise identification and elimination.

[0055] In order to solve this problem, this embodiment proposes a non-stationary noise detection model, the main task of which is to analyze audio data and determine whether it contains non-stationary noise. Only audio data that does not contain noise or only contains stationary noise and has passed through the noise reduction module can be used as clean corpus for speech separation model training.

[0056] For example, Figure 3 This is a schematic diagram of a non-stationary noise detection model training process provided by an exemplary embodiment of the present disclosure, such as Figure 3 As shown, the training process of the non-stationary noise detection model may include:

[0057] Step S310: construct a training database.

[0058] The training database may include: a high-fidelity speech library, a stationary noise library, and a non-stationary noise library.

[0059] The high-fidelity speech library contains clear, noise-free human voice samples, which is the basis for building training data. The stationary noise library and non-stationary noise library collect common stationary noise and impact noise respectively, which will be superimposed with clear speech in subsequent steps to simulate various auditory scenarios in real environments.

[0060] Step S320: superimposing noise according to different signal-to-noise ratios to obtain mixed speech.

[0061] For example, stationary noise, non-stationary noise or a combination thereof can be superimposed on the high-fidelity speech in the time domain at different signal-to-noise ratios to construct a mixed speech as a model input, and the mixed speech is used to generate a corresponding training set and development set. The training label of a speech sample containing no noise or only stationary noise is marked as 1, while the training label of a speech sample containing non-stationary noise is marked as 0.

[0062] The types of mixed speech may include: no superimposed noise, superimposed stationary noise only, superimposed non-stationary noise only, and superimposed stationary noise and non-stationary noise at the same time.

[0063] Step S330: training a non-stationary noise detection model.

[0064] The input of the non-stationary noise detection model is mixed speech, and the mixed speech is processed by short-time Fourier transform to extract audio features. The audio features can be logmel spectrum, and the basic structure and noise information of the audio can be captured according to the logmel spectrum.

[0065] Then, the audio features are input into a bidirectional long short-term memory network (BiLSTM) and a series of fully connected layers (FC). BiLSTM can effectively process sequence data and capture context information, while the fully connected layer is used to extract features for classification from the output of BiLSTM. The output of the model is a binary classification prediction of whether each sentence contains non-stationary noise. During the iterative training process, the cross entropy loss (CE loss) can be used to optimize the model.

[0066] The termination condition of the model training iteration is the convergence of the loss on the development set, indicating that the model has learned the ability to distinguish different types of noise.

[0067] After the model training is completed, the original audio data is input into the non-stationary noise detection model to perform noise detection, and obtain a detection result of whether the current audio segment contains non-stationary noise. If it contains non-stationary noise, the audio segment is unusable. If it does not contain noise, or only contains stationary noise, the audio segment is usable. The usable audio segment is determined as the first audio data.

[0068] Based on this, building and training a non-stationary noise detection model and performing noise evaluation on the original audio data can ensure that the data used is as clean and interference-free as possible, thereby improving the accuracy and effectiveness of subsequent processing.

[0069] Step S140: Detect the first audio data using a speaker number detection module.

[0070] The speaker number detection module is mainly used to determine whether there is only a single person speaking in the first audio data without other human voice interference.

[0071] To solve this problem, this embodiment proposes a speaker number detection model, the main task of which is to analyze audio data and determine whether it contains speech from multiple speakers. Only audio data with a single speaker can be used as clean corpus for training a speech separation model.

[0072] For example, Figure 4 FIG. 1 is a schematic diagram of a training process of a speaker number detection model provided by an exemplary embodiment of the present disclosure, such as Figure 4 As shown, the training process of the speaker number detection model may include:

[0073] Step S410: construct a training database.

[0074] The training database may include: a high-fidelity speech library, a stable noise library, and an interference speech noise library.

[0075] The high-fidelity speech library contains clear, noise-free human voice samples, which is the basis for building training data. The stationary noise library collects common stationary noises, and the interference speech noise library contains single-person and multi-person voices. These noises will be superimposed with clear speech in subsequent steps to simulate various auditory scenarios in real environments.

[0076] Step S420: superimposing noise according to different signal-to-noise ratios to obtain mixed speech.

[0077] For example, stationary noise and interfering speech can be superimposed on the high-fidelity speech time domain at different signal-to-noise ratios to construct a mixed speech as a model input, and the mixed speech is used to generate a corresponding training set and development set. The training label of the speech sample without interfering speech is marked as 1, while the training label of the speech sample superimposed with interfering speech is marked as 0.

[0078] The types of mixed speech may include: not superimposing stationary noise and interfering speech, superimposing stationary noise only, superimposing interfering speech only, and superimposing stationary noise and interfering speech at the same time.

[0079] Step S430: training a speaker number detection model.

[0080] The input of the speaker count detection model is mixed speech, and the mixed speech is processed by short-time Fourier transform to extract audio features. The audio features can be logmel spectrum, which can capture the basic structure and noise information of the audio.

[0081] Then, the audio features are input into a bidirectional long short-term memory network (BiLSTM) and a series of fully connected layers (FC). The BiLSTM can effectively process sequence data and capture context information, while the fully connected layer is used to extract features for classification from the output of the BiLSTM. The output of the model is a binary classification prediction of whether each sentence contains interfering speech. During the iterative training process, the cross entropy loss (CE loss) can be used to optimize the model.

[0082] The termination condition of the model training iteration is that the loss on the development set converges, indicating that the model has learned the ability to distinguish interfering speech.

[0083] After the model training is completed, the first audio data is input into the speaker number detection model to perform speaker number detection, and obtain a detection result of whether the current audio segment contains interfering speech. If the current audio segment contains interfering speech, the current audio segment is unusable. If the current audio segment does not contain interfering speech, the current audio segment is usable. The usable audio segment is determined as the second audio data.

[0084] Based on this, a speaker number detection model is constructed and trained to evaluate the number of speakers of the first audio data, which can ensure that the data used is as clean and interference-free as possible, thereby improving the accuracy and effectiveness of subsequent processing.

[0085] Step S150: performing speech enhancement processing on the second audio data.

[0086] Speech enhancement processing is performed on the second audio data that does not contain non-stationary noise and multi-person voice interference, and the audio quality is improved by eliminating noise and reverberation to obtain clearer speech.

[0087] In order to solve this problem, this embodiment proposes a speech noise reduction model, the main task of which is to eliminate stationary noise and reverberation in the second audio data.

[0088] For example, Figure 5 The following is a flow chart of a speech noise reduction model training process provided by an exemplary embodiment of the present disclosure. Figure 5 As shown, the training process of the speech noise reduction model may include:

[0089] Step S510: construct a training database.

[0090] The training database may include: a high-fidelity speech library and a stable noise library.

[0091] The high-fidelity speech library contains clear, noise-free human voice samples and is the basis for building training data. The stationary noise library collects common stationary noises. Stationary noise will be superimposed with clear speech in subsequent steps to simulate various auditory scenarios in real environments.

[0092] Step S520: superimposing noise according to different signal-to-noise ratios to obtain mixed speech.

[0093] For example, stationary noise can be superimposed on the high-fidelity speech in the time domain at different signal-to-noise ratios to construct a mixed speech as a model input, and the mixed speech is used to generate a corresponding training set and development set. The training label of the speech sample without stationary noise is marked as 1, while the training label of the speech sample with stationary noise is marked as 0.

[0094] The types of mixed speech may include: no stationary noise superimposed and stationary noise superimposed.

[0095] Step S530: training a speech noise reduction model.

[0096] The core of the speech denoising model is a neural network structure based on Conv-TasNet, which is an end-to-end processing model that is particularly suitable for processing time-domain audio signals. The main advantage of the speech denoising model is that it does not require feature extraction steps and can process audio signals directly in the time domain, which helps to retain more original audio details.

[0097] During the model training process, the input is mixed speech, which passes through Conv-TasNet and high-fidelity speech time domain signal to minimize the SI-SNR loss function. The termination condition of the model training iteration is that the loss on the development set converges, indicating that the model has learned how to separate clean speech from mixed speech.

[0098] After the model training is completed, the second audio data is input into the speech denoising model to perform speech enhancement, and the third audio data with noise and reverberation removed is obtained.

[0099] Based on this, constructing and training a speech noise reduction model and performing speech enhancement on the second audio data can ensure that the data used is as clean and interference-free as possible, thereby improving the accuracy and effectiveness of subsequent processing.

[0100] Step S160: Audio and video consistency detection.

[0101] In the process of audio and video processing, the most common problem is that the audio and lip movements in the video do not match.

[0102] To solve the above problem, this embodiment uses an audio and video consistency detection module to determine whether there is a biased frame in the audio and video and whether the lip movement is consistent with the voice content according to the lip movement and voice consistency of the face in the video.

[0103] For example, the Syncnet model can be used for audio and video consistency detection. Syncnet is an end-to-end network for determining audio and video lip synchronization. The model input is the audio and video to be detected, and the model outputs the offset frame between the audio and video and gives the confidence of the offset frame result. However, the original Syncnet model is mainly for English and has poor effect on Chinese. Therefore, it is necessary to fine-tune Syncnet to adapt to the Chinese environment, and use the fine-tuned Syncnet as the audio and video consistency detection model.

[0104] For example, Figure 6 This is a schematic diagram of the training process of the audio and video consistency detection model provided by the present disclosure as an example. Figure 6 As shown, the training process of the audio and video consistency detection model may include:

[0105] First, build a training database.

[0106] The training database may include: video data after face detection, and audio data after noise detection, speaker number detection, and speech enhancement processing.

[0107] These data have been denoised and only contain a single face. Audio and video data mainly in Chinese are selected. No additional labels are required for the training data because the training process mainly focuses on judging the synchronization of audio and video through the model itself.

[0108] Then, the training audio and video data is input into the Syncnet model, which will evaluate the synchronization between audio and video and output the frame deviation result and its confidence. According to the output, the audio and video data with a frame deviation result of 0 and a confidence level higher than 2 are marked as aligned and retained; while the audio and video data with a frame deviation result of not 0 or a frame deviation result of 0 but a confidence level lower than 2 are marked as misaligned and discarded.

[0109] Then, data enhancement is performed on the retained aligned audio and video data.

[0110] Exemplarily, data enhancement methods may include: adding noise, adjusting volume, distorting or rotating video frames, etc., to increase the robustness and adaptability of the model.

[0111] Then, the audio and video data after data enhancement are input into the Syncnet model again, the biased frame results and their confidence are output again, and the Syncnet model parameters are updated according to the biased frame results and their confidence.

[0112] Then, the output biased frame number is scanned to determine the minimized Euclidean distance, and the biased frame number is output.

[0113] After the model training is completed, the video data after face detection in step S120 and the third audio data after speech enhancement in step S150 are input into the trained Syncnet model. The model calculates all possible Euclidean distances within the range of biased frames (-10 to 10) to find the number of biased frames corresponding to the minimum distance and its confidence. If the biased frame result is 0 and the confidence is higher than 2, the audio and video are determined to be aligned and the data are retained; otherwise, the audio and video are considered to be not aligned and are discarded.

[0114] The retained audio and video data is used as multi-modal speech separation training corpus.

[0115] One or more technical solutions provided in the exemplary embodiments of the present disclosure select the audio and video clips that best meet the requirements through voice activity detection and face detection, thereby ensuring the accuracy and consistency of the data. In the face detection stage, only the video of a single complete face is retained, which greatly reduces the variables during model training and improves the accuracy of the correspondence between voice and visual data. In addition, the audio data is further cleaned through impact noise detection and speaker number detection to remove unnecessary noise and interference, thereby ensuring the purity of the audio. Finally, the consistency detection of the enhanced audio and the corresponding video not only verifies the synchronization accuracy, but also further ensures the reliability of the data through high-confidence bias frame number judgment.

[0116] Therefore, the multimodal speech processing method provided in the exemplary embodiments of the present disclosure can support the implementation of multimodal speech separation tasks.

[0117] Based on the above embodiments, the present disclosure also provides a multimodal speech processing method. Figure 7 A flowchart of a multimodal speech processing method provided by an exemplary embodiment of the present disclosure is shown in FIG. Figure 7 As shown, the method may include the following steps:

[0118] Step S710: Acquire the audio and video data to be processed, and perform data separation processing on the audio and video data to be processed to obtain original audio data and original video data.

[0119] In the embodiment, the audio and video data to be processed are obtained, and the audio and video data to be processed are preprocessed.

[0120] The first step of preprocessing is to separate the audio and video signals in the audio and video data to be processed to obtain original audio data and original video data.

[0121] Exemplarily, the audio and video signals can be separated by ffmpeg, which is a widely used multimedia framework that can process audio and video data in various formats and supports conversion and separation between audio and video.

[0122] Furthermore, the second step of preprocessing is to use voice activity detection (VAD) technology to identify the valid speech part in the original audio data. VAD can distinguish between background noise and human voice, and can accurately identify the original audio segment that needs further analysis. In practical applications, the length of the original audio segment can be controlled within 2s to 20s, which can ensure the integrity of the voice information and facilitate processing and analysis.

[0123] Based on this, by preprocessing the audio and video data to be processed, the original audio data and the original video data can be separated, and the effective voice part in the original audio data can be detected to provide data for subsequent audio detection and video detection.

[0124] Step S720: Perform face detection processing on the original video data to obtain initial video data.

[0125] In an embodiment, a face detection algorithm can be used to perform face detection processing on the original video data, identify and locate faces from the video frames of the original video data, and then determine the number and position of faces in the video. Facial expressions and other related features can also be evaluated to provide necessary data support for subsequent video processing.

[0126] Step S730: Perform audio detection processing on the original audio data to obtain initial audio data.

[0127] In an embodiment, the audio detection process may include noise detection, speaker number detection and speech enhancement processing. By performing audio detection processing on the original audio data, the speech of the target speaker and the background noise can be identified and separated, ensuring that the voice of the target speaker is extracted and processed, thereby improving the accuracy and efficiency of speech recognition.

[0128] Step S740: Perform consistency check processing on the initial video data and the initial audio data, and output the target audio data and the target video data when the consistency check condition is met.

[0129] In the embodiment, during the audio and video processing, the most common problem is that the audio and lip movements in the video do not match. To solve this problem, it is necessary to perform consistency detection processing on the initial video data and the initial audio data.

[0130] By comparing the timestamps and content of audio and video data, the synchronization of the two in expressing the same information is ensured. The consistency check not only helps to confirm the accurate synchronization of audio and video streams, but also detects any possible mismatches, such as whether the lip shape in the video corresponds to the sound in the audio. Only when the consistency check conditions are met will the target audio data and target video data be output for subsequent use as multi-modal speech separation training corpus.

[0131] Based on this, the original audio and video data is separated and processed to obtain separate audio and video data streams so that the audio and video data can be processed and analyzed independently. Secondly, through face detection, faces and related features in the video, such as the position and number of faces, can be identified. This provides visual positioning information for the speaker in the video, which is convenient for subsequent combination with the sound source positioning in the audio signal. In the audio detection stage, by identifying different sound components and speakers, this processing is crucial for identifying and separating background noise from the voice of the main speaker. Finally, consistency detection ensures the matching of audio and video data in time and content, and ensures the synchronization of visual information and auditory information, which can increase the accuracy of the signal separation algorithm, thereby accurately separating the voice of a single speaker from a multi-person environment. Therefore, the multimodal speech processing method provided in the exemplary embodiment of the present disclosure can support the implementation of multimodal speech separation tasks.

[0132] Based on the above embodiment, in another embodiment provided by the present disclosure, the above step S720 may include:

[0133] Perform face detection on the original video data to obtain the number of faces and face key points in each video frame of the original video data;

[0134] When the number of faces is equal to 1 and the facial key points are continuously visible, the corresponding video frames are retained, and the retained video frames are determined as initial video data.

[0135] In an embodiment, a face detection module may be used to detect whether there are faces in the original video data, the number of faces, and whether the faces are completely visible throughout the entire process.

[0136] For example, MTCNN (Multi-task Cascaded Convolutional Networks) can be used as a face detection model to perform face detection on raw video data. MTCNN is an efficient face detection tool that can detect the position and number of faces in video frames and provide detailed information on facial key points, including important features such as eyes, nose, and mouth. Key points can be used to characterize the direction and expression state of the face.

[0137] During the processing, each video frame in the original video data is input into the MTCNN model in turn, and the model outputs the face frame and face key point information.

[0138] In order to ensure the consistency and availability of the data, it is also necessary to check whether the face in the video clip continues to appear. Specifically, it can be verified whether the face frame is continuously equal to 1 (i.e., a face is always detected) throughout the video clip, and whether the five main facial key points (eyes, nose, mouth, etc.) are always visible. This step can help exclude video clips with incomplete facial information due to factors such as mask occlusion and head rotation.

[0139] If the face frame is continuously equal to 1 and the five faces are not continuously visible, the corresponding video frame is retained.

[0140] If the face frame is not continuously equal to 1 or the five facial key points are not continuously visible, the corresponding video frame is removed.

[0141] In addition, in order to simplify the processing steps and reduce the computational burden, the face detection model does not involve face recognition, that is, it does not consider whether the faces in the video are the same person. If different people's faces appear in the video data, the audio and video inconsistency problem caused by it will be identified and processed in the subsequent audio and video consistency detection module.

[0142] Based on this, the face detection module can remove video frames in the original video data where the face frame is not continuously equal to 1 or the five face key points are not continuously visible, further reducing the amount of calculation for subsequent processing and reducing the computational burden.

[0143] Based on the above embodiment, in another embodiment provided by the present disclosure, the above step S730 may include:

[0144] Performing noise detection on the original audio data to obtain first audio data;

[0145] Detecting the first audio data to obtain second audio data;

[0146] Perform speech enhancement processing on the second audio data to obtain initial audio data.

[0147] In an embodiment, the audio detection process may include noise detection, speaker number detection, and speech enhancement processing, which steps together improve the quality and usability of the audio data.

[0148] First, the original audio data is subjected to noise detection through the non-stationary noise detection module, which is specially designed to identify and process non-stationary noise in audio signals, such as traffic noise, crowd noise, etc. Non-stationary noise refers to noise whose statistical properties (such as mean and variance) change over time. It is very common in real environments and difficult to process. By effectively identifying and pre-processing these noises, the audio quality can be significantly improved.

[0149] Next, the audio data after preliminary noise processing will be submitted to the speaker number detection module. This module can identify and distinguish the number of different speakers contained in the audio, which is particularly important for audio analysis in multi-speaker scenarios. By accurately detecting the number of speakers, the system can track and separate sounds more effectively, especially in application scenarios where it is necessary to extract the voice of a specific speaker from a multi-speaker conversation.

[0150] Finally, the audio data detected by the number of speakers is subjected to speech enhancement. Speech enhancement aims to further clarify the speech signal, reduce or eliminate the remaining noise, and improve the intelligibility and listening quality of the speech. This step is particularly critical to improving the performance of the speech recognition system, because clear speech signals can significantly improve the accuracy and efficiency of recognition.

[0151] Based on this, non-stationary noise detection can effectively cope with dynamically changing background noise and provide purer audio input. Secondly, speaker number detection provides key information for sound separation and personalized processing. Finally, speech enhancement technology ensures the high definition and high quality of the final output audio. Therefore, the combination of the above audio detection technologies greatly improves the practicality and application scope of audio data, making it efficient and accurate in various noisy and complex environments.

[0152] Based on the above embodiment, in another embodiment provided by the present disclosure, performing noise detection on the original audio data to obtain the first audio data may include:

[0153] The original audio data is input into a pre-constructed non-stationary noise detection model for noise detection, and the noise types contained in the multiple audio segments in the original audio data are obtained; the noise types include: no noise, stationary noise and non-stationary noise;

[0154] When the noise type of the audio segment includes non-stationary noise, removing the corresponding audio segment;

[0155] In the case that the noise type of the audio segment is no noise or contains only stationary noise, the corresponding audio segment is retained, and the retained audio segment is determined as the first audio data.

[0156] In the embodiment, in the current speech processing technology, whether it is a single-channel noise reduction module based on a deep learning neural network or a single-channel noise reduction module based on speech signal processing technology, both have shown excellent performance in suppressing stationary noises such as wind noise. However, for non-stationary noises such as whistles or percussion sounds, the noise reduction effect is often not ideal. Due to its unpredictable nature, non-stationary noise brings additional challenges to accurate noise identification and elimination.

[0157] In order to solve this problem, this embodiment proposes a non-stationary noise detection model, the main task of which is to analyze audio data and determine whether it contains non-stationary noise. Only audio data that does not contain noise or only contains stationary noise and has passed through the noise reduction module can be used as clean corpus for speech separation model training.

[0158] Exemplarily, the training process of the non-stationary noise detection model can refer to the above steps S310 to S330.

[0159] After the model training is completed, the original audio data is input into the non-stationary noise detection model to perform noise detection, and obtain a detection result of whether the current audio segment contains non-stationary noise. If it contains non-stationary noise, the audio segment is unusable. If it does not contain noise, or only contains stationary noise, the audio segment is usable. The usable audio segment is determined as the first audio data.

[0160] Based on this, building and training a non-stationary noise detection model and performing noise evaluation on the original audio data can ensure that the data used is as clean and interference-free as possible, thereby improving the accuracy and effectiveness of subsequent processing.

[0161] Based on the above embodiment, in another embodiment provided by the present disclosure, the above detecting the first audio data to obtain the second audio data may include:

[0162] Inputting the first audio data into a pre-constructed speaker number detection model to perform audio detection, and obtaining the number of speakers corresponding to the multiple audio segments in the first audio data;

[0163] If the number of speakers in the audio clip is greater than one, remove the corresponding audio clip;

[0164] When the number of speakers in the audio segment is one, the corresponding audio segment is retained, and the retained audio segment is determined as the second audio data.

[0165] In the embodiment, a speaker number detection module may be used to determine whether the first audio data contains only the speech of a single person without interference from other human voices.

[0166] To solve this problem, this embodiment proposes a speaker number detection model, the main task of which is to analyze audio data and determine whether it contains speech from multiple speakers. Only audio data with a single speaker can be used as clean corpus for training a speech separation model.

[0167] Exemplarily, the training process of the speaker number detection model can refer to the above steps S410 to S430.

[0168] After the model training is completed, the first audio data is input into the speaker number detection model to perform speaker number detection, and obtain a detection result of whether the current audio segment contains interfering speech. If interfering speech is contained, it indicates that the number of speakers in the voice segment is greater than 1, and the audio segment is unusable. If interfering speech is not contained, it indicates that the number of speakers in the voice segment is equal to 1, and the audio segment is usable. The usable audio segment is determined as the second audio data.

[0169] Based on this, a speaker number detection model is constructed and trained to evaluate the number of speakers of the first audio data, which can ensure that the data used is as clean and interference-free as possible, thereby improving the accuracy and effectiveness of subsequent processing.

[0170] Based on the above embodiment, in another embodiment provided by the present disclosure, performing speech enhancement processing on the second audio data to obtain initial audio data may include:

[0171] Inputting the second audio data into a pre-constructed speech noise reduction model for speech enhancement processing, and obtaining noise types respectively contained in the multiple audio segments in the second audio data; the noise types include: no noise and stable noise;

[0172] When the noise type of the audio segment includes stationary noise, removing the corresponding audio segment;

[0173] When the noise type of the audio segment is no noise, the corresponding audio segment is retained, and the retained audio segment is determined as initial audio data.

[0174] In the embodiment, speech enhancement processing is performed on the second audio data that does not contain non-stationary noise and multi-person voice interference, and the audio quality is improved by eliminating noise and reverberation to obtain clearer speech.

[0175] In order to solve this problem, this embodiment proposes a speech noise reduction model, the main task of which is to eliminate stationary noise and reverberation in the second audio data.

[0176] Exemplarily, the training process of the speech noise reduction model may refer to steps S510 to S530 above.

[0177] After the model training is completed, the second audio data is input into the speech denoising model to perform speech enhancement to obtain the initial audio data with noise and reverberation removed.

[0178] Based on this, constructing and training a speech noise reduction model and performing speech enhancement on the second audio data can ensure that the data used is as clean and interference-free as possible, thereby improving the accuracy and effectiveness of subsequent processing.

[0179] Based on the above embodiment, in another embodiment provided by the present disclosure, the above step S740 may include:

[0180] Inputting the initial video data and the initial audio data into the audio and video consistency detection model to obtain the offset frame and the confidence of the offset frame result between the initial video data and the initial audio data;

[0181] When the offset frame and the confidence of the offset frame result meet the consistency verification condition, the corresponding initial video data and initial audio data are retained, and the initial video data and initial audio data are determined as the target audio data and target video data respectively.

[0182] In the embodiment, during the audio and video processing, the most common problem is that the audio and lip movements in the video do not match.

[0183] To solve the above problem, this embodiment uses an audio and video consistency detection module to determine whether there is a biased frame in the audio and video and whether the lip movement is consistent with the voice content according to the lip movement and voice consistency of the face in the video.

[0184] For example, the Syncnet model can be used for audio and video consistency detection. Syncnet is an end-to-end network for determining audio and video lip synchronization. The model input is the audio and video to be detected, and the model outputs the offset frame between the audio and video and gives the confidence of the offset frame result. However, the original Syncnet model is mainly for English and has poor effect on Chinese. Therefore, it is necessary to fine-tune Syncnet to adapt to the Chinese environment, and use the fine-tuned Syncnet as the audio and video consistency detection model.

[0185] For example, the training process of the audio and video consistency detection model can be found in the above Figure 6 Related content.

[0186] After the model training is completed, the initial video data after face detection and the initial audio data after speech enhancement are input into the trained Syncnet model. The model calculates all possible Euclidean distances within the range of offset frames (-10 to 10) to find the offset frame number and its confidence level corresponding to the minimum distance. If the offset frame result is 0 and the confidence level is higher than 2, the audio and video are considered aligned and the data are retained; otherwise, the audio and video are considered not aligned and are discarded.

[0187] The retained audio and video data are respectively determined as target audio data and target video data as multi-modal speech separation training corpus.

[0188] One or more technical solutions provided in the exemplary embodiments of the present disclosure select the audio and video clips that best meet the requirements through voice activity detection and face detection, thereby ensuring the accuracy and consistency of the data. In the face detection stage, only the video of a single complete face is retained, which greatly reduces the variables during model training and improves the accuracy of the correspondence between voice and visual data. In addition, the audio data is further cleaned through impact noise detection and speaker number detection to remove unnecessary noise and interference, thereby ensuring the purity of the audio. Finally, the consistency detection of the enhanced audio and the corresponding video not only verifies the synchronization accuracy, but also further ensures the reliability of the data through high-confidence bias frame number judgment.

[0189] Therefore, the multimodal speech processing method provided in the exemplary embodiments of the present disclosure can support the implementation of multimodal speech separation tasks.

[0190] The above mainly introduces the solutions provided by the exemplary embodiments of the present disclosure. It is understandable that in order to implement the above functions, the electronic device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present disclosure.

[0191] The exemplary embodiments of the present disclosure can divide the functional units of the electronic device according to the above method examples. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the exemplary embodiments of the present disclosure is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0192] In the case of dividing each functional module according to each function, an exemplary embodiment of the present disclosure provides a multimodal speech processing device, which may be a server or a chip applied to a server. Figure 8 This is a schematic block diagram of the functional modules of a multimodal speech processing device provided by the present disclosure. Figure 8 As shown, the multimodal speech processing device 800 includes:

[0193] The data acquisition module 810 is used to acquire the audio and video data to be processed, and perform data separation processing on the audio and video data to be processed to obtain original audio data and original video data;

[0194] The data processing module 820 is used to perform face detection processing on the original video data to obtain initial video data;

[0195] The data processing module 820 is further used to perform audio detection processing on the original audio data to obtain initial audio data;

[0196] The data processing module 820 is further configured to perform consistency detection processing on the initial video data and the initial audio data, and output target audio data and target video data when consistency verification conditions are met.

[0197] In another embodiment provided by the present disclosure, the data processing module 820 is also used to perform face detection on the original video data through a face detection module to obtain the number of faces and face key points in each video frame in the original video data; when the number of faces is equal to 1 and the face key points are continuously visible, the corresponding video frames are retained, and the retained video frames are determined as initial video data.

[0198] In another embodiment provided by the present disclosure, the data processing module 820 is further used to perform noise detection on the original audio data through a non-stationary noise detection module to obtain first audio data; detect the first audio data through a speaker number detection module to obtain second audio data; and perform speech enhancement processing on the second audio data to obtain initial audio data.

[0199] In another embodiment provided by the present disclosure, the data processing module 820 is further used to input the original audio data into a pre-constructed non-stationary noise detection model for noise detection, and obtain the noise types respectively contained in multiple audio segments in the original audio data; the noise types include: no noise, stationary noise and non-stationary noise; when the noise type of the audio segment contains non-stationary noise, remove the corresponding audio segment; when the noise type of the audio segment is no noise or only contains stationary noise, retain the corresponding audio segment, and determine the retained audio segment as the first audio data.

[0200] In another embodiment provided by the present disclosure, the data processing module 820 is further used to input the first audio data into a pre-constructed speaker number detection model for audio detection to obtain the number of speakers corresponding to multiple audio segments in the first audio data; when the number of speakers in the audio segment is greater than one, remove the corresponding audio segment; when the number of speakers in the audio segment is one, retain the corresponding audio segment, and determine the retained audio segment as the second audio data.

[0201] In another embodiment provided by the present disclosure, the data processing module 820 is also used to input the second audio data into a pre-constructed speech noise reduction model for speech enhancement processing, and obtain the noise types respectively contained in multiple audio segments in the second audio data; the noise types include: no noise and smooth noise; when the noise type of the audio segment includes smooth noise, remove the corresponding audio segment; when the noise type of the audio segment is no noise, retain the corresponding audio segment, and determine the retained audio segment as the initial audio data.

[0202] In another embodiment provided by the present disclosure, the data processing module 820 is also used to input the initial video data and the initial audio data into an audio and video consistency detection model to obtain the frame deviation and frame deviation result confidence between the initial video data and the initial audio data; when the frame deviation and the frame deviation result confidence meet the consistency verification conditions, retain the corresponding initial video data and initial audio data, and determine the initial video data and the initial audio data as target audio data and target video data, respectively.

[0203] The exemplary embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to cause the electronic device to perform the method according to the embodiment of the present disclosure when executed by the at least one processor.

[0204] The exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform the method according to the embodiments of the present disclosure.

[0205] Fig. 9The block diagram of the electronic device provided as an example of the present disclosure, the block diagram of the electronic device 900 that can be used as the server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0206] like Fig. 9 As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0207] A plurality of components in the electronic device 900 are connected to the I / O interface 905, including: an input unit 906, an output unit 907, a storage unit 908, and a communication unit 909. The input unit 906 may be any type of device capable of inputting information to the electronic device 900, and the input unit 906 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 907 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 908 may include, but is not limited to, a disk, an optical disk. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0208] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above. Each of the methods described above may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via the ROM 902 and / or the communication unit 909.

[0209] Fig.10 This is a schematic diagram of an exemplary computer program product provided by the present disclosure. The exemplary embodiment of the present disclosure also provides a computer program product 1000, including a computer program 1001, wherein the computer program 1001 is used to enable the computer to perform a method according to an embodiment of the present disclosure when executed by a processor of the computer.

[0210] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0211] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0212] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0213] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0214] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0215] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0216] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present disclosure is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a terminal, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD).

[0217] Although the present disclosure has been described in conjunction with specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present disclosure. Accordingly, this specification and the drawings are merely exemplary illustrations of the present disclosure as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present disclosure. Obviously, those skilled in the art may make various modifications and variations to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A multimodal speech processing method, characterized in that: The method comprises: Acquire the audio and video data to be processed, and perform data separation processing on the audio and video data to be processed to obtain original audio data and original video data; Performing face detection processing on the original video data to obtain initial video data; Performing audio detection processing on the original audio data to obtain initial audio data; The initial video data and the initial audio data are subjected to consistency check processing, and when consistency check conditions are met, target audio data and target video data are output.

2. The method according to claim 1, characterized in that The performing face detection processing on the original video data to obtain initial video data includes: Perform face detection on the original video data to obtain the number of faces and face key points in each video frame of the original video data; When the number of faces is equal to 1 and the facial key points are continuously visible, the corresponding video frames are retained, and the retained video frames are determined as initial video data.

3. The method according to claim 1, characterized in that The performing audio detection processing on the original audio data to obtain initial audio data includes: Performing noise detection on the original audio data to obtain first audio data; Detecting the first audio data to obtain second audio data; Perform speech enhancement processing on the second audio data to obtain initial audio data.

4. The method according to claim 3, characterized in that The performing noise detection on the original audio data to obtain the first audio data includes: Inputting the original audio data into a pre-constructed non-stationary noise detection model for noise detection, and obtaining the noise types respectively contained in the multiple audio segments in the original audio data; the noise types include: no noise, stationary noise, and non-stationary noise; When the noise type of the audio segment includes non-stationary noise, removing the corresponding audio segment; In a case where the noise type of the audio segment is no noise or only contains stationary noise, the corresponding audio segment is retained, and the retained audio segment is determined as the first audio data.

5. The method according to claim 3, characterized in that: The detecting the first audio data to obtain the second audio data includes: Inputting the first audio data into a pre-constructed speaker number detection model to perform audio detection, and obtaining the number of speakers corresponding to the multiple audio segments in the first audio data; If the number of speakers in the audio segment is greater than one, remove the corresponding audio segment; When the number of speakers in the audio segment is one, the corresponding audio segment is retained, and the retained audio segment is determined as the second audio data.

6. The method according to claim 3, characterized in that The performing speech enhancement processing on the second audio data to obtain initial audio data includes: Inputting the second audio data into a pre-constructed speech noise reduction model for speech enhancement processing, and obtaining noise types respectively contained in a plurality of audio segments in the second audio data; the noise types include: no noise and stable noise; When the noise type of the audio segment includes stationary noise, removing the corresponding audio segment; In the case that the noise type of the audio segment is no noise, the corresponding audio segment is retained, and the retained audio segment is determined as the initial audio data.

7. The method according to claim 1, characterized in that The performing consistency detection processing on the initial video data and the initial audio data, and outputting target audio data and target video data when a consistency check condition is met, comprises: Inputting the initial video data and the initial audio data into an audio and video consistency detection model to obtain a frame deviation and a confidence level of a frame deviation result between the initial video data and the initial audio data; When the biased frame and the confidence of the biased frame result meet the consistency verification condition, the corresponding initial video data and initial audio data are retained, and the initial video data and the initial audio data are determined as the target audio data and the target video data respectively.

8. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images