Voice separation method, system, vehicle and storage medium

Abstract

A voice separation method, a system, a vehicle and a storage medium. The method comprises: collecting an image sequence and a mixed voice signal of at least one object in a space of a vehicle (100); and performing voice signal separation on the basis of the image sequence and the mixed voice signal, so as to determine a voice signal of each object in a control of the vehicle (100).

Description

Voice separation method, system, vehicle and storage medium

[0001] Cross-reference to Related Applications

[0002] This application claims priority to the application filed on September 30, 2024, in the China Patent Office, with the application number 202411392623.8, and the patent name: Voice separation method, system, vehicle and storage medium, the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the technical field of voice interaction, in particular to a voice separation method, system, vehicle and storage medium. BACKGROUND

[0004] With the development of intelligent vehicles, the use of vehicle-mounted voice interaction technology has also gradually become widespread. However, in the related technology, in order to perform effective voice separation, a distributed microphone array is often used, which relies on the number and position of microphones, resulting in high hardware costs. TECHNICAL SOLUTION

[0005] The present application provides a voice separation method, system, vehicle and storage medium, which aims to solve the problem of the need to assemble a large number of microphones to improve hardware costs in the above technical solution for effective voice separation.

[0006] In a first aspect, the present application provides a voice separation method, comprising:

[0007] Collecting an image sequence of at least one object in the vehicle space and a mixed voice signal;

[0008] Performing voice signal separation according to the image sequence and the mixed voice signal to determine the voice signal of each object in the vehicle control.

[0009] Optionally, the method further comprises:

[0010] In response to the image quality corresponding to the face region of the object in the image sequence meeting the preset quality condition, extracting a face image sequence corresponding to the face region of the object in the image sequence;

[0011] The voice signal separation according to the image sequence and the mixed voice signal to determine the voice signal of each object in the vehicle control comprises:

[0012] Performing voice signal separation according to the face image sequence and the mixed voice signal to determine the voice signal of each object in the vehicle control.

[0013] Optionally, the method further comprises:

[0014] In response to the image quality not satisfying the preset quality condition, performing speech signal separation according to the mixed speech signal to determine the speech signal of each object in the vehicle control.

[0015] Optionally, the method further comprises:

[0016] In response to identifying the object face region in the image sequence, determining that the image quality corresponding to the object face region satisfies the preset quality condition;

[0017] In response to not identifying the object face region in the image sequence, determining that the image quality corresponding to the object face region does not satisfy the preset quality condition, and marking that the image sequence is unusable.

[0018] Optionally, the method further comprises:

[0019] Converting the mixed speech signal to obtain a spectrogram corresponding to the mixed speech signal, wherein the sampling frequency corresponding to the mixed speech signal in the spectrogram includes multiple frames of speech information;

[0020] The speech signal separation according to the face image sequence and the mixed speech signal to determine the speech signal of each object in the vehicle control comprises:

[0021] Performing speech signal separation according to the face image sequence and the spectrogram to determine the speech signal of each object in the vehicle control.

[0022] Optionally, the speech signal separation according to the face image sequence and the mixed speech signal to determine the speech signal of each object in the vehicle control comprises:

[0023] Aligning the face image sequence and the spectrogram;

[0024] Performing speech signal separation processing according to the aligned face image sequence and the spectrogram to determine the speech signal of each object in the vehicle control.

[0025] Optionally, the aligning the face image sequence and the spectrogram comprises:

[0026] In response to the image acquisition frequency of the image sequence being less than a preset sampling frequency of the spectrogram, performing interpolation processing on the face image sequence to align the preset sampling frequency and the image acquisition frequency.

[0027] Optionally, the method is applied to a speech separation model, and the speech separation model comprises:

[0028] a fusion processing layer, configured to perform feature fusion processing on input features to obtain fused features, the input features being image coding features corresponding to a face image sequence and speech coding features corresponding to the mixed speech signal, or the input features being the speech coding features corresponding to the mixed speech signal;

[0029] a speech separation layer, configured to perform speech separation on the fused features to output separated speech signals of different dimensions.

[0030] Optionally, the fusion processing layer comprises:

[0031] an up-sampling alignment layer, configured to perform coding feature alignment processing on the speech coding features and the image coding features, and perform concatenation on the speech coding features and the image coding features after the alignment processing to obtain first output features, or configured to perform processing on the speech coding features to obtain the first output features;

[0032] a fully connected layer, configured to perform feature fusion on the first output features to output fused features.

[0033] Optionally, the speech separation layer comprises:

[0034] an up-sampling convolution layer, configured to perform up-sampling convolution processing on the fused features to obtain a convolution result;

[0035] an output layer, configured to perform mask mapping processing on the convolution result to obtain corresponding mask information of each dimension, and convert the mask information into separated speech signals.

[0036] Optionally, the up-sampling convolution layer comprises a first sub-layer and a plurality of second sub-layers connected in sequence.

[0037] an output end of the first sub-layer is connected to a first input end of one of the second sub-layers;

[0038] a second output end of the second sub-layer is configured to input speech coding features of the same size as the features of the first input end, so as to perform concatenation processing on the input of the first input end and the input of the second input end, and perform up-sampling convolution processing on the concatenated input and output the processed input;

[0039] an output end of one of the second sub-layers is connected to an input end of the output layer.

[0040] Optionally, the output layer comprises:

[0041] an activation function layer, configured to map the convolution result to a mask value range;

[0042] A mask layer is configured to perform multi-dimensional mask processing on the convolution result in the mask value range, and output corresponding mask information of each dimension.

[0043] A conversion layer is configured to perform point multiplication processing according to the corresponding mask information of each dimension and the spectrogram, and output multi-dimensional separated speech signals, each of which is a speech signal of each object in the vehicle control.

[0044] Optionally, the speech separation model further comprises a speech encoding layer connected with the fusion processing layer.

[0045] The speech encoding layer is configured to perform encoding processing on the mixed speech signal, and output corresponding speech encoding features.

[0046] Optionally, the speech encoding layer comprises a plurality of dilated convolution layers configured to output speech encoding features of different sizes.

[0047] Optionally, the speech separation model further comprises an image encoding layer connected with the fusion processing layer.

[0048] The image encoding layer is configured to encode the face image sequence, and output corresponding image encoding features.

[0049] In a second aspect, the present application provides a vehicle speech processing system, comprising an image acquisition device, a processing device and an audio acquisition device, wherein the audio acquisition device and the image acquisition device are connected with the processing device respectively.

[0050] The image acquisition device is configured to acquire an image sequence of at least one object in a vehicle space.

[0051] The audio acquisition device is configured to acquire a mixed speech signal of at least one object in the vehicle space.

[0052] The processing device is configured to perform speech signal separation according to the image sequence and the mixed speech signal, so as to determine a speech signal of each object in the vehicle control.

[0053] In a third aspect, the present application provides a vehicle comprising the vehicle speech processing system, wherein the processing device in the vehicle speech processing system comprises:

[0054] One or more processors;

[0055] A memory; and

[0056] One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the speech separation method.

[0057] In a fourth aspect, the present application provides a computer readable storage medium having stored thereon computer instructions, which are loaded by a processor to perform the steps in any of the speech separation methods.

[0058] The present application provides a speech separation method, system, vehicle and storage medium, wherein an image sequence and a mixed speech signal of at least one object in a vehicle space are collected; speech signal separation is performed according to the image sequence and the mixed speech signal to determine the speech signal of each object in the vehicle control. The present scheme discards the traditional scheme of configuring multiple speech collection devices to collect speech to ensure the accuracy of speech separation. Through the collection of corresponding image sequences combined with mixed speech signals, the accuracy of speech separation is improved, the dependence of speech separation on speech collection equipment is reduced, and thus the configuration of speech collection devices can be reduced and the cost is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0060] FIG. 1 is a schematic diagram of a scene of a speech separation method according to an embodiment of the present application;

[0061] FIG. 2 is a flowchart of an embodiment of a speech separation method according to an embodiment of the present application;

[0062] FIG. 3 is a flowchart of another embodiment of a speech separation method according to an embodiment of the present application;

[0063] FIG. 4 is a schematic diagram of some embodiments of judging whether the image of the face region of the object meets the preset quality condition according to an embodiment of the present application;

[0064] FIG. 5 is a flowchart of another embodiment of a speech separation method according to an embodiment of the present application;

[0065] FIG. 6 is a flowchart of a speech separation method according to an embodiment of the present application;

[0066] FIG. 7 is a schematic diagram of one structure of a speech encoder network according to an embodiment of the present application;

[0067] FIG. 8 is a schematic diagram of one structure of an image encoder network according to an embodiment of the present application;

[0068] Fig. 9 is a schematic diagram of one of the speech separation model structures in the speech separation method provided by the present application;

[0069] Fig. 10 is a schematic diagram of one of the implementation structures of the vehicle speech processing system provided by the present application;

[0070] Fig. 11 is a schematic diagram of one of the implementation structures of the vehicle provided by the present application.

[0071] Embodiments of the present application

[0072] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0073] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application. In addition, the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined as "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0074] In the embodiments of the present application, the association relationship between the associated objects described by "and / or" represents that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / ", unless otherwise specified, generally represents a "or" relationship between the associated objects before and after it.

[0075] In this application, the word "exemplary" is used to mean "serving as an example, instance, or illustration." Any implementation described as "exemplary" in this application is not necessarily to be construed as preferred or advantageous over other implementations. The following description is presented to enable any person skilled in the art to make and use the application. In the following description, for purposes of explanation, specific details are set forth to provide a thorough understanding of the present application. It will be apparent to one skilled in the art, however, that the present application can be practiced without using these specific details. In other instances, well-known structures and processes are not elaborated in order not to obscure the description of the present application with unnecessary details. Thus, the present application is not intended to be limited by the embodiments shown, but is to be accorded with the widest scope consistent with the principles and features disclosed herein.

[0076] The embodiments of the present application provide a speech separation method, system, vehicle and computer readable storage medium (the computer readable storage medium can be referred to as storage medium in the following).

[0077] The speech separation method in the embodiments of the present application is applied to a vehicle speech processing system, the vehicle speech processing system is arranged in a vehicle, the vehicle is provided with one or more processors, a memory and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to realize the speech separation method.

[0078] As shown in FIG. 1, FIG. 1 is a scene schematic diagram of the speech separation method in the embodiments of the present application, the speech separation scene in the embodiments of the present application includes a vehicle 100 (the vehicle 100 is integrated with a vehicle speech processing system), and a computer readable storage medium corresponding to the speech separation method is run in the vehicle 100 to execute the steps of the speech separation method.

[0079] It can be understood that the vehicle in the scene of the speech separation method shown in FIG. 1, or the device contained in the vehicle does not constitute a limitation on the embodiments of the present application, that is, the number and type of devices contained in the scene of the speech separation method, or the number and type of devices contained in each device do not affect the overall implementation of the technical solutions in the embodiments of the present application, and can be regarded as equivalent replacement or derivation of the technical solutions claimed in the embodiments of the present application.

[0080] The vehicle 100 in the embodiments of the present application is mainly used for: collecting an image sequence of at least one object in a vehicle space and a mixed speech signal; performing speech signal separation according to the image sequence and the mixed speech signal to determine the speech signal of each object in the vehicle control.

[0081] The vehicle 100 in the embodiments of the present application can be an independent vehicle, or a vehicle network or vehicle cluster composed of vehicles. For example, the vehicle 100 described in the embodiments of the present application includes but is not limited to a computer, a network host, a single network vehicle, a plurality of network vehicle sets, or a cloud vehicle composed of a plurality of vehicles. The cloud vehicle is composed of a large number of computers or network vehicles based on cloud computing.

[0082] Those skilled in the art can understand that the application environment shown in FIG. 1 is only one application scenario of the present application scheme, and does not constitute a limitation on the application scenario of the present application scheme. Other application environments can include more or fewer vehicles or vehicle network connection relationships than those shown in FIG. 1. For example, only one vehicle is shown in FIG. 1. It can be understood that the scene of the voice separation method can also include one or more other vehicles, which is not limited here. The vehicle 100 can also include a memory for storing data, such as storing image information obtained by shooting.

[0083] In addition, the vehicle 100 in the scene of the voice separation method of the present application can be provided with a display device, or the vehicle 100 can be communicatively connected to an external display device without a display device. The display device is used to output the result of the execution of the voice separation method in the vehicle. The vehicle 100 can access the background database 200 (the background database can be a local memory of the vehicle, and the background database can also be set in the cloud). The background database 200 stores information related to voice separation.

[0084] It should be noted that the voice separation method scene shown in FIG. 1 is only an example. The voice separation method scene described in the embodiments of the present application is used to more clearly illustrate the technical solutions of the embodiments of the present application, and does not constitute a limitation on the technical solutions provided by the embodiments of the present application.

[0085] Based on the above-mentioned voice separation method scene, an embodiment of the voice separation method is provided.

[0086] As shown in FIG. 2, an embodiment flowchart of the voice separation method in the embodiments of the present application is shown. The voice separation method includes steps S201-S202:

[0087] S201, collecting an image sequence of at least one object in a vehicle space and a mixed voice signal.

[0088] Specifically, the image sequence includes a plurality of continuously collected images, and the image sequence is collected by an image collection device according to a preset image collection frequency.

[0089] Specifically, the mixed voice signal includes voice signals of multiple objects in the vehicle space, and the mixed voice signal can be collected by at least one microphone.

[0090] S202, performing voice signal separation on the image sequence and the mixed voice signal to determine a voice signal of each object in the vehicle control.

[0091] Specifically, in some embodiments of the present application, the voice signal separation can be performed on the image sequence and the mixed voice signal by a preset voice separation model. It can be understood that in the embodiments of the present application, the voice separation model can be obtained by training.

[0092] Specifically, in one embodiment of the present application, face recognition is further performed on the image sequence, the quality of the image sequence is determined according to the face recognition, and whether the image sequence is used to participate in voice signal separation is further determined according to the quality.

[0093] Specifically, referring to FIG. 3, FIG. 3 is a flowchart of an embodiment of another voice separation method provided by the embodiments of the present application, which specifically includes steps S301-S303:

[0094] S301, collecting an image sequence of at least one object in a vehicle space and a mixed voice signal.

[0095] S302, in response to an image quality corresponding to an object face region in the image sequence satisfying a preset quality condition, extracting a face image sequence corresponding to the object face region in the image sequence.

[0096] Specifically, in the embodiments of the present application, after the image sequence is obtained, the object face region in the image sequence is recognized, in response to recognizing the object face region in the image sequence, it is determined that the image quality corresponding to the object face region satisfies the preset quality condition; in response to not recognizing the object face region in the image sequence, it is determined that the image quality corresponding to the object face region does not satisfy the preset quality condition, and the image sequence is marked as unusable.

[0097] In some other embodiments of the present application, the preset quality condition can further include face integrity, face clarity, etc. of the object face region.

[0098] For example, referring to FIG. 4, FIG. 4 is some embodiments of judging whether the image of the object face region satisfies the preset quality condition provided by the embodiments of the present application, which specifically includes steps:

[0099] (1) Use an image sequence collection device (in-vehicle device), in this embodiment, a camera, to respectively obtain the region image sequence of the driver and the passenger.

[0100] (2):Judge whether the ROI region of the image is readable, wherein the criterion for judging whether it is readable is whether a face region can be detected.

[0101] (3):If the ROI region is readable, extract the ROI region image sequence (i.e., the face image sequence).

[0102] (4):If the ROI region is not readable, do not perform any operation, but mark the image sequence as an unusable image.

[0103] In a specific implementation, the ROI region is a face image of a person, i.e., an image corresponding to a face region of a subject. First, a face detection algorithm needs to be used to detect a face. A preferred solution is an algorithm based on deep learning, such as a YOLO series network that has been trained, and an alternative solution is a traditional face detection algorithm, such as AdaBoost. After face detection, face alignment can be performed or not. Face alignment is beneficial to the robustness and accuracy of subsequent model processing. If the detected face is complete, the image is readable, otherwise it is not readable. Readable means that the region still retains face information. If it is readable, face information is directly extracted; if it is not readable, the image sequence is marked as an unusable image.

[0104] S303, perform speech signal separation according to the face image sequence and the mixed speech signal to determine the speech signal of each object in the vehicle control.

[0105] Specifically, in the embodiments of the present application, after the image sequence is extracted, a face image sequence is obtained, the face image sequence is preprocessed as an input, the mixed speech signal is preprocessed as another input, and a preset speech separation model is input for speech separation processing.

[0106] Specifically, in one of the embodiments of the present application, the manner of preprocessing the face image includes but is not limited to: inputting the face image sequence into a trained image feature extraction model to extract the feature embedding of the image as the input of the subsequent speech separation model. The image feature extraction model can be a neural network-based model to extract image features, such as FRCNN, FACENET, etc. For example, FACENET is used to extract embedding.

[0107] Further, in one of the embodiments of the present application, in response to the image quality not satisfying the preset quality condition, speech signal separation is performed according to the mixed speech signal to determine the speech signal of each object in the vehicle control.

[0108] Specifically, in the embodiments of the present application, if the face image quality does not meet the preset quality condition, indicating that the image sequence is not available, the mixed speech signal is preprocessed, and the preprocessed mixed speech information is further input as the only input into the speech separation model for speech separation processing.

[0109] Specifically, in some embodiments of the present application, the preprocessing of the mixed speech signal further includes: performing signal conversion on the mixed speech signal to obtain a spectrogram corresponding to the mixed speech signal, wherein the sampling frequency corresponding to the mixed speech signal in the spectrogram includes multiple frames of speech information.

[0110] Specifically, in the embodiments of the present application, the mixed speech signal is first filtered, and then the filtered mixed speech signal is further converted into a spectrogram.

[0111] Specifically, the present application realizes the speech separation embodiment one flowchart by acquiring information, and see FIG. 5, which is another embodiment flowchart of the speech separation method provided by one of the embodiments of the present application, specifically including: the embodiment is that the driver and the vehicle machine voice interact while other passengers are talking at the same time, and the data acquisition device is a camera and a microphone. The processing method specifically includes the following steps:

[0112] (1) Acquire visual and speech information using in-vehicle equipment, specifically including: acquiring face image sequences of the driver and the passengers based on the in-vehicle camera (in-vehicle equipment), and acquiring a mixed speech signal based on the in-vehicle microphone;

[0113] (2) Perform image quality preprocessing and speech preprocessing, specifically including: performing image quality judgment and preprocessing on the face image sequences of different persons, and preprocessing the mixed speech signal, so that the image sequence and the mixed speech signal meet the input conditions of the subsequent speech separation model.

[0114] (3) Process the visual and speech information using a separation model to obtain separated speech signals of the speakers, specifically including: inputting the preprocessed mixed speech signal and the preprocessed face image sequence into the model to obtain separated speech signals.

[0115] In this embodiment, the data acquired by the camera is video data, which needs to be divided into frames to extract and converted into a picture sequence. Specifically, based on a set frame rate, for example, 25 frames or 30 frames, one picture is extracted per frame, and all the pictures on the time axis constitute an image sequence.

[0116] In this embodiment, the microphone can be a single microphone or a microphone array composed of multiple microphones. If it is a single microphone, the speech signal is a single-channel speech signal, and if it is a microphone array, it is a multi-channel microphone speech signal.

[0117] In the case of a single microphone voice signal, the mixed voice can be pre-filtered and denoised. For example, a hybrid single-channel voice enhancement technology based on deep learning and traditional methods can be used as an alternative to traditional single-channel voice enhancement technologies such as Wiener filter, spectral subtraction, etc.

[0118] In the case of a multi-channel voice signal, the mixed voice can also be pre-enhanced. For example, a multi-channel voice enhancement method based on deep learning such as deep learning-based BF can be used as an alternative to traditional methods such as traditional BF methods.

[0119] Regardless of whether the microphone is configured as a single microphone or a microphone array, after pre-enhancement, the voice signal is a single-channel voice signal. If the subsequent voice separation model is a time-domain processing model, the voice does not need to be post-processed. If the subsequent voice separation model is a frequency-domain processing model, it needs to be converted into voice features as input for the subsequent voice separation model. For example, the embodiment of the present application needs to be converted into a spectrogram after STFT.

[0120] In the embodiment of the present application, if the voice separation model includes the above two inputs, the voice signal separation is performed according to the pre-processed face image sequence and the spectrogram to determine the voice signal of each object in the vehicle control. It can be understood that if there is only one input, the pre-processed voice mixed signal only needs to be input into the voice separation model for processing.

[0121] Specifically, on the basis of any one of the above embodiments, the present application also provides some embodiments for pre-processing the voice mixed signal, which specifically includes the following steps:

[0122] (1) Aligning the face image sequence and the spectrogram;

[0123] (2) Performing voice signal separation processing on the aligned face image sequence and the spectrogram to determine the voice signal of each object in the vehicle control.

[0124] Specifically, in the embodiment of the present application, the alignment is performed by combining the image acquisition frequency and the sampling rate of the spectrogram corresponding to the mixed voice signal.

[0125] Specifically, in response to the image acquisition frequency of the image sequence being less than the preset sampling frequency of the spectrogram, the face image sequence is interpolated to align the preset sampling frequency and the image acquisition frequency.

[0126] It can be understood that if the image acquisition frequency of the image sequence is greater than the preset sampling frequency of the spectrogram, the spectrogram can also be subjected to a nearest neighbor interpolation processing to align the preset sampling frequency and the image acquisition frequency.

[0127] Further, referring to FIG. 6, FIG. 6 is a flowchart of a speech separation method according to an embodiment of the present application, which specifically includes:

[0128] (1): Obtain speech input, which in this embodiment is a spectrogram after STFT.

[0129] (2): Speech input encoder network, used to extract speech embedding.

[0130] (3): Determine the judgment result of whether the image ROI region is readable.

[0131] (4): If the image is reliable, input the face image sequence into the image encoder network.

[0132] (5): Image encoder network coding processing, the visual encoder network learns and extracts deep features of vision based on the separation model target.

[0133] (6): Input the output of the image encoder network and the output of the speech encoder into the speech separation network, or input the output of the speech encoder into the speech separation network, which is responsible for multi-modal fusion and speech separation. When the image quality is reliable, the model input is two; when the image quality is not reliable, the model input is only one. In both cases, the model uses the same structure but different weights.

[0134] (7): Separated speech output by the model.

[0135] Specifically, since the model in this embodiment is a frequency domain processing model, the mixed speech information is converted speech features. Specifically, the mixed speech information is a spectrogram after STFT, which contains joint information of the time-frequency domain of the mixed speech and can well represent different speakers together with visual information. The visual information is the image embedding output by the image preprocessing model. The speech encoder network extracts deep features of speech information, which is composed of one or more CNN layers.

[0136] In one embodiment, the image encoder network is a U-Net network of a Backbone dilated CNN, which can reduce the number of neural network layers while ensuring the receptive field, thereby reducing the model size. Similarly, the speech encoder is also composed of one or more CNN layers. For example, the speech encoder (the image encoder network can be referred to as an image encoder) is a dilated CNN network with Skip Connection, which can achieve feature extraction while protecting the local information of the shallow layer.

[0137] For example, referring to FIG. 7, FIG. 7 is a schematic diagram of one structure of a speech encoder network provided in an embodiment of the present application.

[0138] In this scheme, the speech encoder network is a dilated CNN network without Skip Connection. Alternatively, the speech encoder network can be a Conformer network. The input of the speech separator network is the speech feature and the visual feature, and the input of the visual feature is dynamically controlled according to the image quality. The preferred fusion scheme of the modal information is MLP, and the alternative scheme is direct splicing. After fusion, the speech mask is obtained by processing the separator backbone network, which has a corresponding structure with the speech encoder network. The separated speech signal is obtained by combining the speech mask with the speech.

[0139] In this scheme:

[0140] 401, three dilated 2D CNNs with the same structure, the convolution kernel size is 3, and the dilation factor is 1. The input is the spectrogram S ∈ R T×F×2 of the mixed speech, T is the time frame, F is the frequency frame, 2 is the channel number, and the real and imaginary parts of the mask are represented. The outputs of the three layers are represented as Conv1, Conv2, and Conv3, which are spliced with the inputs of the corresponding layers in the speech separation model.

[0141] 402: dilated 2D CNN, the convolution kernel size is 3, and the dilation factor is 2. The output of this layer is represented as Conv4, which is spliced with the input of the corresponding layer in the speech separation model.

[0142] 403: dilated 2D CNN, the convolution kernel size is 3, and the dilation factor is 4. The output of this layer is represented as Conv5, which is spliced with the input of the corresponding layer in the speech separation model.

[0143] 404: dilated 2D CNN, the convolution kernel size is 3, and the dilation factor is 8. The output of this layer is represented as Conv6, which is spliced with the input of the corresponding layer in the speech separation model.

[0144] 405: dilated 2D CNN, the convolution kernel size is 3, and the dilation factor is 16. The output of this layer is represented as Conv7, which is spliced with the input of the corresponding layer in the speech separation model.

[0145] 406: dilated 2D CNN with kernel size 3 and dilation factor 32.

[0146] 407: output of the speech decoder. The output of 406 is dimensionally transformed at this time to facilitate subsequent concatenation with the output of the image encoder.

[0147] The encoder network is generally based on the U-Net structure, while using dilated CNNs instead of the ordinary CNNs and pooling layers in the U-Net to keep the resolution of the feature maps consistent across different levels, thus preserving the receptive field and resolution of features while capturing the contextual information of the speech. The input of the layer is the spectrogram S ∈ RT×F×2 of the mixed speech, where T is the time frame, F is the frequency frame, and 2 is the real and imaginary parts of the spectrogram. The function of each layer of dilated convolution is:

[0148] The output of the network is the speech embedding: S' = reshape(Conv8), Thus, the layer is expressed by the formula: S' = encoder A (S), where encoder A To simplify the description and facilitate understanding, the above description omits that each layer of the dilated CNN network uses Batch Normalization as post-norm and is subjected to a Relu activation function.

[0149] Specifically, referring to FIG. 8, FIG. 8 is a schematic diagram of one of the structures of the image encoder network provided in the embodiments of the present application.

[0150] In the present scheme, a multi-layer dilated CNN is used to constitute the image encoder (the image encoder network can be referred to as the image encoder for short), which can also maintain the receptive field and protect the shallow information, but the use of the pooling operation when downsampling will cause the loss of small-scale features when downsampling.

[0151] In the present scheme:

[0152] 501: dilated 2D CNN with kernel size 3 and dilation factor 1.

[0153] 502: dilated 2D CNN with kernel size 3 and dilation factor 1.

[0154] 503: dilated 2D CNN with kernel size 5 and dilation factor 1.

[0155] 504: Dilated 2D CNN, kernel size 1, dilation factor 1. This layer changes the number of channels using convolution with kernel size 1, which is convenient for the channel alignment of the skip connection. Meanwhile, this layer increases the depth of the network, which can improve the nonlinear ability of the network.

[0156] 505: Dilated 2D CNN, kernel size 5, dilation factor 2. The output of this layer is concatenated with the input of 503 to form the skip connection.

[0157] 506: Dilated 2D CNN, kernel size 5, dilation factor 4. The output of this layer is concatenated with the input of 502 to form the skip connection.

[0158] 507: Dilated 2D CNN, kernel size 3, dilation factor 8. The output of this layer is concatenated with the input of 501 to form the skip connection.

[0159] 508: Dilated 2D CNN, kernel size 3, dilation factor 16.

[0160] 509: The output of the image encoder.

[0161] The network of the image encoder uses a dilated CNN network with a skip connection. The input is the image embedding processed by the pre-processing model: i represents the ith speaker, T V represents the number of frames of the image sequence, C V is the number of channels of the processed embedding. In this embodiment, the networks of different speakers share network weights. The function of each dilated convolution layer is

[0162] Specifically, in the implementation scheme of the present application, the speech separation method is applied to a speech separation model, and the speech separation model comprises:

[0163] a fusion processing layer, configured to perform feature fusion processing on input features to output fused features, the input features being image encoding features corresponding to the face image sequence and speech encoding features corresponding to the mixed speech signal, or the input features being the speech encoding features corresponding to the mixed speech signal;

[0164] a speech separation layer, configured to perform speech separation on the fused features to output speech signals of different dimensions.

[0165] Specifically, in the embodiments of the present application, the input features are the pre-processed mixed speech signal, or the pre-processed mixed speech signal and the pre-processed image sequence.

[0166] Specifically, in the embodiments of the present application, the pre-processing of the mixed speech signal includes but is not limited to filtering processing of the mixed speech signal, and / or signal conversion coding processing of the mixed speech signal, and the specific implementation is shown in any of the above.

[0167] Specifically, in the embodiments of the present application, the pre-processing of the image sequence includes but is not limited to image sequence quality judgment and face image sequence extraction.

[0168] Specifically, in the embodiments of the present application, the fusion processing layer includes:

[0169] The up-sampling alignment layer is configured to perform coding feature alignment processing on the speech coding features and the image coding features, and perform splicing on the speech coding features and the image coding features after the alignment processing to obtain first output features, or to perform processing on the speech coding features to obtain first output features.

[0170] The full connection layer is configured to perform feature fusion on the first output features to output fusion features.

[0171] Specifically, the speech coding features are a spectrogram obtained by signal conversion on the mixed speech signal, and the collection frequency of the corresponding speech in the spectrogram includes multi-frame speech information. The multi-frame speech information and the image coding features are time-aligned.

[0172] It can be understood that, in the embodiments of the present application, the image sequence and the mixed speech signal are collected at the same time.

[0173] Specifically, in one of the embodiments of the present application, the speech separation layer includes:

[0174] The up-sampling convolution layer is configured to perform up-sampling convolution processing on the fusion features to output a convolution result.

[0175] The output layer is configured to perform mask mapping processing on the convolution result to output corresponding mask information of each dimension, and convert each mask information into a separated speech signal.

[0176] Specifically, the up-sampling convolution layer includes an up-sampling layer and a convolution layer.

[0177] Specifically, in one of the embodiments of the present application, the up-sampling convolution layer includes a first sub-layer and a plurality of second sub-layers connected in sequence.

[0178] an output end of the first sub-layer is connected with a first input end of one of the second sub-layers;

[0179] a second output end of the second sub-layer is configured to input a speech feature encoded feature with a same feature size as the first input end, so as to splice the input of the first input end and the input of the second input end, and output a convolution result of the spliced input after up-sampling convolution processing;

[0180] an output end of one of the second sub-layers is connected with an input end of the output layer.

[0181] Specifically, in some embodiments of the present application, the output layer comprises:

[0182] an activation function layer configured to map the convolution result to a mask value range;

[0183] a mask layer configured to perform multi-dimensional mask processing on the convolution result in the mask value range, and output corresponding mask information of each dimension;

[0184] a conversion layer configured to perform point multiplication processing on the corresponding mask information of each dimension and the spectrogram, and output multi-dimensional separated speech signals, each of which is a speech signal of each object in the vehicle control.

[0185] In an embodiment of the present application, the speech separation model further comprises a speech encoding layer connected with the fusion processing layer in the speech separation model.

[0186] The speech encoding layer is configured to encode a mixed speech signal to output corresponding speech encoded features.

[0187] Specifically, the speech encoding layer comprises a plurality of dilated convolution layers configured to output speech encoded features with different sizes.

[0188] In an embodiment of the present application, the speech separation model further comprises an image encoding layer connected with the fusion processing layer.

[0189] The image encoding layer is configured to encode a face image sequence to output corresponding image encoded features.

[0190] It can be understood that the speech encoding layer and the image encoding processing layer refer to the image encoder and the speech encoder in the above-mentioned embodiments, and will not be described in detail.

[0191] Specifically, referring to FIG. 9, FIG. 9 is a structure diagram of a speech separation model in a speech separation method provided by the present application.

[0192] Specifically, in the embodiments of the present application:

[0193] 601: upsample alignment, that is, corresponding to the upsample alignment layer in the above embodiments, using nearest neighbor interpolation to interpolate the output of the image encoder in the time domain dimension, so that the frame rate is aligned with the sampling rate of the spectrogram corresponding to the mixed speech signal, avoiding the difference in sampling rate: V' i = upsamp(I8 i ), The speech encoder output and the visual encoder output are spliced as the output. Since the speaker in this embodiment is 2, the audio-visual information of the two speakers is spliced: D = [S', V1', V2'], and its dimension is transformed to

[0194] 602: fully connected layer, that is, corresponding to the fully connected layer in the above, the spliced audio-visual joint information is preliminarily fused, which is expressed by the formula: W = fc(D), W ∈ R T×F×N .

[0195] 603: upsample + 2D convolution, that is, corresponding to the first sublayer in the above, upsample increases the dimension of the input channel, which facilitates the convolution layer of this layer to learn. The convolution kernel size is 3, which is expressed by the formula: X0 = conv(upsamp(W)),

[0196] 604: 7 layers with similar structures, that is, corresponding to the second sublayer in the above, which is composed of upsample and 2D convolution. The upsample uses nearest neighbor interpolation to increase the dimension of the input channel, so that the input of this layer is aligned with the dimension of the speech embedding from the speech encoder. The convolution kernel size is 3. The input of each sublayer is spliced with Conv7 to Conv1 in the speech encoder to input the CNN:

[0197] X j+1 = conv([X j ,E 8-j ]),

[0198] 605: the activation function is a Tanh() function, that is, corresponding to the activation function layer in the above, which maps the output result back to the value range [-1, 1] of the mask.

[0199] 606: output mask, that is, corresponding to the mask layer in the above, the output is dimensionally transformed to generate masks of different speakers:

[0200] Y = resize(X8), Exemplarily, in this embodiment, the speaker is 2, so the last dimension of the mask is 2.

[0201] 607: Mask to speech, that is, corresponding to the conversion layer, different masks are multiplied with the spectrogram of the mixed speech to obtain the mask of the separated speech, and then ISTFT is performed to obtain the separated speech:

[0202] The layer fuses the cross-modal fusion and the speech decoder two sub-layers. The speech decoder is based on the U-Net architecture, and the input of each layer of the network is spliced with the corresponding encoder output. To simplify the description and facilitate understanding, the above description omits that each layer of CNN network will use Batch Normalization as post-norm and pass through a Relu activation function except the output layer. The loss function is the L2 loss of the mask: Where n represents the number of speakers, which is 2 in this embodiment.

[0203] The embodiment uses a speech encoder + image encoder + speech separation model, which can greatly improve the speech separation accuracy.

[0204] Where s i 、 x is the original speech, separated speech and total mixed speech of the i-th speaker.

[0205] Specifically, the application also provides a vehicle speech processing system, comprising an image acquisition device, a processing device and an audio acquisition device, the audio acquisition device and the image acquisition device are connected with the processing device respectively;

[0206] The image acquisition device is configured to acquire an image sequence of at least one object in a vehicle space.

[0207] The audio acquisition device is configured to acquire a mixed speech signal of at least one object in a vehicle space.

[0208] The processing device is configured to perform speech signal separation according to the image sequence and the mixed speech signal to determine the speech signal of each object in the vehicle control.

[0209] The scheme discards the traditional scheme of configuring multiple speech acquisition devices to perform speech acquisition to ensure the accuracy of speech separation, and performs multi-modal speech separation by acquiring corresponding image sequences combined with mixed speech signals, improves the accuracy of speech separation, reduces the dependence of speech separation on speech acquisition equipment, and thus can reduce the configuration of speech acquisition devices and reduce costs.

[0210] And when the face in the image sequence is blocked or damaged, the subsequent separation model automatically switches to a pure speech separation model, thereby ensuring model effectiveness and achieving speech separation. That is, the present application does not rely on prior information of the speaker, and the pure speech model and the audio-visual joint model use similar architectures, so the deployment is simple, and the model does not need to be trained online, greatly simplifying the deployment steps and reducing the deployment difficulty.

[0211] The above improvements enable speech separation with visual information when the microphone array configuration is low, reducing the number of microphones required and hardware costs. At the same time, a single model can also achieve speech separation effects when image quality is high and low, reducing the number of models and reducing model deployment costs, laying the foundation for the realization of technology downgrading to low-end vehicles.

[0212] Referring to FIG. 10, FIG. 10 is a structural schematic diagram of one of the embodiments of the vehicle speech processing system provided by the present application.

[0213] 701 is the front-end microphone in the vehicle, i.e., the speech acquisition device, responsible for receiving the speaker's voice.

[0214] 702 is the audio communication line in the vehicle, which realizes the communication between the speech signal received by the microphone and the electronic control components in the vehicle.

[0215] 703 is the camera in the vehicle, i.e., the image acquisition device, which can clearly capture the face region.

[0216] 704 is the video communication line in the vehicle, which realizes the communication between the image captured by the camera and the electronic control components in the vehicle.

[0217] 705 is the electronic control component in the vehicle, i.e., the processing device, responsible for processing the algorithm of the present application.

[0218] The following will be described in detail in steps based on the above exemplary system. The front-end microphone in the vehicle is preferably a microphone array, such as 6-microphone, 4-microphone, or dual-microphone, placed in the front row of the driver. The alternative is a single microphone. The audio communication line in the vehicle is preferably an A2B Bus bus, which can realize high-bandwidth, low-latency, and low-cost communication. The alternative is AVB technology, which can realize real-time transmission and device synchronization, etc. The preferred solution for the camera in the vehicle uses a vehicle-grade high-resolution camera, including but not limited to ISX031, etc., to realize high-resolution data acquisition. The alternative uses a camera with lower resolution, including but not limited to IMX390, etc. The preferred solution for the video communication line uses GMSL2, and the alternative chooses GMSL. The preferred solution for the electronic control component in the vehicle is an ECU integrated with a SOC, including but not limited to Qualcomm 8155, 8295 chips, etc. The alternative is an ECU integrated with an MCU.

[0219] Further, on the basis of the above-mentioned embodiments, the present application also provides a vehicle, which comprises the vehicle voice processing system as described in the above-mentioned embodiments, the processing device in the vehicle voice processing system comprises:

[0220] one or more processors;

[0221] a memory; and

[0222] one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the steps in the voice separation method as described in any one of the above-mentioned embodiments.

[0223] In combination with the above-mentioned embodiments, in some embodiments of the present application, the vehicle in the vehicle part comprises the vehicle in the above-mentioned embodiments, and the one or more processors and the memory in the vehicle part are all integrated on a circuit board body included in the vehicle, and the circuit board body is arranged in the vehicle.

[0224] It can be understood that, in some other embodiments of the present application, the processor and the memory in the vehicle can also not be integrated on the vehicle, i.e., the processor and the memory are arranged in the vehicle as components of the vehicle respectively.

[0225] As shown in FIG. 11, FIG. 11 is a structural schematic diagram of one embodiment of the vehicle provided in the embodiments of the present application.

[0226] Specifically, the vehicle can comprise a processor 1001 with one or more than one processing core, a memory 1002 with one or more than one computer readable storage medium, a power supply 1003, an input part 1004, and the like. It can be understood by those skilled in the art that the vehicle structure shown in FIG. 11 does not constitute a limitation on the vehicle, and can comprise more or fewer components than those shown in the figure, or combine certain components, or different component arrangements. Among them:

[0227] The processor 1001 is the image processing center, which connects various parts of the vehicle through various interfaces and lines, executes software programs and / or layers stored in the memory 1002 and calls data stored in the memory 1002, performs various functions and processes data of the vehicle, and thus monitors the vehicle as a whole. It can be understood that the processor 1001 can transmit signals with the controller, and optionally, the processor 1001 can comprise one or more processing cores; preferably, the processor 1001 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 1001.

[0228] The memory 1002 can be used to store software programs and layers, and the processor 1001 executes various functional applications and data processing by running the software programs and layers stored in the memory 1002. The memory 1002 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the vehicle, etc. In addition, the memory 1002 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 1002 can also include a memory controller to provide the processor 1001 with access to the memory 1002.

[0229] In some embodiments of the present application, the speech separation device can be implemented in the form of computer instructions, which can run on a vehicle as shown in FIG. 11. The memory of the vehicle can store various program layers constituting the speech separation device, and the computer instructions constituted by the various program layers enable the processor to perform the steps in the speech separation method of each embodiment of the present application described in the specification.

[0230] For example, the vehicle includes a processor, a memory, and a network interface connected through a system bus. The processor of the vehicle is used to provide computing and control capabilities. The memory of the vehicle includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer instructions. The internal memory provides an environment for the operating system and computer instructions in the non-volatile storage medium to run. The network interface of the vehicle is used to communicate with external vehicles through network connection. The computer instructions can implement a speech separation method when executed by the processor.

[0231] The vehicle also includes a power supply 1003 for powering various components. Preferably, the power supply 1003 can be logically connected to the processor 1001 through a power management system, so as to realize functions such as charge management, discharge management, and power consumption management through the power management system. The power supply 1003 can also include one or more than one direct current or alternating current power supply, a recharging system, a power supply fault detection circuit, a power supply converter or inverter, a power supply status indicator, and any other components.

[0232] The vehicle can also include an input unit 1004, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0233] Although not shown, the vehicle can also include a display device or the like, which will not be described here. In particular in the present embodiment, the processor 1001 in the vehicle will load the executable file corresponding to the process of one or more than one application into the memory 1002 according to the following instructions, and run the application stored in the memory 1002 by the processor 1001, thereby realizing various functions as follows:

[0234] acquire an image sequence of at least one object in a vehicle space and a mixed speech signal;

[0235] perform speech signal separation according to the image sequence and the mixed speech signal to determine a speech signal of each object in the vehicle control.

[0236] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions or controlled by related hardware through instructions, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0237] To this end, the embodiments of the present application provide a computer readable storage medium, which can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc. Computer instructions are stored thereon, which are loaded by a processor to execute the steps in any speech separation method provided by the embodiments of the present application. For example, the computer instructions loaded by the processor can execute the following steps:

[0238] acquire an image sequence of at least one object in a vehicle space and a mixed speech signal;

[0239] perform speech signal separation according to the image sequence and the mixed speech signal to determine a speech signal of each object in the vehicle control.

[0240] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the detailed description of other embodiments above, which will not be described here.

[0241] In particular implementation, each of the above layers or structures can be implemented as an independent entity, or can be combined as the same or several entities, and the specific implementation of each of the above layers or structures can be referred to the method embodiments above, which will not be described here.

[0242] The specific implementation of each of the above operations can be referred to the embodiments above, which will not be described here.

[0243] The above describes in detail the voice separation method, system, vehicle and storage medium provided by the embodiments of the application. The principles and implementation manners of the application are described by using specific examples. The above description of the embodiments is only used to help understand the method of the application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the application, the specific implementation manners and application ranges will be changed. In conclusion, the content of the specification should not be understood as a limitation of the application.

Claims

A method of speech separation, wherein, The method comprises: collecting an image sequence of at least one object in a vehicle space and a mixed voice signal; and performing voice signal separation according to the image sequence and the mixed voice signal to determine a voice signal of each object in a vehicle control. The method further comprises: The voice separation method according to claim 1, wherein, in response to an image quality corresponding to a face region of an object in the image sequence satisfying a preset quality condition, extracting a face image sequence corresponding to the face region of the object in the image sequence; the voice signal separation according to the image sequence and the mixed voice signal to determine a voice signal of each object in the vehicle control comprises: performing voice signal separation according to the face image sequence and the mixed voice signal to determine a voice signal of each object in the vehicle control. The method further comprises: The voice separation method according to claim 2, wherein, in response to the image quality not satisfying the preset quality condition, performing voice signal separation according to the mixed voice signal to determine a voice signal of each object in the vehicle control. The method further comprises: The voice separation method according to claim 2, wherein, in response to identifying a face region of an object in the image sequence, determining that an image quality corresponding to the face region of the object satisfies a preset quality condition; and in response to not identifying a face region of an object in the image sequence, determining that an image quality corresponding to the face region of the object does not satisfy a preset quality condition, and marking the image sequence as unusable. The method further comprises: The voice separation method according to claim 2, wherein, performing signal conversion on the mixed voice signal to obtain a spectrogram corresponding to the mixed voice signal, wherein a sampling frequency corresponding to the mixed voice signal in the spectrogram comprises multiple frames of voice information; the voice signal separation according to the face image sequence and the mixed voice signal to determine a voice signal of each object in the vehicle control comprises: performing voice signal separation according to the face image sequence and the spectrogram to determine a voice signal of each object in the vehicle control. The voice signal separation according to the face image sequence and the mixed voice signal to determine a voice signal of each object in the vehicle control comprises: The voice separation method according to claim 5, wherein, aligning the face image sequence and the spectrogram; and performing voice signal separation processing on the aligned face image sequence and the spectrogram to determine a voice signal of each object in the vehicle control. The aligning the face image sequence and the spectrogram comprises: The voice separation method according to claim 6, wherein in response to an image acquisition frequency of the image sequence being less than a preset sampling frequency of the spectrogram, performing interpolation processing on the face image sequence to align the preset sampling frequency and the image acquisition frequency. The method is applied to a voice separation model, and the voice separation model comprises: The voice separation method according to claim 3, wherein, a fusion processing layer configured to perform feature fusion processing on input features to output fused features, wherein the input features are image encoding features corresponding to a face image sequence and voice encoding features corresponding to a mixed voice signal, or the input features are voice encoding features corresponding to the mixed voice signal; a voice separation layer configured to perform voice separation on the fused features to output separated voice signals of different dimensions. The fusion processing layer comprises: The voice separation method according to claim 8, wherein, ​ An up-sampling alignment layer is configured to perform feature alignment on the speech coding features and the image coding features, and concatenate the speech coding features and the image coding features after the alignment to obtain first output features, or perform processing on the speech coding features to obtain first output features. A full connection layer is configured to perform feature fusion on the first output features, and output fused features. The voice separation method according to claim 8, wherein, The speech separation layer includes: An up-sampling convolution layer is configured to perform up-sampling convolution processing on the fused features, and output convolution results. An output layer is configured to perform mask mapping processing on the convolution results, output corresponding mask information of each dimension, and convert the mask information into separated speech signals. The voice separation method according to claim 10, wherein, The up-sampling convolution layer includes a first sub-layer and a plurality of sequentially connected second sub-layers. An output end of the first sub-layer is connected to a first input end of one of the second sub-layers. A second output end of the second sub-layer is configured to input speech coding features with the same feature size as the first input end, so as to concatenate the input of the first input end and the input of the second input end, and output after up-sampling convolution processing on the concatenated input. An output end of one of the second sub-layers is connected to an input end of the output layer. The voice separation method according to claim 10, wherein, The output layer includes: An activation function layer is configured to map the convolution results to a mask value range. A mask layer is configured to perform multi-dimensional mask processing on the convolution results in the mask value range, and output corresponding mask information of each dimension. A conversion layer is configured to perform point multiplication processing on the corresponding mask information of each dimension and the spectrogram, and output multi-dimensional separated speech signals, each of which is a speech signal of each object in the vehicle control. The voice separation method according to claim 7, wherein The speech separation model further includes a speech coding layer connected to the fusion processing layer. The speech coding layer is configured to perform coding processing on the mixed speech signals, and output corresponding speech coding features. The voice separation method according to claim 13, wherein, The speech coding layer includes a plurality of dilated convolution layers configured to output speech coding features with different sizes. The voice separation method according to any one of claims 7-14, wherein, The speech separation model further includes an image coding layer connected to the fusion processing layer. The image coding layer is configured to encode the face image sequence, and output corresponding image coding features. A vehicle speech processing system, wherein The system includes an image acquisition device, a processing device, and an audio acquisition device, wherein the audio acquisition device and the image acquisition device are connected to the processing device. The image acquisition device is configured to acquire an image sequence of at least one object in a vehicle space. The audio acquisition device is configured to acquire a mixed speech signal of at least one object in the vehicle space. The processing device is configured to perform speech signal separation based on the image sequence and the mixed speech signal to determine a speech signal of each object in the vehicle control. A vehicle, wherein, The vehicle includes the vehicle speech processing system of claim 16, and the processing device in the vehicle speech processing system includes: one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the voice separation method of any one of claims 1 to 15. A computer readable storage medium for use in the speech separation method of claim 1, wherein, a computer program product, having stored thereon computer instructions, the computer instructions being loadable into a processor to cause execution of the steps of the voice separation method of any one of claims 1 to 15.

Images