Voice separation method and system, vehicle and storage medium

Abstract

The invention provides a voice separation method and system, a vehicle and a storage medium. The method comprises the steps of collecting an image sequence and a mixed voice signal of at least one object in a vehicle space; and performing voice signal separation according to the image sequence and the mixed voice signal to determine a voice signal of each object in the vehicle control. According to the scheme, a traditional scheme of configuring a plurality of voice acquisition devices for voice acquisition to ensure accurate voice separation is abandoned, and multi-mode voice separation is performed by acquiring the corresponding image sequence and combining the mixed voice signals, so that the voice separation accuracy is improved, the dependence of voice separation on voice acquisition equipment is reduced, and the voice separation efficiency is improved. Configuration of the voice acquisition device can be reduced, and the cost is reduced.

Description

Technical Field

[0001] The present application relates to the field of voice interaction technology, and in particular to a voice separation method, system, vehicle and storage medium. Background Art

[0002] With the development of smart cars, the application of in-vehicle voice interaction technology has become increasingly widespread. However, among related technologies, in-vehicle voice interaction technology often uses a distributed microphone array to perform effective voice separation. This solution relies on the number and position of microphones, resulting in excessively high hardware costs. Summary of the invention

[0003] The present application provides a speech separation method, system, vehicle and storage medium, aiming to solve the problem in the above technical solution that a large number of microphones need to be installed for effective speech separation, thereby increasing hardware costs.

[0004] In a first aspect, the present application provides a speech separation method, comprising:

[0005] collecting an image sequence and a mixed speech signal of at least one object in a vehicle space;

[0006] Speech signal separation is performed based on the image sequence and the mixed speech signal to determine the speech signal of each object in the vehicle control.

[0007] Optionally, the method further includes:

[0008] If the image quality corresponding to the facial region of the object in the image sequence meets a preset quality condition, extracting a facial image sequence corresponding to the facial region of the object in the image sequence;

[0009] The performing speech signal separation according to the image sequence and the mixed speech signal to determine the speech signal of each object in the vehicle control comprises:

[0010] Speech signal separation is performed based on the facial image sequence and the mixed speech signal to determine the speech signal of each object in the vehicle control.

[0011] Optionally, the method further includes:

[0012] If the quality of the facial image does not meet a preset quality condition, voice signal separation is performed based on the mixed voice signal to determine the voice signal of each object in the vehicle control.

[0013] Optionally, the method further includes:

[0014] If a facial region of the subject in the image sequence is identified, determining that the image quality corresponding to the facial region of the subject meets a preset quality condition;

[0015] If the facial region of the subject in the image sequence is not recognized, it is determined that the image quality corresponding to the facial region of the subject does not meet a preset quality condition, and the image sequence is marked as unusable.

[0016] Optionally, the method further includes:

[0017] Performing signal conversion on the mixed speech signal to obtain a spectrogram corresponding to the mixed speech signal, wherein the sampling frequency corresponding to the mixed speech signal in the spectrogram includes multiple frames of speech information;

[0018] The performing voice signal separation according to the facial image sequence and the mixed voice signal to determine the voice signal of each object in the vehicle control comprises:

[0019] Speech signal separation is performed based on the facial image sequence and the spectrogram to determine the speech signal of each object in the vehicle control.

[0020] Optionally, performing voice signal separation according to the facial image sequence and the mixed voice signal to determine the voice signal of each object in the vehicle control comprises:

[0021] Performing alignment processing on the facial image sequence and the spectrogram;

[0022] A speech signal separation process is performed based on the aligned facial image sequence and the spectrogram to determine the speech signal of each object in the vehicle control.

[0023] Optionally, the aligning the facial image sequence and the spectrogram includes:

[0024] If the image acquisition frequency of the image sequence is less than the preset sampling frequency of the spectrogram, interpolation processing is performed on the facial image sequence to align the preset sampling frequency with the image acquisition frequency.

[0025] Optionally, the method is applied to a speech separation model, and the speech separation model includes:

[0026] A fusion processing module, used for performing feature fusion processing on input features and outputting fusion features, wherein the input features are image coding features corresponding to the facial image sequence and speech coding features corresponding to the mixed speech signal, or the input features are speech coding features corresponding to the mixed speech signal;

[0027] The speech separation module is used to perform speech separation on the fusion features and output separated speech signals of different dimensions.

[0028] Optionally, the fusion processing module includes:

[0029] An upsampling alignment unit, used to perform coding feature alignment processing on the speech coding feature and the image coding feature, and to concatenate the speech coding feature and the image coding feature after the alignment processing to obtain a first output feature, or to process the speech coding feature to obtain a first output feature;

[0030] The fully connected unit is used to perform feature fusion on the first output features and output fused features.

[0031] Optionally, the speech separation module includes:

[0032] An upsampling convolution unit, used to perform upsampling convolution processing on the fused features and output a convolution result;

[0033] The output unit is used to perform mask mapping processing on the convolution result, output corresponding mask information of each dimension, and convert each mask information into a separated speech signal.

[0034] Optionally, the upsampling convolution unit includes a first subunit and a plurality of second subunits connected in sequence;

[0035] The output terminal of the first subunit is connected to the first input terminal of one of the second subunits;

[0036] The second output end of the second subunit is used to input speech feature speech coding features of the same feature size as the first input end, so as to realize splicing processing of the input of the first input end and the input of the second input end, and output after upsampling convolution processing on the spliced ​​input;

[0037] An output terminal of one of the second subunits is connected to an input terminal of the output unit.

[0038] Optionally, the output unit includes:

[0039] An activation function layer, used to map the convolution result into a mask value range;

[0040] The mask layer is used to perform multi-dimensional mask processing on the convolution results within the mask value range, and output the corresponding mask information of each dimension;

[0041] The conversion layer is used to perform point multiplication processing according to the corresponding mask information of each dimension and the spectrogram, and output a multi-dimensional separated speech signal, each dimensional separated speech signal is a speech signal of each object in the vehicle control.

[0042] Optionally, the speech separation model further includes a speech encoding module, and the speech encoding module is connected to the fusion processing module;

[0043] The speech coding module is used to perform coding processing on the mixed speech signal and output corresponding speech coding features.

[0044] Optionally, the speech coding module includes multiple dilated convolutional layers for outputting speech coding features of different sizes.

[0045] Optionally, the speech separation model further includes an image encoding module, and the image encoding module is connected to the fusion processing module;

[0046] The image encoding module is used to encode the facial image sequence and output the corresponding image encoding features.

[0047] In a second aspect, the present application provides a vehicle voice processing system, comprising an image acquisition device, a processing device and an audio acquisition device, wherein the audio acquisition device and the image acquisition device are respectively connected to the processing device;

[0048] The image acquisition device is used to acquire a sequence of images of at least one object in the vehicle space;

[0049] The audio collection device is used to collect mixed speech signals of at least one object in the vehicle space;

[0050] The processing device is used to perform voice signal separation based on the image sequence and the mixed voice signal to determine the voice signal of each object in the vehicle control.

[0051] In a third aspect, the present application provides a vehicle, the vehicle comprising the vehicle voice processing system, the processing device in the vehicle voice processing system comprising:

[0052] one or more processors;

[0053] Memory; and

[0054] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement any one of the speech separation methods.

[0055] In a fourth aspect, the present application provides a computer-readable storage medium, characterized in that a computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any one of the speech separation methods described.

[0056] The present application provides a speech separation method, system, vehicle and storage medium, which collects an image sequence and a mixed speech signal of at least one object in the vehicle space; performs speech signal separation based on the image sequence and the mixed speech signal to determine the speech signal of each object in the vehicle control. This solution abandons the traditional solution of configuring multiple speech acquisition devices for speech acquisition to ensure accurate speech separation, and performs multimodal speech separation by collecting corresponding image sequences combined with mixed speech signals, thereby improving the accuracy of speech separation and reducing the dependence of speech separation on speech acquisition equipment, thereby reducing the configuration of speech acquisition devices and reducing costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0058] Figure 1 is a scene diagram of the speech separation method provided in an embodiment of the present application;

[0059] Figure 2 This is a flow chart of an embodiment of the speech separation method provided in the embodiments of the present application;

[0060] Figure 3 A flowchart of another speech separation method according to an embodiment of the present application;

[0061] Figure 4 Some of the embodiments of the present application provide for determining whether the image of the face area of ​​the subject meets the preset quality conditions;

[0062] Figure 5 A flowchart of another embodiment of the speech separation method provided in one of the embodiments of the present application;

[0063] Figure 6 A flow chart of a speech separation method is provided for one embodiment of the present application;

[0064] Figure 7 A schematic diagram of one structure of a speech encoder network provided in an embodiment of the present application;

[0065] Figure 8 A schematic diagram of one structure of an image encoder network provided for an implementation scheme of the present application;

[0066] Fig. 9 A schematic diagram of a speech separation model structure in the speech separation method provided in this application;

[0067] Fig.10 A schematic diagram of the structure of one implementation scheme of the vehicle speech processing system provided in this application;

[0068] Fig.11 It is a schematic diagram of the structure of an embodiment of a vehicle provided in the embodiments of the present application. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0070] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0071] In the embodiment of the present application, "and / or" describes the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / ", unless otherwise specified, generally indicates that the associated objects before and after are in an "or" relationship.

[0072] In this application, the word "exemplary" is used to mean "serving as an example, illustration, or illustration." Any embodiment described in this application as "exemplary" is not necessarily to be construed as being preferred or advantageous over other embodiments. The following description is given to enable any person skilled in the art to implement and use the invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the invention can be implemented without using these specific details. In other instances, well-known structures and processes are not elaborated in detail to avoid obscuring the description of the invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in this application.

[0073] The embodiments of the present application provide a speech separation method, system, vehicle and computer-readable storage medium (the computer-readable storage medium may be referred to as storage medium in its entirety), which are described in detail below.

[0074] The speech separation method in an embodiment of the present invention is applied to a vehicle speech processing system, which is arranged in a vehicle. The vehicle is provided with one or more processors, a memory, and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the speech separation method.

[0075] like Figure 1 As shown, Figure 1 This is a scenario diagram of the speech separation method in an embodiment of the present application. The speech separation scenario in the embodiment of the present invention includes a vehicle 100 (a vehicle speech processing system is integrated in the vehicle 100), and a computer-readable storage medium corresponding to the speech separation method is run in the vehicle 100 to execute the steps of the speech separation method.

[0076] Understandably, Figure 1 The vehicle in the scenario of the speech separation method shown, or the devices contained in the vehicle, does not constitute a limitation on the embodiments of the present invention. That is, the number of devices and types of devices contained in the scenario of the speech separation method, or the number of devices and types of devices contained in each device, do not affect the overall implementation of the technical solution in the embodiments of the present invention, and can all be regarded as equivalent replacements or derivatives of the technical solution claimed to be protected in the embodiments of the present invention.

[0077] In the embodiment of the present invention, the vehicle 100 is mainly used for: collecting an image sequence and a mixed voice signal of at least one object in the vehicle space; and performing voice signal separation according to the image sequence and the mixed voice signal to determine the voice signal of each object in the vehicle control.

[0078] The vehicle 100 in the embodiment of the present invention may be an independent vehicle, or a vehicle network or a vehicle cluster composed of vehicles. For example, the vehicle 100 described in the embodiment of the present invention includes but is not limited to a computer, a network host, a single network vehicle, a plurality of network vehicles, or a cloud vehicle composed of a plurality of vehicles. The cloud vehicle is composed of a large number of computers or network vehicles based on cloud computing.

[0079] Those skilled in the art will understand that Figure 1 The application environment shown in the figure is only one application scenario of the present application solution and does not constitute a limitation on the application scenario of the present application solution. Other application environments may also include Figure 1 More or fewer vehicles, or vehicle network connections, such as Figure 1 Only one vehicle is shown. It can be understood that the scenario of the speech separation method can also include one or more other vehicles, which are not limited here. The vehicle 100 can also include a memory for storing data, for example, storing image information obtained by shooting.

[0080] In addition, in the scenario of the speech separation method of the present application, the vehicle 100 may be provided with a display device, or the vehicle 100 may not be provided with a display device and may be communicated with an external display device, and the display device is used to output the result of the speech separation method in the vehicle. The vehicle 100 may access a backend database 200 (the backend database may be in the local memory of the vehicle, or may be provided in the cloud), and the backend database 200 may store information related to speech separation.

[0081] It should be noted that Figure 1 The scenario diagram of the speech separation method shown is only an example. The scenario of the speech separation method described in the embodiment of the present invention is to more clearly illustrate the technical solution of the embodiment of the present invention, and does not constitute a limitation on the technical solution provided by the embodiment of the present invention.

[0082] Based on the scenario of the above-mentioned speech separation method, an embodiment of the speech separation method is proposed.

[0083] like Figure 2 FIG. 2 is a flow chart of an embodiment of a speech separation method in an embodiment of the present application, wherein the speech separation method comprises steps S201-S202:

[0084] S201 . Collect an image sequence and a mixed speech signal of at least one object in a vehicle space.

[0085] Specifically, the image sequence includes a plurality of continuously acquired images, and the image sequence is acquired by an image acquisition device according to a preset image acquisition frequency.

[0086] Specifically, the mixed voice signal includes voice signals of multiple objects in the vehicle space, and the mixed voice signal can be collected by at least one microphone.

[0087] S202: Perform voice signal separation according to the image sequence and the mixed voice signal to determine the voice signal of each object in the vehicle control.

[0088] Specifically, in some embodiments of the present application, the image sequence and the mixed speech signal can be separated into speech signals by a preset speech separation model. It is understandable that in the embodiments of the present application, the speech separation model can be obtained by training.

[0089] Specifically, in one of the implementation schemes of the present application, it also includes performing face recognition on the image sequence, judging the quality of the image sequence based on the face recognition, and further determining whether to use the image sequence to participate in speech signal separation based on the quality.

[0090] For details, see Figure 3 , Figure 3 A flowchart of another speech separation method provided in the present application embodiment specifically includes steps S301-S303:

[0091] S301 , collecting an image sequence and a mixed speech signal of at least one object in a vehicle space.

[0092] S302: If the image quality corresponding to the facial region of the object in the image sequence meets a preset quality condition, extract a facial image sequence corresponding to the facial region of the object in the image sequence.

[0093] Specifically, in an implementation scheme of the present application, after acquiring an image sequence, the facial region of the object in the image sequence is identified. If the facial region of the object in the image sequence is identified, it is determined that the image quality corresponding to the facial region of the object meets the preset quality conditions; if the facial region of the object in the image sequence is not identified, it is determined that the image quality corresponding to the facial region of the object does not meet the preset quality conditions, and the image sequence is calibrated to be unusable.

[0094] In some other embodiments of the present application, the preset quality conditions may also include facial integrity, facial clarity, etc. of the facial area of ​​the object.

[0095] For example, see Figure 4 , Figure 4 Some of the embodiments of the present application for determining whether the image of the face area of ​​the object meets the preset quality conditions specifically include the steps of:

[0096] (1): Using an image sequence acquisition device (in-vehicle device), a camera in this embodiment, to obtain regional image sequences of the driver and passengers respectively.

[0097] (2): Determine whether the ROI area of ​​the image is readable, wherein the criterion for determining whether it is readable is whether the face area can be detected.

[0098] (3): If the ROI area is readable, extract the ROI area image sequence (ie, facial image sequence).

[0099] (4): If the ROI area is unreadable, no operation is performed, but the image sequence is marked as an unusable image.

[0100] In a specific implementation, the ROI area is a facial image of a person, that is, an image corresponding to the facial area of ​​the object. First, it is necessary to use a face detection algorithm to detect the face. The preferred solution is an algorithm based on deep learning, such as a trained YOLO series network, and the alternative is a traditional face detection algorithm, such as AdaBoost. Face alignment can be used after face detection, or it may not be used. Face alignment is beneficial to the robustness and accuracy of subsequent model processing. If the detected face is complete, the image is readable, otherwise it is unreadable. Readable means that the area still retains facial information. If readable, the facial information is directly extracted; if unreadable, the image sequence is marked as an unusable image.

[0101] S303: Perform voice signal separation according to the facial image sequence and the mixed voice signal to determine the voice signal of each object in the vehicle control.

[0102] Specifically, in the implementation scheme of the present application, a facial image sequence is obtained after the image sequence is extracted, the facial image sequence is preprocessed and used as an input, and the mixed speech signal is preprocessed and used as another input, and input into a preset speech separation model for speech separation processing.

[0103] Specifically, in one embodiment of the present application, the facial image is preprocessed in a manner including but not limited to: inputting the facial image sequence into a trained image feature extraction model, extracting image feature embeddings, and using them as inputs for a subsequent speech separation model. The image feature extraction model may be a neural network-based model for extracting image features, such as FRCNN, FACENET, etc. For example, embeddings are extracted using FACENET.

[0104] Furthermore, in one embodiment of the present application, if the quality of the facial image does not meet a preset quality condition, a speech signal separation is performed based on the mixed speech signal to determine the speech signal of each object in the vehicle control.

[0105] Specifically, in the implementation scheme of the present application, if the quality of the facial image does not meet the preset quality conditions, it means that the image sequence is unavailable, then the mixed speech signal is preprocessed, and the preprocessed mixed speech information is further used as the only input and input into the speech separation model for speech separation processing.

[0106] Specifically, in some implementation schemes of the present application, preprocessing the mixed speech signal also includes: performing signal conversion on the mixed speech signal to obtain a spectrogram corresponding to the mixed speech signal, wherein the sampling frequency corresponding to the mixed speech signal in the spectrogram includes multiple frames of speech information.

[0107] Specifically, in the implementation scheme of the present application, the mixed speech signal is first filtered, and then the filtered mixed speech signal is further converted into a spectrogram.

[0108] Specifically, the present application obtains information and implements a flow chart of the first embodiment of speech separation, see Figure 5 , Figure 5 A flow chart of another implementation scheme of the speech separation method provided for one of the implementation schemes of the present application specifically includes: In the embodiment, when the driver interacts with the vehicle computer by voice, other passengers speak at the same time, and the data collection equipment is a camera and a microphone. The processing method specifically includes the steps of:

[0109] (1): Using in-vehicle equipment to obtain visual and voice information, specifically including: obtaining facial image sequences of the driver and passenger based on the in-vehicle camera (in-vehicle equipment), and obtaining mixed voice signals based on the in-vehicle microphone;

[0110] (2): Perform image quality preprocessing and speech preprocessing, specifically including: performing image quality judgment and preprocessing on facial image sequences of different characters, and preprocessing the mixed speech signal, so that the image sequence and the mixed speech signal meet the input conditions of the subsequent speech separation model.

[0111] (3): Use the separation model to process the visual and speech information to obtain the separated speaker's speech signal, specifically including: inputting the preprocessed mixed speech signal and the preprocessed facial image sequence into the model to obtain the separated speech signal.

[0112] In this embodiment, the data acquired by the camera is video data, which needs to be frame-by-frame extracted and converted into a picture sequence. Specifically, based on the set frame rate, such as 25 frames or 30 frames, one picture is extracted from each frame, and all pictures on the timeline constitute an image sequence.

[0113] In this embodiment, the microphone may be a single microphone or a microphone array composed of multiple microphones. If it is a single microphone, the speech signal is a single-channel speech signal, and if it is a microphone array, it is a multi-channel microphone speech signal.

[0114] Under a single-microphone speech signal, the mixed speech can be pre-filtered and denoised. For example, a hybrid single-channel speech enhancement technology that combines deep learning with traditional methods is used. The alternative is the single-channel speech enhancement technology of traditional methods, such as Wiener filter, spectral subtraction, etc.

[0115] In the case of multi-channel speech signals, the mixed speech can also be pre-enhanced. For example, a multi-channel speech enhancement method based on deep learning, such as BF based on deep learning, etc., is an alternative to traditional methods, such as the traditional BF method.

[0116] Regardless of whether the microphone is configured as a single microphone or a microphone array, after pre-enhancement, the speech signal is a single-channel speech signal. If the subsequent speech separation model is a time domain processing model, the speech can be input without post-processing. If the subsequent speech separation model is a frequency domain processing model, it needs to be converted into speech features as the input of the subsequent speech separation model. For example, the implementation scheme of the present application needs to be transformed into a spectrogram after STFT.

[0117] In the embodiment of the present application, if the speech separation model includes the above two inputs, speech signal separation is performed based on the preprocessed facial image sequence and the spectrogram to determine the speech signal of each object in the vehicle control. It is understandable that if there is only one input, only the preprocessed speech mixed signal needs to be input into the speech separation model for processing.

[0118] Specifically, based on any of the above implementation schemes, the present application also provides some implementation schemes for preprocessing the speech mixed signal, specifically including the steps of:

[0119] (1) aligning the facial image sequence and the spectrogram;

[0120] (2) performing speech signal separation processing based on the aligned facial image sequence and the spectrogram to determine the speech signal of each object in the vehicle control.

[0121] Specifically, in the implementation scheme of the present application, alignment processing is performed by combining the image acquisition frequency and the sampling rate of the spectrogram corresponding to the mixed speech signal.

[0122] Specifically, if the image acquisition frequency of the image sequence is less than the preset sampling frequency of the spectrogram, interpolation processing is performed on the facial image sequence to align the preset sampling frequency with the image acquisition frequency.

[0123] It is understandable that if the image acquisition frequency of the image sequence is greater than the preset sampling frequency of the spectrogram, the spectrogram may also be subjected to nearest neighbor interpolation processing to align the preset sampling frequency with the image acquisition frequency.

[0124] For further information, see Figure 6 , Figure 6 A flow chart of a speech separation method is provided for one embodiment of the present application, which specifically includes:

[0125] (1): Acquire speech input. In this embodiment, the speech input is a spectrogram after STFT.

[0126] (2): Speech input encoder network, used to extract speech embedding.

[0127] (3): Determine whether the image ROI area is readable.

[0128] (4): If the image is reliable, the face image sequence is input into the image encoder network.

[0129] (5): Image encoder network encoding processing, the visual encoder network learns and extracts deep visual features based on the separation model objective.

[0130] (6): The output of the image encoder network and the output of the speech encoder are input into the speech separation network, or the output of the speech encoder is input into the speech separation network, which is responsible for multimodal fusion and speech separation. When the image quality is reliable, the model input is two; when the image quality is unreliable, the model input is only one input. In both cases, the model uses the same structure but different weights.

[0131] (7): Separated speech output by the model.

[0132] Specifically, since the model in this embodiment is a frequency domain processing model, the mixed speech information is the converted speech feature. Specifically, the mixed speech information is the spectrogram after STFT, which contains the joint information of the time and frequency domain of the mixed speech, and can be well integrated with the visual representation of different speaking objects. The visual information is the image embedding output by the image preprocessing model. The speech encoder network extracts the deep features of the speech information and is composed of one or more CNN layers.

[0133] In one embodiment, the image encoder network is a U-Net network of Backbone expanded CNN, which can reduce the number of neural network layers while ensuring the receptive field, thereby reducing the model size. Similarly, the speech encoder is also composed of one or more CNN layers. For example, the speech encoder (the image encoder network can be referred to as the image encoder) is an expanded CNN network with Skip Connection, which can protect shallow local information while achieving feature extraction.

[0134] For example, see Figure 7 , Figure 7 A schematic diagram of one structure of a speech encoder network provided in an implementation scheme of the present application.

[0135] In this solution, the speech encoder network is an expanded CNN network without Skip Connection. An alternative solution is also a Conformer network. The input of the speech separator network is speech features and visual features, and the input of visual features is dynamically controlled according to the image quality. The fusion solution of modal information is preferably MLP, and the alternative solution is direct splicing. After fusion, the speech mask is obtained by processing by the separator backbone network, and the network has a structure corresponding to the speech encoder network. The speech mask is combined with the speech to obtain the separated speech signal.

[0136] In this plan:

[0137] 401, three dilated 2D CNNs with the same structure, the convolution kernel size is 3, and the dilation factor is 1. The input is the spectrogram S∈R of the mixed speech T×F×2 , where T is the time frame, F is the frequency frame, and 2 is the number of channels, representing the real and imaginary parts of the mask. The outputs of the three modules are represented as Conv1, Conv2, and Conv3, which are concatenated with the inputs of the corresponding layers in the speech separation model.

[0138] 402: dilated 2D CNN, with kernel size of 3 and dilation factor of 2. The output of this module is represented as Conv4 concatenated with the input of the corresponding layer in the speech separation model.

[0139] 403: dilated 2D CNN, with kernel size of 3 and dilation factor of 4. The output of this module is represented as Conv5 and is concatenated with the input of the corresponding layer in the speech separation model.

[0140] 404: dilated 2D CNN, with kernel size of 3 and dilation factor of 8. The output of this module is represented as Conv6 concatenated with the input of the corresponding layer in the speech separation model.

[0141] 405: dilate 2D CNN, with a convolution kernel size of 3 and a dilation factor of 16. The output of this module is represented as Conv7 concatenated with the input of the corresponding layer in the speech separation model.

[0142] 406: Dilated 2D CNN with kernel size 3 and dilation factor 32.

[0143] 407: Output of the speech decoder. The output of 406 is dimensionally transformed at this point to facilitate subsequent splicing with the output of the image encoder.

[0144] The encoder network is based on the U-Net structure, and the expanded CNN is used to replace the original ordinary CNN and pooling layer in the U-Net, so that the resolution of the feature map is consistent at different levels, thereby capturing the speech context information while protecting the receptive field and feature resolution. The input of the module is the spectrogram S∈R of the mixed speech T×F×2 , where T is the time frame, F is the frequency frame, and 2 is the real and imaginary part of the spectrogram. The function of each layer of dilated convolution is:

[0145]

[0146] The output of the network is speech embedding: S' = reshape(Conv8), Therefore, the module is expressed by the formula: S' = encoder A (S), where encoder A () is the speech encoder. To simplify the description and facilitate understanding, the above description omits that each layer of the expanded CNN network uses Batch Normalization as the post-norm and passes through the Relu activation function.

[0147] For details, see Figure 8 , Figure 8 A schematic diagram of one of the structures of the image encoder network provided in the implementation scheme of the present application.

[0148] In this scheme, a multi-layer expanded CNN is used to form an image encoder (the image encoder network can be referred to as the image encoder for short), which can also maintain the receptive field and protect shallow information, but the use of pooling operation during downsampling will lead to the loss of small-scale features during downsampling.

[0149] In this plan:

[0150] 501: Dilated 2D CNN with kernel size of 3 and dilation factor of 1.

[0151] 502: Dilated 2D CNN with kernel size of 3 and dilation factor of 1.

[0152] 503: Dilated 2D CNN with kernel size of 5 and dilation factor of 1.

[0153] 504: dilate 2D CNN, with a convolution kernel size of 1 and a dilation factor of 1. This layer uses convolution with a convolution kernel of 1 to change the number of channels, which facilitates the alignment of the channels of the subsequent skip connection; at the same time, this layer increases the network depth and can improve the nonlinear ability of the network.

[0154] 505: dilated 2D CNN, with a convolution kernel size of 5 and a dilation factor of 2. The output of this module is concatenated with the input of 503 to form a Skip Connection.

[0155] 506: dilated 2D CNN, with a convolution kernel size of 5 and a dilation factor of 4. The output of this module is concatenated with the input of 502 to form a Skip Connection.

[0156] 507: dilated 2D CNN, with a convolution kernel size of 3 and a dilation factor of 8. The output of this module is concatenated with the input of 501 to form a Skip Connection.

[0157] 508: Dilated 2D CNN with kernel size 3 and dilation factor 16.

[0158] 509: Output of the image encoder.

[0159] The network of the image encoder uses an expanded CNN network with Skip Connection. The input is the image embedding processed by the preprocessing model: i represents the i-th speaker, T V Represents the number of frames in the image sequence, C V is the number of channels of the processed embedding. In this embodiment, the networks of different speakers share the network weights. The function of each layer of dilated convolution is

[0160]

[0161] Specifically, in the embodiment of the present application, the speech separation method is applied to a speech separation model, and the speech separation model includes:

[0162] A fusion processing module, used for performing feature fusion processing on input features and outputting fusion features, wherein the input features are image coding features corresponding to the facial image sequence and speech coding features corresponding to the mixed speech signal, or the input features are speech coding features corresponding to the mixed speech signal;

[0163] The speech separation module is used to perform speech separation on the fusion features and output separated speech signals of different dimensions.

[0164] Specifically, in the implementation scheme of the present application, the input feature is the preprocessed mixed speech signal, or the preprocessed mixed speech signal and the preprocessed image sequence.

[0165] Specifically, in the implementation scheme of the present application, preprocessing the mixed voice signal includes but is not limited to: filtering the mixed voice signal, and / or performing signal conversion coding processing on the mixed voice signal. The specific implementation is shown in any of the above items.

[0166] Specifically, in the embodiment of the present application, preprocessing the image sequence includes but is not limited to: image sequence quality judgment, and facial image sequence extraction.

[0167] Specifically, in the implementation scheme of the present application, the fusion processing module includes:

[0168] An upsampling alignment unit, used to perform coding feature alignment processing on the speech coding feature and the image coding feature, and to concatenate the speech coding feature and the image coding feature after the alignment processing to obtain a first output feature, or to process the speech coding feature to obtain a first output feature;

[0169] The fully connected unit is used to perform feature fusion on the first output features and output fused features.

[0170] Specifically, the speech coding feature is a spectrogram obtained by performing signal conversion on the mixed speech signal, the acquisition frequency of the corresponding speech in the spectrogram includes multiple frames of speech information, and the multiple frames of speech information and the image coding feature are time-aligned.

[0171] It can be understood that, in the embodiment of the present application, the image sequence and the mixed speech signal are collected simultaneously.

[0172] Specifically, in one embodiment of the present application, the speech separation module includes:

[0173] An upsampling convolution unit, used to perform upsampling convolution processing on the fused features and output a convolution result;

[0174] The output unit is used to perform mask mapping processing on the convolution result, output corresponding mask information of each dimension, and convert each mask information into a separated speech signal.

[0175] Specifically, the upsampling convolution unit includes an upsampling layer and a convolution layer.

[0176] Specifically, in one embodiment of the present application, the upsampling convolution unit includes a first subunit and a plurality of second subunits connected in sequence;

[0177] The output terminal of the first subunit is connected to the first input terminal of one of the second subunits;

[0178] The second output end of the second subunit is used to input speech feature speech coding features of the same feature size as the first input end, so as to realize splicing processing of the input of the first input end and the input of the second input end, and output after upsampling convolution processing on the spliced ​​input;

[0179] An output terminal of one of the second subunits is connected to an input terminal of the output unit.

[0180] Specifically, in some embodiments of the present application, the output unit includes:

[0181] An activation function layer, used to map the convolution result into a mask value range;

[0182] The mask layer is used to perform multi-dimensional mask processing on the convolution results within the mask value range, and output the corresponding mask information of each dimension;

[0183] The conversion layer is used to perform point multiplication processing according to the corresponding mask information of each dimension and the spectrogram, and output a multi-dimensional separated speech signal, each dimensional separated speech signal is a speech signal of each object in the vehicle control.

[0184] In one embodiment of the present application, the speech separation model further includes a speech encoding module, and the speech encoding module is connected to the fusion processing module in the speech separation model;

[0185] The speech coding module is used to perform coding processing on the mixed speech signal and output corresponding speech coding features.

[0186] Specifically, the speech coding module includes a plurality of dilated convolutional layers for outputting speech coding features of different sizes.

[0187] In one embodiment of the present application, the speech separation model further includes an image encoding module, and the image encoding module is connected to the fusion processing module;

[0188] The image encoding module is used to encode the facial image sequence and output the corresponding image encoding features.

[0189] It can be understood that the speech encoding module and the image encoding processing module refer to the image encoder and speech encoder in the above-mentioned implementation scheme, and the details are not repeated here.

[0190] For details, see Fig. 9 , Fig. 9 This is a schematic diagram of the structure of one of the speech separation models in the speech separation method provided in this application.

[0191] Specifically, in the implementation scheme of this application:

[0192] 601: Upsampling alignment, that is, corresponding to the upsampling alignment unit in the above implementation scheme, the output of the image encoder is interpolated in the time domain dimension using the nearest neighbor interpolation, so that its frame rate is aligned with the sampling rate of the spectrogram corresponding to the mixed speech signal to avoid sampling rate differences: V' i =upsamp(I8 i ), The output of the speech encoder and the output of the visual encoder are concatenated as the output. Since there are two speakers in this embodiment, the audio and visual information of the two speakers is concatenated: D = [S', V1', V2'], and its dimension is transformed into

[0193] 602: Fully connected layer, that is, corresponding to the above fully connected unit, the concatenated audio-visual joint information is initially fused, expressed by the formula: W = fc(D), W∈R T×F×N .

[0194] 603: Upsampling + 2D convolution, that is, corresponding to the first subunit, upsampling increases the dimension of the input channel to facilitate the learning of the convolution layer of this layer. The convolution kernel size is 3, which is expressed by the formula X0 = conv(upsamp(W)).

[0195] 604: 7 modules with similar structures, i.e., corresponding to the second sub-unit mentioned above, consisting of upsampling and 2D convolution. Upsampling uses nearest neighbor interpolation to increase the dimension of the input channel so that the input of this layer is aligned with the speech embedding dimension from the speech encoder. The convolution kernel size is 3. The input of each sub-module is concatenated with Conv7 to Conv1 in the speech encoder and input into CNN: X j+1 =conv([X j ,E 8-j ]), j∈[1,...,7].

[0196] 605: The activation function is a Tanh() function, that is, corresponding to the above activation function layer, the output result is mapped back to the value range [-1, 1] of the mask.

[0197] 606: Output mask, that is, corresponding to the above mask layer, the output is dimensionally transformed to generate masks of different speakers: Y = resize (X8), Exemplarily, in this embodiment, the speaker is 2, so the last dimension of the mask is 2.

[0198] 607: Mask to speech conversion, that is, corresponding to the above conversion layer, dot multiplication is performed on different masks and the spectrogram of the mixed speech to obtain the mask of the separated speech, and then ISTFT is performed to obtain the separated speech:

[0199] This module integrates two submodules: cross-modal fusion and speech decoder. The speech decoder is based on the U-Net architecture, and the input of each layer of the network is concatenated with the corresponding encoder output. To simplify the description and facilitate understanding, the above description omits that each layer of the CNN network except the output layer will use Batch Normalization as post-norm and pass through the Relu activation function. The loss function is the masked L2 loss: Wherein n represents the number of speakers, which is 2 in this embodiment.

[0200] This embodiment uses a speech encoder + an image encoder + a speech separation model, which can greatly improve the accuracy of speech separation.

[0201] where s i , x are the original speech, separated speech and total mixed speech of the ith speaker respectively.

[0202] Specifically, the present application also provides a vehicle voice processing system, comprising an image acquisition device, a processing device and an audio acquisition device, wherein the audio acquisition device and the image acquisition device are respectively connected to the processing device;

[0203] The image acquisition device is used to acquire a sequence of images of at least one object in the vehicle space;

[0204] The audio collection device is used to collect mixed speech signals of at least one object in the vehicle space;

[0205] The processing device is used to perform voice signal separation based on the image sequence and the mixed voice signal to determine the voice signal of each object in the vehicle control.

[0206] This solution abandons the traditional solution of configuring multiple voice acquisition devices for voice acquisition to ensure accurate voice separation. It performs multimodal voice separation by acquiring corresponding image sequences combined with mixed voice signals, thereby improving the accuracy of voice separation and reducing the dependence of voice separation on voice acquisition equipment, thereby reducing the configuration of voice acquisition devices and reducing costs.

[0207] When the face in the image sequence is blocked or destroyed, the subsequent separation model automatically switches to a pure voice separation model, thereby ensuring the model effect and achieving voice separation. That is, the present invention does not rely on the prior information of the interlocutor, and the pure voice model and the audio-visual joint model use a similar architecture, so the deployment is simple, and the model does not require online training, which greatly simplifies the deployment steps and reduces the deployment difficulty.

[0208] The above improvements make it possible to separate speech by combining voice with visual information when the microphone array configuration is low, reducing the number of microphones required and hardware costs. At the same time, a single model can also achieve speech separation effects when the image quality is high or low, reducing the number of models and the cost of model deployment, laying the foundation for the decentralization of technology to low-end models.

[0209] See Fig.10 , Fig.10 This is a schematic diagram of the structure of one implementation scheme of the vehicle voice processing system provided in this application.

[0210] 701 is the front microphone in the car, that is, the voice collection device, which is responsible for receiving the speaker's voice.

[0211] 702 is an in-vehicle audio communication line, which realizes the communication between the voice signal received by the microphone and the in-vehicle electronic control unit.

[0212] 703 is an in-car camera, that is, an image acquisition device, which can clearly capture the face area.

[0213] 704 is an in-vehicle video communication line, which realizes the communication between the image captured by the camera and the electronic control unit in the vehicle.

[0214] 705 is an in-vehicle electronic control unit, that is, a processing device, responsible for processing the algorithm of the present invention.

[0215] The following will be elaborated in detail step by step based on an exemplary system mentioned above. The front microphone in the car is preferably a microphone array, such as 6 microphones, 4 microphones, or dual microphones, placed in the front row of the driver. The alternative is a single microphone. The in-car audio communication line is preferably an A2B Bus, which can achieve high-bandwidth, low-latency, and low-cost communication. The alternative is AVB technology, which can transmit in real time and achieve device synchronization, etc. The preferred solution for the in-car camera is to use a car-grade high-resolution camera, including but not limited to ISX031, etc., to achieve high-resolution data acquisition. The alternative solution uses a camera with a lower resolution, including but not limited to IMX390, etc. The preferred solution for the video communication line uses GMSL2, and the alternative solution chooses GMSL. The preferred solution for the in-car electronic control unit is an ECU with integrated SOC, and the SOC includes but is not limited to Qualcomm 8155, 8295 chips, etc. The alternative solution is an ECU with integrated MCU.

[0216] Further, based on the above embodiments, the present application also provides a vehicle, the vehicle comprising the vehicle voice processing system as described in the above embodiments, the processing device in the vehicle voice processing system comprising:

[0217] one or more processors;

[0218] Memory; and

[0219] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the steps in the speech separation method described in any of the above embodiments.

[0220] In combination with the above-mentioned embodiments, it can be known that in some of the embodiments of the present application, the vehicle unit may include the vehicle of the above-mentioned embodiments, and one or more processors and memories in the vehicle unit are integrated on a circuit board body included in the vehicle, and the circuit board body is arranged in the vehicle.

[0221] It is understandable that in some other embodiments of the present application, the processor and memory in the vehicle may not be integrated in the vehicle, that is, the processor and memory are respectively arranged in the vehicle as components of the vehicle.

[0222] like Fig.11 As shown, Fig.11 It is a schematic diagram of the structure of an embodiment of a vehicle provided in the embodiments of the present application.

[0223] Specifically, the vehicle may include one or more processing core processors 1001, one or more computer-readable storage media memories 1002, a power supply 1003, an input unit 1004, and other components. Those skilled in the art will appreciate that Fig.11 The vehicle structure shown in the figure does not constitute a limitation on the vehicle, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0224] in:

[0225] The processor 1001 is the image processing center, which uses various interfaces and lines to connect various parts of the entire vehicle, and executes various functions of the vehicle and processes data by running or executing software programs and / or modules stored in the memory 1002, and calling data stored in the memory 1002, so as to monitor the vehicle as a whole. It can be understood that the processor 1001 transmits signals with the controller. Optionally, the processor 1001 may include one or more processing cores; preferably, the processor 1001 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 1001.

[0226] The memory 1002 can be used to store software programs and modules. The processor 1001 executes various functional applications and data processing by running the software programs and modules stored in the memory 1002. The memory 1002 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the vehicle, etc. In addition, the memory 1002 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 1002 may also include a memory controller to provide the processor 1001 with access to the memory 1002.

[0227] In some embodiments of the present application, the speech separation device can be implemented in the form of a computer program. The computer program can be used in Fig.11 The vehicle memory may store various program modules constituting the speech separation device, and the computer program composed of the various program modules enables the processor to execute the steps of the speech separation method of each embodiment of the present application described in this specification.

[0228] For example, the vehicle includes a processor, a memory, and a network interface connected via a system bus. The processor of the vehicle is used to provide computing and control capabilities. The memory of the vehicle includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the vehicle is used to communicate with an external vehicle via a network connection. When the computer program is executed by the processor, a speech separation method can be implemented.

[0229] The vehicle further includes a power supply 1003 for supplying power to various components. Preferably, the power supply 1003 can be logically connected to the processor 1001 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 1003 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0230] The vehicle may further include an input unit 1004, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0231] Although not shown, the vehicle may also include a display unit, etc., which will not be described in detail here. Specifically in this embodiment, the processor 1001 in the vehicle will load the executable files corresponding to the processes of one or more applications into the memory 1002 according to the following instructions, and the processor 1001 will run the applications stored in the memory 1002, thereby realizing various functions, as follows:

[0232] collecting an image sequence and a mixed speech signal of at least one object in a vehicle space;

[0233] Speech signal separation is performed based on the image sequence and the mixed speech signal to determine the speech signal of each object in the vehicle control.

[0234] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0235] To this end, an embodiment of the present invention provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any of the speech separation methods provided in the embodiments of the present invention. For example, the computer program can be loaded by a processor to execute the following steps:

[0236] collecting an image sequence and a mixed speech signal of at least one object in a vehicle space;

[0237] Speech signal separation is performed based on the image sequence and the mixed speech signal to determine the speech signal of each object in the vehicle control.

[0238] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the detailed description of other embodiments above, and will not be repeated here.

[0239] In specific implementation, the above units or structures can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units or structures can refer to the previous method embodiments, which will not be repeated here.

[0240] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0241] The above is a detailed introduction to a speech separation method, system, vehicle and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A speech separation method, characterized in that: include: collecting an image sequence and a mixed speech signal of at least one object in a vehicle space; Speech signal separation is performed based on the image sequence and the mixed speech signal to determine the speech signal of each object in the vehicle control.

2. The speech separation method according to claim 1, characterized in that: The method further comprises: If the image quality corresponding to the facial region of the object in the image sequence meets a preset quality condition, extracting a facial image sequence corresponding to the facial region of the object in the image sequence; The performing speech signal separation according to the image sequence and the mixed speech signal to determine the speech signal of each object in the vehicle control comprises: Speech signal separation is performed based on the facial image sequence and the mixed speech signal to determine the speech signal of each object in the vehicle control.

3. The speech separation method according to claim 2, characterized in that: The method further comprises: If the quality of the facial image does not meet a preset quality condition, voice signal separation is performed based on the mixed voice signal to determine the voice signal of each object in the vehicle control.

4. The speech separation method according to claim 2, characterized in that: The method further comprises: If a facial region of the subject in the image sequence is identified, determining that the image quality corresponding to the facial region of the subject meets a preset quality condition; If the facial region of the subject in the image sequence is not recognized, it is determined that the image quality corresponding to the facial region of the subject does not meet a preset quality condition, and the image sequence is marked as unusable.

5. The speech separation method according to claim 2, characterized in that: The method further comprises: Performing signal conversion on the mixed speech signal to obtain a spectrogram corresponding to the mixed speech signal, wherein the sampling frequency corresponding to the mixed speech signal in the spectrogram includes multiple frames of speech information; The performing voice signal separation according to the facial image sequence and the mixed voice signal to determine the voice signal of each object in the vehicle control comprises: Speech signal separation is performed based on the facial image sequence and the spectrogram to determine the speech signal of each object in the vehicle control.

6. The speech separation method according to claim 5, characterized in that: The performing voice signal separation according to the facial image sequence and the mixed voice signal to determine the voice signal of each object in the vehicle control comprises: Performing alignment processing on the facial image sequence and the spectrogram; A speech signal separation process is performed based on the aligned facial image sequence and the spectrogram to determine the speech signal of each object in the vehicle control.

7. The speech separation method according to claim 6, characterized in that: The aligning process of the facial image sequence and the spectrogram comprises: If the image acquisition frequency of the image sequence is less than the preset sampling frequency of the spectrogram, interpolation processing is performed on the facial image sequence to align the preset sampling frequency with the image acquisition frequency.

8. The speech separation method according to claim 3, characterized in that: The method is applied to a speech separation model, which includes: A fusion processing module, used for performing feature fusion processing on input features and outputting fusion features, wherein the input features are image coding features corresponding to the facial image sequence and speech coding features corresponding to the mixed speech signal, or the input features are speech coding features corresponding to the mixed speech signal; The speech separation module is used to perform speech separation on the fusion features and output separated speech signals of different dimensions.

9. The speech separation method according to claim 7, characterized in that: The fusion processing module includes: an upsampling alignment unit, configured to perform coding feature alignment processing on the speech coding feature and the image coding feature, and to concatenate the speech coding feature and the image coding feature after the alignment processing to obtain a first output feature, or to process the speech coding feature to obtain a first output feature; The fully connected unit is used to perform feature fusion on the first output features and output fused features.

10. The speech separation method according to claim 8, characterized in that: The speech separation module comprises: An upsampling convolution unit, used to perform upsampling convolution processing on the fused features and output a convolution result; The output unit is used to perform mask mapping processing on the convolution result, output corresponding mask information of each dimension, and convert each mask information into a separated speech signal.

11. The speech separation method according to claim 10, characterized in that: The upsampling convolution unit includes a first subunit and a plurality of second subunits connected in sequence; The output terminal of the first subunit is connected to the first input terminal of one of the second subunits; The second output end of the second subunit is used to input speech feature speech coding features of the same feature size as the first input end, so as to realize splicing processing of the input of the first input end and the input of the second input end, and output after upsampling convolution processing on the spliced ​​input; An output terminal of one of the second subunits is connected to an input terminal of the output unit.

12. The speech separation method according to claim 10, characterized in that: The output unit comprises: An activation function layer, used to map the convolution result into a mask value range; The mask layer is used to perform multi-dimensional mask processing on the convolution results within the mask value range, and output the corresponding mask information of each dimension; The conversion layer is used to perform point multiplication processing according to the corresponding mask information of each dimension and the spectrogram, and output a multi-dimensional separated speech signal, each dimensional separated speech signal is a speech signal of each object in the vehicle control.

13. The speech separation method according to claim 7, characterized in that: The speech separation model also includes a speech encoding module, and the speech encoding module is connected to the fusion processing module; The speech coding module is used to perform coding processing on the mixed speech signal and output corresponding speech coding features.

14. The speech separation method according to claim 13, characterized in that: The speech coding module includes a plurality of dilated convolutional layers for outputting speech coding features of different sizes.

15. The speech separation method according to claim 7, characterized in that: The speech separation model also includes an image encoding module, and the image encoding module is connected to the fusion processing module; The image encoding module is used to encode the facial image sequence and output the corresponding image encoding features.

16. A vehicle voice processing system, characterized in that: It includes an image acquisition device, a processing device and an audio acquisition device, wherein the audio acquisition device and the image acquisition device are respectively connected to the processing device; The image acquisition device is used to acquire a sequence of images of at least one object in the vehicle space; The audio collection device is used to collect mixed speech signals of at least one object in the vehicle space; The processing device is used to perform voice signal separation based on the image sequence and the mixed voice signal to determine the voice signal of each object in the vehicle control.

17. A vehicle, characterized in that: The vehicle includes the vehicle voice processing system according to claim 16, wherein the processing device in the vehicle voice processing system includes: one or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the speech separation method according to any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the speech separation method according to any one of claims 1 to 15.

Images