Voice separation method, device, system, equipment, storage medium and program product

Abstract

The invention provides a voice separation method, device, system and equipment, a storage medium and a program product, and the method comprises the steps: obtaining a voice signal on a target vehicle, and enabling the voice signal to be collected through a microphone array loaded on the target vehicle; extracting voice features and directivity features of the voice signals, wherein the voice features and the directivity features are irrelevant to the formation of the microphone array; and voice separation is performed based on the voice features and the directivity features to obtain a voice separation result corresponding to the voice signals, so that complete isolation of sounds on the vehicle can be realized without depending on formation information of a microphone array, guarantee is provided for independent interaction of each sound area, and the user experience is improved. The problem that in a traditional scheme, voice separation on a vehicle depends on a separation model obtained through formation information training, and consequently the dilemma that one vehicle is customized is caused is solved, voice separation is carried out based on formation-irrelevant features, the large-scale delivery efficiency of the system is greatly improved, cost is reduced, and resource waste is reduced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a speech separation method, device, system, equipment, storage medium and program product. Background Art

[0002] With the development of automobile intelligence, in-vehicle intelligent voice interaction systems have gradually become an important part of modern automobiles, providing drivers and passengers with a more convenient and intelligent interactive experience. The system controls various functions in the car by recognizing the wake-up words and commands of passengers. However, in actual applications, due to differences in vehicle size and interior layout, the formations of microphone arrays of different models of cars are diverse, which brings challenges to the large-scale delivery of in-vehicle voice interaction systems.

[0003] Traditional in-vehicle interactive systems mostly perform voice separation based on the array information of the microphone array, thereby achieving sound isolation and avoiding interference between different sound zones. However, this method requires a customized separation model for each car model, and the "one car, one customization" approach is not only inefficient but also costly. Whenever a new car model is launched, or even if there is a slight change in the microphone array layout of the same car model, the separation model needs to be customized again, which consumes a lot of time and resources and reduces the efficiency of large-scale delivery. Summary of the invention

[0004] The present invention provides a speech separation method, device, system, equipment, storage medium and program product, which are used to solve the problem in the prior art that the vehicle-mounted interactive system is prone to fall into the dilemma of "one car, one customization" when realizing sound isolation so that each sound zone is independent and does not interfere with each other.

[0005] The present invention provides a speech separation method, comprising: Acquiring a voice signal on a target vehicle, wherein the voice signal is collected by a microphone array mounted on the target vehicle; Extracting speech features and directional features of the speech signal, wherein the speech features and the directional features are independent of the formation of the microphone array; Speech separation is performed based on the speech feature and the directional feature to obtain a speech separation result corresponding to the speech signal.

[0006] According to a speech separation method provided by the present invention, the speech separation is performed based on the speech feature and the directional feature to obtain a speech separation result corresponding to the speech signal, including: Based on the speech feature and the directional feature, applying a speech separation model to perform speech separation to obtain a speech separation result corresponding to the speech signal; The speech separation model is trained based on microphone signals collected by microphones at different intervals in sample scenarios and corresponding sample speech separation results.

[0007] According to a speech separation method provided by the present invention, the speech separation model is trained based on the following steps: Acquire multi-channel microphone signals in the sample scene; Encode the microphone signals of each channel, and fuse the features of the encoded microphone signals to obtain sample microphone features; Performing single-channel speech separation on each microphone signal to obtain a time-frequency mask of each pure microphone signal, and determining a sample directional feature based on the time-frequency mask of each pure microphone signal and the frequency domain phase of each microphone signal; Inputting the sample microphone feature and the sample directional feature into an initial speech separation model to obtain a predicted speech separation result output by the initial speech separation model; Based on the predicted speech separation result and the sample speech separation result, the initial speech separation model is iterated on parameters to obtain a speech separation model.

[0008] According to a speech separation method provided by the present invention, the sample directional characteristics are determined based on the time-frequency mask of each channel of pure microphone signal and the frequency domain phase of each channel of microphone signal, including: Determine the spatial covariance matrix of each channel of clean microphone signal based on the time-frequency mask of each channel of clean microphone signal; Based on the spatial covariance matrix of each channel of pure microphone signal and the frequency domain phase of each channel of microphone signal, the sample directional characteristics are determined.

[0009] According to a speech separation method provided by the present invention, the method of determining the sample directional characteristics based on the spatial covariance matrix of each channel of the pure microphone signal and the frequency domain phase of each channel of the microphone signal includes: Performing eigenvalue decomposition on the spatial covariance matrix of each channel of the clean microphone signal to obtain a steering vector of each channel of the clean microphone signal; Based on the steering vectors of the pure microphone signals and the frequency domain phases of the microphone signals, the sample directional characteristics are determined.

[0010] According to a speech separation method provided by the present invention, the step of determining the sample directional characteristics based on the steering vectors of the pure microphone signals and the frequency domain phases of the microphone signals comprises: Determining a time delay difference between two microphones in each microphone pair based on the steering vectors of the pure microphone signals; Determining a phase difference between two microphones in each microphone pair based on the frequency domain phase of each microphone signal; Based on the time delay difference and phase difference of each microphone pair, the cosine distance of each microphone pair in the direction of the microphone signal source is determined, and based on the cosine distance of each microphone pair in the direction of the microphone signal source, the sample directivity feature is determined.

[0011] According to a speech separation method provided by the present invention, the determining of the spatial covariance matrix of each channel of clean microphone signal based on the time-frequency mask of each channel of clean microphone signal comprises: Determine a median time-frequency mask based on the time-frequency masks of the pure microphone signals; Based on the median time-frequency mask and the microphone signals of each channel, a spatial covariance matrix of the pure microphone signals of each channel is determined.

[0012] According to a speech separation method provided by the present invention, encoding each microphone signal and fusing the features of each microphone signal obtained by encoding to obtain a sample microphone feature includes: Encode each microphone signal, and perform nonlinear activation on the features of each microphone signal obtained by encoding to obtain each microphone feature; The microphone features are averaged and pooled to obtain global microphone features, and the global microphone features are fused to obtain sample microphone features.

[0013] According to a speech separation method provided by the present invention, the global microphone features are fused to obtain sample microphone features, including: The global microphone feature of each channel is fused with the corresponding microphone feature to obtain the target microphone feature of each channel; The target microphone features of each channel are fused to obtain the sample microphone features.

[0014] The present invention also provides a speech separation device, comprising: A signal acquisition unit, used to acquire a voice signal on a target vehicle, wherein the voice signal is acquired by a microphone array mounted on the target vehicle; A feature extraction unit, used for extracting speech features and directional features of the speech signal, wherein the speech features and the directional features are independent of the formation of the microphone array; The speech separation unit is used to perform speech separation based on the speech feature and the directional feature to obtain a speech separation result corresponding to the speech signal.

[0015] The present invention also provides a vehicle-mounted interactive system, comprising a microphone array, a processor and an audio player; The processor is used to obtain a voice signal collected by a microphone array on a target vehicle, extract voice features and directional features of the voice signal, perform voice separation based on the voice features and the directional features, obtain a voice separation result corresponding to the voice signal, perform a signal response based on the voice separation result, generate reply content, and control the audio player to play the reply content; The speech feature and the directional feature are independent of the formation of the microphone array.

[0016] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the speech separation method described in any one of the above is implemented.

[0017] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the speech separation method as described above is implemented.

[0018] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the speech separation method as described above is implemented.

[0019] The speech separation method, apparatus, system, equipment, storage medium and program product provided by the present invention extract the speech features and directional features of the speech signal collected by the microphone array on the target vehicle that are independent of the formation, and perform speech separation based on the features. This can achieve complete isolation of the sound on the vehicle without relying on the formation information of the microphone array, provide guarantee for independent interaction of each sound zone, and avoid the problem that the speech separation on the vehicle in the traditional solution relies on the separation model obtained by training with the formation information, resulting in the dilemma of "one vehicle, one customization". Speech separation is performed based on formation-independent features, which greatly improves the efficiency of large-scale delivery of the system, reduces costs, and reduces waste of resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 It is a flow chart of the speech separation method provided by the present invention; Figure 2 It is an overall framework diagram of the speech separation model training process provided by the present invention; Figure 3 is an overall block diagram of the sample directional feature determination process provided by the present invention; Figure 4 is a general framework diagram of the sample microphone feature determination process provided by the present invention; Figure 5 It is a structural schematic diagram of the speech separation device provided by the present invention; Figure 6 is a structural schematic diagram of the vehicle-mounted interactive system provided by the present invention; Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] In the field of smart cars, a highlight of the in-vehicle intelligent voice interaction system (in-vehicle interaction system) is the realization of independent sound zones, so that each seat can obtain a unique interactive experience, that is, people in different positions (passengers and drivers) can independently perform voice interactions without interference from other sound zones. In order to achieve complete sound isolation, the separation model is currently trained using spatial information of known formations. However, this approach will fall into the dilemma of "one car, one customization". That is, the separation model needs to be customized for each model, and every time a new model is launched, or even if the same model only makes minor changes in the microphone array layout, the separation model needs to be customized again, which poses a great challenge to the large-scale delivery of the system.

[0024] In this regard, the present invention provides a speech separation method, which aims to extract two types of features that are independent of the formation from the speech signal, and perform speech separation based on the features to obtain the signals of each sound zone, thereby achieving complete isolation of the sound and providing guarantee for the independent interaction of each sound zone. This avoids the problem that the current speech separation on vehicles relies on the separation model obtained by training with formation information, which may fall into the dilemma of "one vehicle, one customization", and greatly improves the system delivery efficiency.

[0025] Figure 1 It is a flow chart of the speech separation method provided by the present invention, such as Figure 1 As shown, the method can be applied to the in-vehicle interactive system, and the specific execution subject can be a processor in the in-vehicle interactive system. The method includes: Step 110, acquiring a voice signal on the target vehicle, where the voice signal is collected by a microphone array mounted on the target vehicle; Step 120, extracting speech features and directional features of the speech signal, wherein the speech features and directional features are independent of the formation of the microphone array; Step 130, performing speech separation based on the speech features and the directional features to obtain a speech separation result corresponding to the speech signal.

[0026] Considering that the current in-vehicle interactive systems mostly use the spatial information of the microphone array formation on the vehicle to customize a separation model for the vehicle in order to achieve independent interaction in each sound zone, that is, passengers and drivers in different positions can conduct independent voice interaction without interference from other areas. However, this method is not only time-consuming and labor-intensive, but the customization of the new model requires a lot of resource investment, which is costly and will cause unnecessary waste.

[0027] Based on this, an embodiment of the present invention proposes a formation-independent separation method. By extracting formation-independent features from the speech signal for speech separation, the speech separation process can be separated from the formation of the microphone array on the vehicle, so that the speech separation on the vehicle no longer depends on the formation information of the microphone array, thereby avoiding the problem of "one car, one customization" and being unaffected by the differences in the microphone array formation. It can achieve accurate separation and positioning of different sound sources, meet the needs of the in-vehicle interactive system in practical applications, and provide a guarantee for the fast, efficient and low-cost delivery of in-vehicle interactive systems of different models.

[0028] It is understandable that in practical applications, before voice separation, the first thing that needs to be determined is the voice signal to be separated, that is, the mixed signal picked up by the vehicle microphone. Here, specifically, when the target vehicle is started and the vehicle interaction system is working, the microphone array installed on the target vehicle can pick up the voice of the occupant in the vehicle, and a mixed signal is obtained through multi-channel sound pickup, which is the voice signal to be separated on the target vehicle.

[0029] After obtaining the voice signal to be separated, in order to achieve the independence of the sound zones, so that each sound zone does not interfere with each other and can interact independently, the embodiment of the present invention also needs to perform voice separation on the voice signal to obtain the voice separation result, that is, the voice signal of each sound zone. Considering that the current voice separation on the vehicle relies on the separation model obtained by training the spatial information of the known array formation, it will lead to the problem of "one car, one customization" and reduce the delivery efficiency. In the embodiment of the present invention, when performing voice separation on the voice signal of the target vehicle, it is selected to extract features unrelated to the formation, and voice separation is performed based on this, which completely solves the vehicle customization problem.

[0030] Specifically, the speech signal may be processed first to obtain speech features, and specifically, effective features may be extracted from the speech signal through operations such as encoding, activation, and pooling, and the effective features extracted from the multi-channel signals may be fused to obtain speech features. Next, considering that the speech features can only guarantee some information of the signal itself, the speech separation based on this is slightly thin and cannot well guarantee the separation effect. Therefore, in the embodiment of the present invention, it is necessary to further extract directional features that can characterize the direction of the signal, so as to combine the speech features and directional features to perform speech separation, realize accurate sound source positioning, and accurately separate the speech signals of each sound zone. Here, specifically, the speech signal may be processed through a signal processing algorithm to extract information related to the signal direction, especially information related to the direction of the signal source, to obtain the directional features of the speech signal.

[0031] It should be noted that, in order to avoid the misunderstanding of "one car, one customization", the voice features and directional features extracted in the embodiment of the present invention are all features that are independent of the formation of the microphone array on the target vehicle. "Independent of formation" does not mean that the entire feature extraction process does not involve information related to the formation, but that the information related to the formation is converted into information independent of the formation during the extraction process, so that the voice features and directional features finally extracted are independent of the microphone array formation and are not affected by the formation.

[0032] Furthermore, after obtaining the speech separation result, in an embodiment of the present invention, interaction can be performed based on the speech separation result, thereby realizing independent interaction of each sound zone, that is, the speech signals of each sound zone obtained by separation can be responded to separately, and reply information can be generated and displayed so that the corresponding personnel on the target vehicle can know the reply information of the system in time, thereby realizing non-interference in the interaction of each seat and enjoying a unique interactive experience.

[0033] In the embodiment of the present invention, two types of formation-independent features obtained by combining signal processing and encoding are used for speech separation. This can achieve accurate positioning and separation of different sound sources without being affected by differences in microphone array formations, thereby completely solving the problem of "one car, one customization" and greatly improving delivery efficiency.

[0034] The speech separation method provided by the present invention extracts the speech features and directional features that are independent of the formation of the speech signal collected by the microphone array on the target vehicle, and performs speech separation based on the features. It can achieve complete isolation of the sound on the vehicle without relying on the formation information of the microphone array, and provide guarantee for the independent interaction of each sound zone. It avoids the problem that the speech separation on the vehicle in the traditional solution depends on the separation model obtained by training with the formation information, resulting in the dilemma of "one vehicle, one customization". Speech separation is performed based on features that are independent of the formation, which greatly improves the efficiency of large-scale delivery of the system, reduces costs, and reduces waste of resources.

[0035] Based on the above embodiment, step 130 includes: Based on the speech features and directional features, the speech separation model is applied to perform speech separation, and the speech separation result corresponding to the speech signal is obtained; The speech separation model is trained based on microphone signals collected by microphones at different distances in sample scenarios and the corresponding sample speech separation results.

[0036] Specifically, in step 130, the process of performing speech separation according to speech features and directional features to obtain a speech separation result corresponding to the speech signal can be implemented by a speech separation model. Here, specifically, based on the extracted speech features and directional features that are independent of the formation, the speech separation model is applied to perform speech separation on the speech signal, and the separated signal is output. In other words, the speech features and directional features of the extracted speech signal are input into the speech separation model, so that the speech separation model performs the speech separation task according to the input features, and outputs the separated signal, so that the speech separation result can be obtained.

[0037] It is worth noting that before applying this speech separation model for speech separation, it is usually necessary to pre-train the speech separation model so that it has better speech separation capabilities and can accurately separate the speech signals of each sound zone from the mixed signal, so that high-precision speech separation can be performed on the speech signals obtained from multi-channel pickup in practical applications.

[0038] In detail, the speech separation model can be obtained by pre-training with application samples and labels. Considering the problem caused by the current separation model training relying on the spatial information of the known formation, in an embodiment of the present invention, in order to make the trained speech separation model not restricted by the formation and applicable to vehicle-mounted interactive systems of various formations, the speech signals collected by microphones of different formations can be accurately separated. Here, multiple microphones with different spacings can be selected to collect signals, and the separated signals in the corresponding scenes can be marked, that is, the microphone signals collected by multiple microphones with different spacings in the sample scene are obtained, and the corresponding sample speech separation results in the sample scene are determined. Based on this, the model is trained so that the model can learn the corresponding relationship between input and output during the training process, so as to have the ability to accurately separate signals of different sound zones according to the features in the microphone signal that are not related to the formation, and then different sound sources can be efficiently and accurately located and separated in actual applications without being affected by the formation.

[0039] Based on the above embodiment, the speech separation model is trained based on the following steps: Get multi-channel microphone signals in sample scenarios; Encode the microphone signals of each channel, and fuse the features of the encoded microphone signals to obtain sample microphone features; Performing single-channel speech separation on each microphone signal to obtain a time-frequency mask of each pure microphone signal, and determining the sample directional characteristics based on the time-frequency mask of each pure microphone signal and the frequency domain phase of each microphone signal; Inputting the sample microphone feature and the sample directional feature into the initial speech separation model to obtain the predicted speech separation result output by the initial speech separation model; Based on the predicted speech separation results and the sample speech separation results, the parameters of the initial speech separation model are iterated to obtain a speech separation model.

[0040] Specifically, the training process of the speech separation model may include the following steps: Figure 2 It is the overall framework diagram of the speech separation model training process provided by the present invention, such as Figure 2 As shown, first, the speech signal in the sample scene needs to be determined, that is, the speech signals collected by multiple microphones with different spacings in the sample scene are obtained, which are referred to as microphone signals here, that is, the multi-channel microphone signals in the sample scene , , …, , is the total number of microphones.

[0041] Next, feature extraction can be performed on the multi-channel microphone signals, and the features extracted from the microphone signals that are independent of the formation can be fused to obtain speech features and directional features corresponding to the microphone signals, which are called sample microphone features and sample directional features.

[0042] Here, specifically, each microphone signal is encoded through a convolution layer to extract effective features of the signal, and the effective features obtained by encoding can be fused to obtain an overall speech feature corresponding to the multi-channel microphone signal, that is, a sample microphone feature. The fusion here can be addition, splicing, etc., which is not specifically limited in the embodiment of the present invention.

[0043] At the same time, each microphone signal can also be processed to extract features related to the signal direction, thereby obtaining sample directional features. Considering the mixed nature of the microphone signals obtained by picking up multiple microphones, that is, they may contain background noise, other environmental factors, etc., in order to improve the accuracy of speech separation, in an embodiment of the present invention, they can be roughly separated by single-channel speech separation, that is, each microphone signal can be first subjected to single-channel speech separation to obtain the time-frequency mask of the clean signal corresponding to each microphone signal, that is, the time-frequency mask of each pure microphone signal. Then, the characteristics of the signal direction can be determined based on this time-frequency mask and the frequency domain phase of the microphone signal. Specifically, the characteristics that are not related to the formation but related to the source direction of the microphone signal can be solved based on the time-frequency mask of each pure microphone signal and the frequency domain phase of each microphone signal to obtain the sample directional features.

[0044] Afterwards, the initial speech separation model can be applied to perform speech separation based on the sample microphone features and the sample directional features, thereby obtaining the corresponding speech separation results in the sample scene, that is, the predicted speech separation results. Here, specifically, the sample microphone features and the sample directional features are input into the initial speech separation model, so that the initial speech separation model performs speech separation on the microphone signal in the sample scene according to the input sample microphone features and the sample directional features, and outputs the separated signal, thereby obtaining the output predicted speech separation result. It should be noted that the initial speech separation model here is the initial model of the pre-constructed training process, which can be constructed on the basis of the existing multi-channel speech separation model. The specific model structure can be set accordingly according to actual conditions, specific needs, etc., and the embodiment of the present invention does not make specific limitations.

[0045] Then, the parameters of the initial speech separation model can be iterated according to the predicted speech separation result and the sample speech separation result to obtain the speech separation model. That is, the loss of the initial speech separation model in the speech separation task can be determined according to the result of the speech separation task output by the model and the label of the sample, and the parameters of the initial speech separation model can be adjusted according to this loss, so that the predicted speech separation result output by the model after the parameter adjustment is as close as possible to the sample speech separation result, and finally a trained speech separation model can be obtained.

[0046] Here, when measuring the loss of the initial speech separation model, the loss function used is SISNR (Scale-Invariant Signal-to-Noise Ratio) loss, and the final loss uses the PIT (Permutation Invariant Training) strategy, that is, the minimum loss values ​​among multiple combinations are arranged.

[0047] Based on the above embodiment, the sample directional characteristics are determined based on the time-frequency mask of each channel of pure microphone signal and the frequency domain phase of each channel of microphone signal, including: Determine the spatial covariance matrix of each clean microphone signal based on the time-frequency mask of each clean microphone signal; Based on the spatial covariance matrix of each pure microphone signal and the frequency domain phase of each microphone signal, the sample directional characteristics are determined.

[0048] Specifically, the process of determining the sample directional characteristics according to the time-frequency mask of each channel of pure microphone signal and the frequency domain phase of each channel of microphone signal specifically includes: First, according to the time-frequency mask of each clean microphone signal obtained by single-channel speech separation, the spatial covariance matrix of each clean microphone signal is determined, and the matrix can reflect the information of the signal source direction, so as to determine the directional characteristics of the sample. Here, specifically, the time-frequency mask that can make the arrangement of the single-channel speech separation result stable is selected in combination with the time-frequency mask of each clean microphone signal, and the spatial covariance matrix of each clean microphone signal is calculated based on this, so as to reflect the information of the corresponding signal in the direction of the signal source through this matrix.

[0049] Subsequently, the sample directional characteristics can be determined based on the spatial covariance matrix of each pure microphone signal and the frequency domain phase of each microphone signal. That is, based on the time-frequency mask and the frequency domain phase, the information related to the direction of the signal source is calculated to obtain the sample directional characteristics.

[0050] Based on the above embodiment, based on the spatial covariance matrix of each channel of pure microphone signal and the frequency domain phase of each channel of microphone signal, the sample directional characteristics are determined, including: Perform eigenvalue decomposition on the spatial covariance matrix of each clean microphone signal to obtain the steering vector of each clean microphone signal; Based on the steering vector of each pure microphone signal and the frequency domain phase of each microphone signal, the sample directional characteristics are determined.

[0051] Specifically, the process of determining the sample directional characteristics according to the spatial covariance matrix of each channel of pure microphone signal and the frequency domain phase of each channel of microphone signal may specifically include: First, the vector information in the direction of the signal source can be calculated based on the spatial covariance matrix of each clean microphone signal, so as to obtain the steering vector of each clean microphone signal. Specifically, the spatial covariance matrix of each clean microphone signal is subjected to eigenvalue decomposition to obtain the steering vector of each clean microphone signal.

[0052] Then, the sample directional characteristics can be determined based on the steering vector and the frequency domain phase of each microphone signal. That is, based on the information of each microphone signal in its source direction indicated by the steering vector and the phase information of the frequency domain signal corresponding to each microphone signal reflected by the frequency domain phase, the directional information independent of the formation is solved to obtain the sample directional characteristics.

[0053] Based on the above embodiment, the steering vector can be calculated by the following formula: In the formula, represents the steering vector of the clean microphone signal, represents the spatial covariance matrix of the clean microphone signal, represents eigenvalue decomposition.

[0054] Based on the above embodiment, based on the steering vector of each pure microphone signal and the frequency domain phase of each microphone signal, the sample directional characteristics are determined, including: Determine the time delay difference between two microphones in each microphone pair based on the steering vector of each pure microphone signal; Determine the phase difference between two microphones in each microphone pair based on the frequency domain phase of each microphone signal; Based on the time delay difference and phase difference of each microphone pair, the cosine distance of each microphone pair in the direction of the microphone signal source is determined, and based on the cosine distance of each microphone pair in the direction of the microphone signal source, the sample directivity feature is determined.

[0055] Specifically, the process of determining the sample directional characteristics according to the steering vectors of the pure microphone signals and the frequency domain phases of the microphone signals may include: Figure 3 is an overall block diagram of the sample directional feature determination process provided by the present invention, such as Figure 3 As shown, after single-channel speech separation and steering vector calculation, in an embodiment of the present invention, multiple microphone signals can be grouped in pairs to form multiple microphone pairs, and the time delay difference between the two microphones can be calculated based on the steering vectors of the pure microphone signals corresponding to the two microphones in each microphone pair, and the phase difference between the two microphones can be calculated based on the frequency domain phases of the microphone signals corresponding to the two microphones in each microphone pair, so that the information of the signal source direction corresponding to each microphone pair is obtained.

[0056] Here, specifically, the time delay difference between the two microphones in the microphone pair can be calculated based on the steering vector of a pure microphone signal and the steering vector of the pure microphone signal corresponding to the other microphone in the corresponding microphone pair; at the same time, the phase difference between the two microphones in the microphone pair can be determined based on the phase of the frequency domain signal of the microphone signals corresponding to the two microphones in the microphone pair.

[0057] Subsequently, considering that the delay difference and phase difference of the microphone pair calculated in the previous step are related to the microphone formation, they are information related to the formation. Therefore, in the embodiment of the present invention, after obtaining the delay difference and phase difference of each microphone pair, it is necessary to process them to convert the information related to the formation into a distance that is independent of the formation, that is, according to the delay difference and phase difference of each microphone pair, calculate the cosine distance of each microphone pair in the direction of the microphone signal source, and determine the sample directional characteristics through the cosine distance that is independent of the formation. That is, the directional characteristics of each microphone pair can be determined through the cosine distance of each microphone pair in the direction of the microphone signal source, and the directional characteristics of each microphone can be solved based on this, and finally the directional characteristics of each microphone are spliced ​​to obtain the sample directional characteristics.

[0058] It should be noted that, compared with the delay difference and phase difference related to the formation, in the embodiment of the present invention, the cosine distance in the direction of the microphone signal source solved is only related to the relative position between the signal source direction and the microphone, that is, whether the line between them forms a certain angle with the signal source direction, and does not involve the specific distance between the microphones or the shape of the entire array, and has nothing to do with the formation. In contrast, the signal delay difference and phase difference are related to the formation, which will be affected by the distance between the microphones. Under the same signal source direction, different microphone formations (such as linear arrays and circular arrays) will result in different distances between the microphones, resulting in different delay differences.

[0059] In short, since the cosine distance is only related to the direction of the signal source and the relative position between the microphones, and has nothing to do with the specific distance and formation of the microphones involved in the delay difference, the cosine distance will not change even if the formation of the microphones is changed (thereby changing the delay difference). This fundamentally solves the "one car, one customization" problem.

[0060] Based on the above embodiment, the sample directional characteristics are determined based on the following formula: In the formula, Indicates The directional characteristics of the microphone signal corresponding to the microphone. Indicates the total number of microphones, It is The number of microphone pairs consisting of the first microphone and the remaining microphones, Indicates 1st microphone and A microphone pair consisting of a pair of microphones. To include The set of all microphone pairs of the microphones in the and Respectively represent The phase of the frequency domain signal of the microphone signal corresponding to the first microphone The phase of the frequency domain signal of the microphone signal corresponding to the microphone. and Respectively represent The steering vector of the pure microphone signal corresponding to the first microphone and the The steering vector of the pure microphone signal corresponding to the microphone.

[0061] Based on the above embodiment, based on the time-frequency mask of each clean microphone signal, the spatial covariance matrix of each clean microphone signal is determined, including: Based on the time-frequency masks of the pure microphone signals of each channel, a median time-frequency mask is determined; Based on the median time-frequency mask and the microphone signals of each channel, the spatial covariance matrix of each channel's pure microphone signal is determined.

[0062] Specifically, the process of determining the spatial covariance matrix of each channel of clean microphone signal according to the time-frequency mask of each channel of clean microphone signal may specifically include: Taking into account that the signal arrangement separated from single-channel speech is not fixed, which is not conducive to the determination of directional characteristics, therefore, in an embodiment of the present invention, after obtaining the time-frequency masks of each clean microphone signal, the median time-frequency mask can be determined through a median solution strategy, that is, the median of the time-frequency masks of multiple clean microphone signals is determined, and it is used as the median time-frequency mask to participate in subsequent calculations, thereby improving the accuracy of the calculation.

[0063] After that, the spatial covariance matrix of each clean microphone signal can be calculated based on the median time-frequency mask and each microphone signal. That is, the spatial covariance matrix of each clean microphone signal is finally obtained by combining the median time-frequency mask determined in the previous step with each microphone signal.

[0064] Based on the above embodiment, the spatial covariance matrix of the pure microphone signal can be calculated by the following formula: In the formula, represents the spatial covariance matrix of the clean microphone signal, For the The median time-frequency mask of the frame, Indicates The microphone signal of the frame, is the total number of frames of the signal, is the transpose.

[0065] The median time-frequency mask can be determined by the following formula: In the formula, is the median time-frequency mask, It means finding the median. is the time-frequency mask of the pure microphone signal corresponding to the multi-channel microphone signal, Indicates the total number of microphones.

[0066] Based on the above embodiment, each microphone signal is encoded, and the features of each microphone signal obtained by encoding are fused to obtain a sample microphone feature, including: Encode each microphone signal, and perform nonlinear activation on the features of each microphone signal obtained by encoding to obtain each microphone feature; The microphone features of each channel are averaged and pooled to obtain the global microphone features of each channel, and the global microphone features of each channel are fused to obtain the sample microphone features.

[0067] Specifically, the process of encoding the microphone signals of each channel and fusing the features of the encoded microphone signals of each channel to obtain the sample microphone features may specifically include: Figure 4 is a general framework diagram of the sample microphone feature determination process provided by the present invention, such as Figure 4As shown in the figure, for the input multi-channel microphone signals, they are first encoded, and the features of the encoded microphone signals are nonlinearly activated to obtain the microphone features of each channel; then, the microphone features of each channel can be averaged and pooled to reduce the feature dimension to obtain the output features, which are used as global variables independent of the formation, that is, the global microphone features of each channel. After that, the global microphone features of each channel and the microphone features of each channel can be fused to obtain the sample microphone features.

[0068] Based on the above embodiment, the global microphone features of each channel are fused to obtain the sample microphone features, including: The global microphone feature of each channel is fused with the corresponding microphone feature to obtain the target microphone feature of each channel; The target microphone features are fused to obtain the sample microphone features.

[0069] Specifically, the process of fusing the global microphone features of various channels to obtain the sample microphone features may include: First, each global microphone feature can be fused with the microphone feature of this channel, that is, each global microphone feature and the microphone feature of this channel can be spliced ​​to obtain each target microphone feature. Then, each target microphone feature can be fused, that is, each target microphone feature can be spliced ​​to obtain the sample microphone feature.

[0070] The speech separation device provided by the present invention is described below. The speech separation device described below and the speech separation method described above can be referred to each other.

[0071] Figure 5 Schematic diagram of the structure of the speech separation device provided by the present invention. Figure 5 As shown, the device comprises: A signal acquisition unit 510 is used to acquire a voice signal on a target vehicle, wherein the voice signal is acquired by a microphone array mounted on the target vehicle; A feature extraction unit 520, configured to extract speech features and directional features of the speech signal, wherein the speech features and directional features are independent of the formation of the microphone array; The speech separation unit 530 is used to perform speech separation based on the speech feature and the directional feature to obtain a speech separation result corresponding to the speech signal.

[0072] The speech separation device provided by the present invention extracts the speech features and directional features that are independent of the formation of the speech signal collected by the microphone array on the target vehicle, and performs speech separation based on the features. It can achieve complete isolation of the sound on the vehicle without relying on the formation information of the microphone array, and provide guarantee for the independent interaction of each sound zone. It avoids the problem that the speech separation on the vehicle in the traditional solution depends on the separation model obtained by training with the formation information, resulting in the dilemma of "one car, one customization". Speech separation is performed based on features that are independent of the formation, which greatly improves the efficiency of large-scale delivery of the system, reduces costs, and reduces waste of resources.

[0073] Based on the above embodiment, the speech separation unit 530 is used for: Based on the speech feature and the directional feature, applying a speech separation model to perform speech separation to obtain a speech separation result corresponding to the speech signal; The speech separation model is trained based on microphone signals collected by microphones at different intervals in sample scenarios and corresponding sample speech separation results.

[0074] Based on the above embodiment, the device further includes a model training unit, which is used to: Acquire multi-channel microphone signals in the sample scene; Encode the microphone signals of each channel, and fuse the features of the encoded microphone signals to obtain sample microphone features; Performing single-channel speech separation on each microphone signal to obtain a time-frequency mask of each pure microphone signal, and determining a sample directional feature based on the time-frequency mask of each pure microphone signal and the frequency domain phase of each microphone signal; Inputting the sample microphone feature and the sample directional feature into an initial speech separation model to obtain a predicted speech separation result output by the initial speech separation model; Based on the predicted speech separation result and the sample speech separation result, the initial speech separation model is iterated on parameters to obtain a speech separation model.

[0075] Based on the above embodiment, the model training unit is used to: Determine the spatial covariance matrix of each channel of clean microphone signal based on the time-frequency mask of each channel of clean microphone signal; Based on the spatial covariance matrix of each channel of pure microphone signal and the frequency domain phase of each channel of microphone signal, the sample directional characteristics are determined.

[0076] Based on the above embodiment, the model training unit is used to: Performing eigenvalue decomposition on the spatial covariance matrix of each channel of the clean microphone signal to obtain a steering vector of each channel of the clean microphone signal; Based on the steering vectors of the pure microphone signals and the frequency domain phases of the microphone signals, the sample directional characteristics are determined.

[0077] Based on the above embodiment, the model training unit is used to: Determining a time delay difference between two microphones in each microphone pair based on the steering vectors of the pure microphone signals; Determining a phase difference between two microphones in each microphone pair based on the frequency domain phase of each microphone signal; Based on the time delay difference and phase difference of each microphone pair, the cosine distance of each microphone pair in the direction of the microphone signal source is determined, and based on the cosine distance of each microphone pair in the direction of the microphone signal source, the sample directivity feature is determined.

[0078] Based on the above embodiment, the model training unit is used to: Determine a median time-frequency mask based on the time-frequency masks of the pure microphone signals; Based on the median time-frequency mask and the microphone signals of each channel, a spatial covariance matrix of the pure microphone signals of each channel is determined.

[0079] Based on the above embodiment, the model training unit is used to: Encode each microphone signal, and perform nonlinear activation on the features of each microphone signal obtained by encoding to obtain each microphone feature; The microphone features are averaged and pooled to obtain global microphone features, and the global microphone features are fused to obtain sample microphone features.

[0080] Based on the above embodiment, the model training unit is used to: The global microphone feature of each channel is fused with the corresponding microphone feature to obtain the target microphone feature of each channel; The target microphone features are fused to obtain the sample microphone features.

[0081] Figure 6 is a schematic diagram of the structure of the vehicle-mounted interactive system provided by the present invention, such as Figure 6 As shown, the system includes a microphone array 610, a processor 620 and an audio player 630; The processor 620 is used to obtain the voice signal collected by the microphone array 610 on the target vehicle, and extract the voice features and directional features of the voice signal, perform voice separation based on the voice features and the directional features, obtain the voice separation result corresponding to the voice signal, and perform signal response based on the voice separation result, generate reply content, and control the audio player 630 to play the reply content; The speech feature and the directional feature are independent of the formation of the microphone array.

[0082] The vehicle-mounted interactive system provided by the present invention collects voice signals on the target vehicle through a microphone array, and extracts the voice features and directional features of the voice signals collected by the microphone array on the target vehicle through a processor, performs voice separation based on the features, obtains a voice separation result, performs a signal response based on the voice separation result, generates reply content, and controls an audio player to play the reply content. It can achieve independent sound zone interaction without relying on the formation information of the microphone array, avoids the problem in the traditional solution that the voice separation on the vehicle depends on the separation model obtained by training the formation information, resulting in the dilemma of "one car, one customization", and performs voice separation based on formation-independent features, which greatly improves the efficiency of large-scale delivery of the system, reduces costs, and reduces waste of resources.

[0083] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the speech separation method, which includes: obtaining a speech signal on a target vehicle, wherein the speech signal is collected by a microphone array mounted on the target vehicle; extracting speech features and directional features of the speech signal, wherein the speech features and the directional features are independent of the formation of the microphone array; performing speech separation based on the speech features and the directional features to obtain a speech separation result corresponding to the speech signal.

[0084] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0085] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the speech separation method provided by the above-mentioned methods, and the method includes: obtaining a speech signal on a target vehicle, and the speech signal is collected by a microphone array loaded on the target vehicle; extracting speech features and directional features of the speech signal, and the speech features and the directional features are independent of the formation of the microphone array; performing speech separation based on the speech features and the directional features to obtain a speech separation result corresponding to the speech signal.

[0086] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the speech separation method provided by the above-mentioned methods, the method comprising: obtaining a speech signal on a target vehicle, wherein the speech signal is collected by a microphone array mounted on the target vehicle; extracting speech features and directional features of the speech signal, wherein the speech features and the directional features are independent of the formation of the microphone array; performing speech separation based on the speech features and the directional features to obtain a speech separation result corresponding to the speech signal.

[0087] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0088] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech separation method, characterized in that: include: Acquiring a voice signal on a target vehicle, wherein the voice signal is collected by a microphone array mounted on the target vehicle; Extracting speech features and directional features of the speech signal, wherein the speech features and the directional features are independent of the formation of the microphone array; Speech separation is performed based on the speech feature and the directional feature to obtain a speech separation result corresponding to the speech signal.

2. The speech separation method according to claim 1, characterized in that: The performing speech separation based on the speech feature and the directional feature to obtain a speech separation result corresponding to the speech signal includes: Based on the speech feature and the directional feature, applying a speech separation model to perform speech separation to obtain a speech separation result corresponding to the speech signal; The speech separation model is trained based on microphone signals collected by microphones at different intervals in sample scenarios and corresponding sample speech separation results.

3. The speech separation method according to claim 2, characterized in that: The speech separation model is trained based on the following steps: Acquire multi-channel microphone signals in the sample scene; Encode the microphone signals of each channel, and fuse the features of the encoded microphone signals to obtain sample microphone features; Performing single-channel speech separation on each microphone signal to obtain a time-frequency mask of each pure microphone signal, and determining a sample directional feature based on the time-frequency mask of each pure microphone signal and the frequency domain phase of each microphone signal; Inputting the sample microphone feature and the sample directional feature into an initial speech separation model to obtain a predicted speech separation result output by the initial speech separation model; Based on the predicted speech separation result and the sample speech separation result, the initial speech separation model is iterated on parameters to obtain a speech separation model.

4. The speech separation method according to claim 3, characterized in that: The determining of the sample directional characteristics based on the time-frequency mask of each channel of the pure microphone signal and the frequency domain phase of each channel of the microphone signal includes: Determine the spatial covariance matrix of each channel of clean microphone signal based on the time-frequency mask of each channel of clean microphone signal; Based on the spatial covariance matrix of each channel of pure microphone signal and the frequency domain phase of each channel of microphone signal, the sample directional characteristics are determined.

5. The speech separation method according to claim 4, characterized in that: The determining of the sample directional characteristics based on the spatial covariance matrix of the pure microphone signals of each channel and the frequency domain phase of the microphone signals of each channel includes: Performing eigenvalue decomposition on the spatial covariance matrix of each channel of the clean microphone signal to obtain a steering vector of each channel of the clean microphone signal; Based on the steering vectors of the pure microphone signals and the frequency domain phases of the microphone signals, the sample directional characteristics are determined.

6. The speech separation method according to claim 5, characterized in that: The determining of the sample directional characteristics based on the steering vectors of the pure microphone signals of each channel and the frequency domain phases of the microphone signals of each channel includes: Determining a time delay difference between two microphones in each microphone pair based on the steering vectors of the pure microphone signals; Determining a phase difference between two microphones in each microphone pair based on the frequency domain phase of each microphone signal; Based on the time delay difference and phase difference of each microphone pair, the cosine distance of each microphone pair in the direction of the microphone signal source is determined, and based on the cosine distance of each microphone pair in the direction of the microphone signal source, the sample directivity feature is determined.

7. The speech separation method according to claim 4, characterized in that: The determining of the spatial covariance matrix of each channel of clean microphone signal based on the time-frequency mask of each channel of clean microphone signal comprises: Determine a median time-frequency mask based on the time-frequency masks of the pure microphone signals; Based on the median time-frequency mask and the microphone signals of each channel, a spatial covariance matrix of the pure microphone signals of each channel is determined.

8. The speech separation method according to any one of claims 3 to 7, characterized in that: The encoding of the microphone signals of each channel and fusing the features of the encoded microphone signals of each channel to obtain the sample microphone features include: Encode each microphone signal, and perform nonlinear activation on the features of each microphone signal obtained by encoding to obtain each microphone feature; The microphone features are averaged and pooled to obtain global microphone features, and the global microphone features are fused to obtain sample microphone features.

9. The speech separation method according to claim 8, characterized in that: The fusing of the global microphone features to obtain sample microphone features includes: The global microphone feature of each channel is fused with the corresponding microphone feature to obtain the target microphone feature of each channel; The target microphone features are fused to obtain the sample microphone features.

10. A speech separation device, characterized in that: include: A signal acquisition unit, used to acquire a voice signal on a target vehicle, wherein the voice signal is acquired by a microphone array mounted on the target vehicle; A feature extraction unit, used for extracting speech features and directional features of the speech signal, wherein the speech features and the directional features are independent of the formation of the microphone array; The speech separation unit is used to perform speech separation based on the speech feature and the directional feature to obtain a speech separation result corresponding to the speech signal.

11. A vehicle-mounted interactive system, characterized in that: Includes microphone array, processor and audio player; The processor is used to obtain a voice signal collected by a microphone array on a target vehicle, extract voice features and directional features of the voice signal, perform voice separation based on the voice features and the directional features, obtain a voice separation result corresponding to the voice signal, perform a signal response based on the voice separation result, generate reply content, and control the audio player to play the reply content; The speech feature and the directional feature are independent of the formation of the microphone array.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the speech separation method according to any one of claims 1 to 9 is implemented.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speech separation method according to any one of claims 1 to 9 is implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the speech separation method according to any one of claims 1 to 9 is implemented.

Images