Audio partitioning model training method and device, audio partitioning method and device, equipment and medium

Abstract

The invention relates to the technical field of voice processing, and discloses an audio partitioning model training method and device, an audio partitioning method and device, equipment and a medium, and the method comprises the steps: obtaining a trained teacher network; determining a target input channel number and a target output channel number required by the student network, wherein the target output channel number of the student network is not greater than the output channel number of the teacher network; adjusting an intermediate structure of the teacher network according to the target input channel number and the target output channel number to obtain an initial student network; acquiring audio training data, and inputting the audio training data into the initial student network and the teacher network; and pruning the initial student network according to the error of the output result between the teacher network and the initial student network and the neuron performance in the initial student network to obtain a target student network. According to the invention, on the premise that the training precision is ensured, the model can be flexibly adjusted according to the actual microphone number and the partition number demand.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, specifically to an audio partitioning model training, audio partitioning method, apparatus, device, and medium. Background Technology

[0002] In-vehicle speech region separation algorithms have become a research hotspot in speech signal processing in recent years. It is a technical branch of the speech separation field. Through a trained speech recognition model, multiple voice signals are collected and then different speech regions are identified. Only the voice of the speaker in each speech region is preserved, while other sound sources and noise outside the speech region are suppressed.

[0003] Due to the limited computing power of vehicle chips, large-scale speech recognition models are difficult to run quickly on vehicles. Therefore, the industry commonly uses knowledge distillation to train a small speech recognition model that can run quickly on vehicles. There are two main training methods for knowledge distillation: probability-based and feature-based. Probabilistic knowledge distillation involves approximating the output of the student network (a small speech recognition model) to the data output of the teacher network (a large-scale speech recognition model). This method simply allows the teacher network to tell the student network the answer, while the student network must figure out the intermediate thought process itself, resulting in poor learning outcomes. Feature-based distillation involves approximating certain intermediate layer features of the student network to the corresponding intermediate layer features of the teacher network, essentially allowing the student network to learn the teacher network's intermediate thought process. However, this method requires the input and output channels of the teacher and student networks to be identical, meaning the model structure must be identical except for the number of layers, in order to learn intermediate features. This is problematic for real-world production scenarios where input and output channels vary flexibly (changes in the number of microphones lead to changes in the number of input channels, and changes in the number of partitions lead to changes in the number of output channels), requiring the pre-training of many different teacher networks, resulting in high training costs. Summary of the Invention

[0004] This invention provides a training method, apparatus, device, and medium for an audio zoning model, to solve the problem that the input and output channels of an in-vehicle audio zoning network are difficult to flexibly adjust according to the actual number of microphones and zoning requirements of the vehicle while ensuring training accuracy.

[0005] In a first aspect, the present invention provides a training method for an audio partitioning model, the method comprising: acquiring a trained teacher network, the teacher network being used to partition audio based on audio data input from multiple microphones and output multiple partitioned audio; determining the target number of input channels and the target number of output channels required by a student network, wherein the target number of output channels in the student network is not greater than the number of output channels in the teacher network; adjusting the intermediate structure of the teacher network according to the target number of input channels and the target number of output channels to obtain an initial student network; acquiring audio training data and inputting the audio training data into the initial student network and the teacher network; pruning the initial student network based on the error between the output results of the teacher network and the initial student network, and based on the neuron performance in the initial student network, to obtain a target student network.

[0006] In one alternative implementation, the teacher network is created based on a network structure of an encoder, a voice region separation network, and a decoder, wherein the encoder includes a downsampling layer, the decoder includes an upsampling layer, and the features of the decoder before the upsampling layer are added to the features of the encoder after the downsampling layer.

[0007] In one optional implementation, the step of pruning the initial student network to obtain the target student network based on the error between the output results of the teacher network and the initial student network, and based on the neuron performance in the initial student network, includes: identifying sub-networks that are repeatedly used in the initial student network, and removing the first target neurons that are repeatedly used in the sub-networks; calculating a first loss value based on the error between the output results of the teacher network and the initial student network in this training; calculating a second loss value based on the error between the output results of the initial student network and the corresponding labels of the training data in this training; and weightedly fusing the first loss value and the second loss value. The process involves obtaining a first fusion loss value; acquiring a second fusion loss value from the previous training session; determining a second target neuron in the initial student network that does not affect the model results during the previous training session when the difference between the first and second fusion loss values ​​does not exceed a preset threshold, and then removing the second target neuron; repeating the steps of calculating the first loss value based on the error between the output results of the teacher network and the initial student network, until the second target neuron that does not affect the model results during training is determined and removed when the difference between the first and second fusion loss values ​​does not exceed a preset threshold, and performing multiple training sessions to obtain the target student network.

[0008] In one alternative implementation, the second target neuron includes neurons with weight values ​​lower than a preset weight threshold, and neurons that were adjusted during the previous training.

[0009] In an optional implementation, before repeating the steps of calculating a first loss value based on the error between the output results of the teacher network and the initial student network, and determining a second target neuron that does not affect the model results during training and deleting the second target neuron when the loss difference between the first fusion loss value and the second fusion loss value does not exceed a preset threshold, and performing multiple training sessions to obtain the target student network, the method further includes: updating the remaining model parameters of the initial student network based on the first fusion loss value after deleting neurons in this iteration.

[0010] In one optional implementation, the training data includes first training data and second training data, wherein the microphone position of the first training data is the same as the microphone position of the data used when training the teacher network, and the microphone position of the second training data is different from the microphone position of the data used when training the teacher network.

[0011] Secondly, the present invention provides an audio partitioning method, the method comprising: receiving audio data collected by a plurality of microphones; inputting the audio data into a target student network trained by any one of the first aspects, wherein the number of input channels of the target student network is the same as the number of microphones; and obtaining partitioned audio based on the output results of the target student network.

[0012] Thirdly, the present invention provides a training apparatus for an audio partitioning model, the apparatus comprising: a teacher network acquisition module, configured to acquire a trained teacher network, the teacher network being used to partition audio based on audio data input from multiple microphones and output multiple partitioned audio; a channel determination module, configured to determine the target number of input channels and the target number of output channels required by a student network, wherein the target number of output channels of the student network is not greater than the number of output channels of the teacher network; a student network acquisition module, configured to adjust the intermediate structure of the teacher network based on the target number of input channels and the target number of output channels to obtain an initial student network; a training module, configured to acquire audio training data and input the audio training data into the initial student network and the teacher network; and a pruning module, configured to prune the initial student network based on the error between the output results of the teacher network and the initial student network, and based on the neuron performance in the initial student network, to obtain a target student network.

[0013] Fourthly, the present invention provides an audio partitioning device, the device comprising: a data acquisition module for receiving audio data acquired by a plurality of microphones; a detection module for inputting the audio data into a target student network trained by any one of the first aspects, wherein the number of input channels of the target student network is the same as the number of microphones; and a result output module for obtaining partitioned audio based on the output result of the target student network.

[0014] Fifthly, the present invention provides an electronic device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the method described in the first aspect or any corresponding embodiment thereof.

[0015] In a sixth aspect, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the method described in the first aspect or any corresponding embodiment thereof.

[0016] In a seventh aspect, the present invention provides a computer program product, including computer instructions for causing a computer to perform the method described in the first aspect or any corresponding embodiment thereof.

[0017] The technical solution provided by this invention has the following advantages: (1) Based on the above technical means, by obtaining the trained teacher network, adjusting the network structure to obtain the initial student network, and combining the error and neuron performance pruning process, the core pain points of "poor channel flexibility" and "difficulty in balancing training accuracy and efficiency" of audio partitioning models in vehicle scenarios are effectively solved. The student network is derived from the teacher network through pruning. The number of channels and intermediate hierarchical structure are adjusted on the basis of the teacher network. There is no need to fit and approximate intermediate features. It can also learn the intermediate thinking mode of the teacher network. It breaks through the limitation of "the input and output channels of the teacher and student networks must be consistent" in traditional knowledge distillation. It allows flexible adjustment according to the number of microphones in the actual vehicle (input channels) and the sound zone requirements (output channels). There is no need to pre-train a large number of teacher networks for different vehicle models, which greatly reduces the training cost and development cycle. Secondly, the pruning process simultaneously refers to "the output error of the teacher and student networks" and "the neuron performance of the initial student network". It integrates the probabilistic knowledge distillation and pruning techniques. It ensures that the student network inherits the core sound zone separation capability of the teacher network and can accurately remove redundant neurons, making the model parameters more compact, adapting to the limited computing power of the vehicle system, and realizing real-time audio partitioning. The aforementioned technologies do not rely on specific hardware layouts and can be quickly adapted to vehicle models with different numbers of microphones and different sound zone requirements, significantly improving the versatility and practicality of the technology and providing an efficient and flexible model training solution for in-vehicle voice interaction scenarios.

[0018] (2) Based on the above technical means, the teacher network adopts an architecture of "encoder, region separation network, decoder". The downsampling layer of the encoder and the upsampling layer of the decoder achieve residual linking through feature addition, which further optimizes the model training effect and region separation accuracy. On the one hand, encoder downsampling can reduce the number of parameters of the subsequent region separation network, reduce computational complexity, and adapt to the computing power of the vehicle system; on the other hand, the residual linking design that incorporates encoder downsampling features before decoder upsampling can effectively avoid the gradient explosion problem in model training, and at the same time make up for the sound detail features lost during downsampling, ensuring that the student network can still maintain excellent region separation performance after parameter simplification.

[0019] (3) Based on the above technical means, redundant neurons in the reusable subnetwork are deleted first, which can quickly remove repetitive and inefficient calculation modules in the model and initially realize parameter simplification. Secondly, by combining the "output error between teacher and student networks (first loss)" and the "error between student network and label (second loss)," the learning effect and accuracy deviation of student network can be comprehensively evaluated through the network output results, avoiding pruning bias caused by single loss. Finally, by comparing the fusion loss value of adjacent training rounds, "neurons that do not affect the result" are deleted only when the model converges and stabilizes, ensuring that the pruning process is gradual, ensuring that the model parameters are continuously simplified to adapt to the vehicle's computing power, and retaining the core ability of sound region separation to the greatest extent, thus achieving a balance between "simplification" and "accuracy."

[0020] (4) According to the above technical means, the second target neurons that do not affect the results include "neurons with weight values ​​lower than the preset threshold" and "neurons adjusted in the last training". On the one hand, neurons with low weight values ​​contribute very little to the model output. Deleting such neurons can significantly simplify the parameters without affecting the accuracy and reduce the computational burden on the vehicle-machine interface. On the other hand, since the loss value of this training has not changed significantly, it can be said that the neurons adjusted in the last training do not have a major impact on the detection effect of the model, so they can be discarded. This technical means makes the pruning process more precise and controllable, avoids blindly deleting effective neurons, further improves the pruning efficiency and model stability, and makes the final target student network both compact and efficient, and able to stably output high-quality sound region separation results.

[0021] (5) Based on the above technical means, by updating the remaining parameters in reverse based on the fusion loss value after each neuron deletion, the remaining core parameters can be quickly adapted to the pruned network structure, making up for the slight loss in feature extraction ability caused by the deletion of neurons. This technical means ensures that the model maintains the rationality and adaptability of parameters throughout the pruning process, avoids a cliff-like drop in accuracy after pruning, and ensures that the model can converge stably after each round of pruning, ultimately obtaining a target student network that is "parameter-simplified, highly accurate, and adapted to vehicle systems".

[0022] (6) Based on the above technical means, when the input channels of the teacher network and the student network are the same, the training data with different microphone positions in the student network and the teacher network can also be adjusted, which solves the generalization problem of the traditional model that "only adapts to a single microphone position and the accuracy drops sharply after the actual vehicle position changes".

[0023] (7) Based on the above technical means, the target student network obtained by the aforementioned training method provides an efficient and accurate in-vehicle audio partitioning implementation solution, which directly solves the problem of "poor real-time performance and low accuracy of audio partitioning" in real vehicle scenarios. The target student network is optimized by pruning, with compact parameters and low computing power requirements, and can realize real-time audio processing on the vehicle system, avoiding instruction response delay caused by model complexity; secondly, the student network inherits the high-precision partitioning capability of the teacher network, and has strong generalization ability after training with multi-location data. Regardless of the number and location of microphones in the real vehicle, it can accurately separate the target sound region, suppress non-target sound sources and noise, and improve the accuracy of in-vehicle voice interaction (such as voice commands and calls). Attached Figure Description

[0024] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of a sound region separation network based on related technologies; Figure 2 This is a flowchart illustrating a training method for an audio partitioning model according to an embodiment of the present invention. Figure 3 This is a flowchart illustrating an audio partitioning method according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a training device for an audio partitioning model according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an audio partitioning device according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0028] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0029] The in-vehicle audio frequency separation method can detect the spatial location of each sound source in the multi-channel audio signals collected by multiple sound sensors in the vehicle, based on a pre-trained target neural network model. This results in the spatial location of each sound source in the multi-channel audio signals, including both in-vehicle and out-of-vehicle sound source locations. The method suppresses sound signals whose spatial location is an out-of-vehicle sound source, thus obtaining the sound signals corresponding to the in-vehicle sound source locations. Finally, it identifies the frequency range of the sound signals corresponding to the in-vehicle sound source locations and performs frequency separation based on the identification results.

[0030] In-vehicle speech region separation algorithms have become a research hotspot in speech signal processing in recent years. It's a branch of speech separation technology that aims to process multiple acquired speech signals and identify different speech regions. Within each region, only the speaker's voice is preserved, while other sound sources and noise outside the region are suppressed. The number of input channels varies depending on the application requirements, typically two, four, or six channels. Currently, the output speech regions are mainly two, four, five, or six.

[0031] Currently, the commonly used vehicle audio zone separation frameworks in academia and industry include... Figure 1As shown, the audio region separation network mainly consists of three parts: an encoder, an audio region separation network, and a decoder. This network can be named TFasnet (Time-Frequence-domain audio separation network). The encoder's role is to transform the time-frequency domain signal of the input mixed speech into another latent space, forming an embedding vector representation of the mixed speech in this latent space. The audio region separation network's role is to perform target speech recognition in the latent space, converting the processed multi-channel time-frequency domain data into a mask, using the mask to suppress speech signals in non-target audio regions, and identifying the vector of the target audio region. The decoder's role is to convert the vector of the target audio region back into a time-frequency domain signal, i.e., the existing audio region separation technology framework. Each module is a neural network. For example, the encoder / decoder can use a 1D convolutional network (1d-CNN), and the audio region separation network can use LSTM, TCN, DPRNN, etc. The loss function obtained during training can be mean squared error or SI-SDR criterion. In this way, different network structures, objective functions, and training methods of different modules can be combined to create numerous algorithms.

[0032] Due to the limited computing power of vehicle chips, large-scale speech recognition models are difficult to run quickly on vehicles. Therefore, the industry commonly uses knowledge distillation techniques to train a small speech recognition model that can run quickly on vehicles. There are two main training methods for knowledge distillation: probability-based and feature-based. Probabilistic knowledge distillation aims to make the output of the student network (the small speech recognition model) approximate the data results of the teacher network (the large-scale speech recognition model). This is achieved by minimizing the difference in KL divergence between the predicted probabilities of the student and teacher for the input data. However, probability-based knowledge distillation does not fully utilize the teacher's knowledge, such as features in other layers of the teacher's network. This technique only allows the teacher network to tell the student network the answer, while the student network needs to figure out the intermediate thought process on its own, resulting in poor learning outcomes.

[0033] To leverage features from other layers of the teacher network, a feature-based distillation method has been proposed. This method encourages the student network to share similarities with the teacher network in intermediate layer features, allowing certain intermediate layer features of the student network to approximate the corresponding intermediate layer features of the teacher network. This technique essentially allows the student network to learn the intermediate thought processes of the teacher network. However, this distillation method requires the teacher and student networks to have identical input and output channels, meaning their model structures must be identical except for the number of layers, in order to learn intermediate features. For real-world production scenarios with flexible input and output channels (changes in the number of microphones leading to changes in the number of input channels, and changes in the number of partitions leading to changes in the number of output channels), many different teacher networks need to be pre-trained, resulting in high training costs. To address these issues, this invention provides the following technical solution.

[0034] According to an embodiment of the present invention, a training method for an audio partitioning model is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0035] This embodiment provides a training method for an audio partitioning model, which can be used with computer devices. Figure 2 This is a flowchart of a training method for an audio partitioning model according to an embodiment of the present invention, the process including the following steps: Step S201: Obtain the trained teacher network. The teacher network is used to partition the audio based on the audio data input from multiple microphones and output multiple partitioned audio.

[0036] Specifically, the teacher network refers to a deep learning network with a large number of parameters and high-precision audio partitioning capabilities. Its training objective is to fully learn the mapping relationship between multi-microphone audio data and sound zone classification in an in-vehicle scenario, providing a "knowledge transfer foundation" for the subsequent student network. In this embodiment of the invention, the audio data input by the microphones are time-domain audio signals collected by multiple microphones deployed in different locations inside the vehicle (such as door handles, rearview mirrors, roof, etc.). Each microphone corresponds to one input channel, and multiple signals can reflect the spatial location characteristics of the sound source (such as phase difference, amplitude difference). Audio partitioning divides the in-vehicle space into several preset areas (such as driver's seat, front passenger seat, rear left seat, rear right seat, etc.), and then the teacher network processes the mixed audio to separate the independent audio of each area, suppressing sound sources and noise in non-target areas. The teacher network obtained in this embodiment of the invention needs to be pre-trained and reach a stable convergence state. The teacher network structure adopts a three-order architecture of "encoder - voice region separation network - decoder". The specific architecture can refer to the existing 1D convolutional networks 1d-cnn, LSTM, TCN, DPRNN, etc., which will not be elaborated in this embodiment of the invention. In this embodiment of the invention, the experimental data of both the teacher network and the student network are obtained through simulation. Based on the size of the car model and the position of the microphone, the number of voice channels that can be collected in the simulated car is determined. For example, four microphones are located above the car door. During the simulation, a simulation function is used to assume that the car is a cuboid shape. The microphones and speakers in the corresponding positions in the car are simulated according to their relative positions. In order to increase the generalization ability of the model, the position of the speaker needs to be spread across various coordinate points as much as possible during the simulation data. Since the real shape of the car model is close to a trapezoid, the coordinates of the microphones in the cuboid are also slightly changed during the simulation to make the simulation data closer to the real data. In this way, the data obtained by simulation covers various data scenarios in the real scene as much as possible. During the simulation, the data should primarily consist of clean speech. To enhance the model's robustness, a certain proportion of noise data can be added; the specific proportion can be adjusted according to actual needs. The total amount of simulation data should be at least one hundred hours.

[0037] The loss function used is the scale-invariant signal-to-noise ratio (SI-SDR) loss function, which takes the target audio region of the input label and the predicted audio output of the teacher network as input. Minimizing this loss function makes the output of the teacher network approximate the true label. For example, if the teacher network is designed with "6 input channels (6 microphones, deployed on the left and right front door handles, the left and right rear door handles, and the left and right front rearview mirrors), and 4 output audio regions (driver's seat, passenger seat, left rear seat, and right rear seat)," then after training, the teacher network can receive the mixed audio collected by the 6 microphones and output audio signals of 4 independent audio regions. It can also suppress external noise (such as engine noise and wind noise) and interference signals from non-target audio regions (such as the passenger's voice interfering with the driver's audio).

[0038] Step S202: Determine the target number of input channels and the target number of output channels required by the student network, wherein the target number of output channels of the student network is no greater than the number of output channels of the teacher network.

[0039] Specifically, the target number of input channels refers to the number of in-vehicle microphones adapted for the student network. This is directly related to the actual vehicle hardware layout and needs to be determined based on the vehicle model configuration (low-end, mid-range, high-end). For example, if a low-end model is only equipped with 4 microphones, then the target number of input channels is 4. The target number of output channels refers to the number of audio zones that the student network needs to separate. This is related to the in-vehicle voice interaction requirements. For example, if only the "front row" and "rear row" areas need to be distinguished, then the target number of output channels is 2; if four independent seats need to be distinguished, then the target number of output channels is 4.

[0040] This invention requires consideration of actual vehicle deployment needs to determine the channel parameters of the student network. First, the hardware configuration of the target vehicle model is investigated. For example, if the target vehicle is a compact sedan (low-spec version), four microphones can be deployed only in the front and rear door handles (no microphones in the rearview mirrors), then the target number of input channels for the student network is determined to be four. Second, user interaction needs are investigated. For example, if the voice interaction function of this vehicle model only needs to "prioritize front-seat commands and suppress rear-seat interference" (e.g., when controlling the air conditioning or navigation, only front-seat voices are recognized), then it is not necessary to separate the four individual seats, and the target number of output channels for the student network is determined to be two (front-seat audio zone and rear-seat audio zone).

[0041] It is important to note that the target number of output channels in the student network must be less than or equal to the number of output channels in the teacher network. For example, if the teacher network has 4 output channels and the target number of output channels in the student network is 2, the constraint is satisfied. However, if the teacher network has 2 output channels and the target number of output channels in the student network is 4, the constraint is not satisfied. In this case, a teacher network with a larger number of output channels must be selected, or the output requirements of the student network must be adjusted (e.g., merging the 4 registers into 2 registers). This is because if the teacher network does not have the ability to recognize more registers, the student network trained through knowledge distillation for recognizing more registers will also lack the corresponding ability.

[0042] If the target vehicle is a mid-range model equipped with 5 microphones (front left and right door handles, rear left and right door handles, and center of the roof), and the user requirement is "separation of the driver's seat from other areas (passenger seat + rear seats)", then the target number of input channels for the student network is determined to be 5, and the target number of output channels is determined to be 2 (driver's seat audio area, other area audio areas); if the number of output channels for the teacher network is 4 (satisfying 2≤4), then the subsequent steps can continue.

[0043] Step S203: Adjust the intermediate structure of the teacher network according to the target number of input channels and the target number of output channels to obtain the initial student network.

[0044] Specifically, in this embodiment of the invention, the student network is obtained by pruning the teacher network. By pruning the teacher network to obtain the student network, even if the number of input and output channels of the student network is different from that of the teacher network, there is no need to approximate and learn intermediate features, thereby indirectly achieving the effect of feature-based knowledge distillation learning.

[0045] Based on this, when the number of input / output channels in the student network differs from that in the teacher network, the structure of the student network will change compared to the teacher network. This also involves adjusting the intermediate structure of the teacher network to obtain the initial student network before pruning. The intermediate structure adjustment refers to structurally modifying the encoder input (input channel adaptation), decoder output (output channel adaptation), and the core layer of the audio region separation network (feature dimension adaptation) of the teacher network. This ensures that the input / output channels of the adjusted network (initial student network) match the target parameters while retaining the core feature extraction capabilities of the teacher network. The initial student network still has more parameters than the final target student network, requiring subsequent pruning optimization to adapt to the vehicle's computing power.

[0046] When the number of teacher input channels is not equal to the number of student target input channels, input channel adaptation is required. For example, in case 1: assuming the number of teacher input channels is greater than the number of student target input channels (e.g., 6 channels for teachers, 4 channels for students), at the input end of the teacher network encoder, the "feature importance score" of each input channel can be calculated. By calculating the L1 norm of each channel weight during training, the top 4 channels with the highest scores (e.g., the 4 channels corresponding to the front left and right door handles and the rear left and right door handles) are retained, and the convolutional kernel weights and corresponding input layers of the remaining 2 channels (the channels corresponding to the front left and right rearview mirrors) are deleted, thus adjusting the number of encoder input channels from 6 to 4. For example, in scenario 2: Assuming the number of teacher input channels is less than the number of student target input channels (e.g., 4 channels for teachers, 5 channels for students), add a new convolutional kernel weight for one input channel at the input end of the teacher network encoder. The new weight can be generated by linearly interpolating the weights of the two closest channels in the teacher network (e.g., adding a center microphone channel on the roof, whose weight is obtained by interpolating the weights of the left and right door handle channels in the front row at a ratio of 0.5:0.5). At the same time, adjust the input dimension of the first layer of the encoder convolution to change the number of input channels from 4 to 5.

[0047] When the number of teacher output channels is less than or equal to the number of student target output channels, output channel adaptation is required. Assuming the number of teacher output channels is greater than the number of student target output channels, a "range aggregation" strategy is used to adjust the number of output channels at the decoder output end of the teacher network. For example, if the teacher outputs 4 ranges (driver's seat, front passenger seat, rear left seat, rear right seat), and the student's target output is 2 ranges (front row, rear row), then the weights of the two output branches corresponding to "driver's seat + front passenger seat" in the teacher decoder are averaged and merged, as are the weights of the two output branches corresponding to "rear left seat + rear right seat". The original 4 output branches are deleted, and the merged 2 output branches are retained, thus adjusting the decoder output channel number from 4 to 2. If the number of teacher output channels equals the number of student target output channels, then no adjustment to the output end structure is needed.

[0048] Because the adjustment of input / output channels causes a mismatch in the feature dimensions of the register separation network (e.g., after adjusting the input channels from 6 to 4, the dimension of the encoder output embedding vector changes from [6×512] to [4×512]), the intermediate layer features of the teacher network also need to be adapted in terms of dimension. For example, adding a 1D convolutional layer (with a kernel size of 3 and a stride of 1) to the first layer of the register separation network converts the feature dimension to [4×512] (consistent with the adjusted input dimension) to ensure the continuity of network signal transmission.

[0049] The fact that the teacher network has more output channels than the student network means it can learn and output finer-grained features (such as more subdivided vocal ranges and richer sound details). During knowledge distillation, the student network indirectly acquires the distribution patterns of these fine-grained features by imitating the teacher network's output. For example, if the teacher network outputs four vocal ranges (driver's seat, front passenger seat, rear left, rear right) and the student network outputs two (front, rear), the student network will automatically associate the spatial relationships between the four vocal ranges in the teacher network (such as the feature differences between the driver's seat and the front passenger seat) during learning. This allows for more accurate aggregation of front / rear features, avoiding information loss due to fewer output channels and ultimately improving its own segmentation accuracy.

[0050] The teacher network has fewer input channels than the student network, indicating that it relies on fewer raw signals during training (e.g., using only two microphones) yet achieves higher accuracy, demonstrating that it has learned core discriminative features (such as phase and amplitude differences at key locations). When the student network has more input channels (e.g., four microphones), by mimicking the teacher network's output, it can focus on the core features extracted by the teacher network, utilizing more raw information and avoiding interference from noise in redundant channels. For example, if the teacher network uses two microphones (driver's seat and rear center) to achieve high-precision zoning, the student network, learning with four microphones, will prioritize the core features relied upon by the teacher network while supplementing details with additional channel data, ultimately achieving better robustness and accuracy than the teacher network under multi-channel input.

[0051] Step S204: Obtain audio training data and input the audio training data into the initial student network and teacher network.

[0052] Step S205: Based on the error between the output results of the teacher network and the initial student network, and based on the neuron performance in the initial student network, prune the initial student network to obtain the target student network.

[0053] Specifically, the audio training data is used for the distillation training of the student network. It needs to be adapted to the input channel parameters of both the teacher network and the initial student network, including "data matching the input of the teacher network" and "data matching the input of the student network" to ensure that the two can process the data and calculate the error synchronously during the distillation process.

[0054] The error between the output of the teacher network and the initial student network refers to the difference between the baseline audio output by the teacher network and the predicted audio output by the initial student network. It is quantified by a loss function and is used to measure the degree to which the student network has learned the "knowledge" of the teacher network.

[0055] In this embodiment of the invention, neuron performance refers to the parameter characteristics such as activation value and weight value of each neuron in the initial student network, which is used to determine the contribution of the neuron to the network output. Neurons with low contribution are "redundant neurons" and can be pruned.

[0056] Pruning means removing redundant neurons, convolutional kernels, or channels from a network. While maintaining the network's accuracy, it reduces the number of parameters and computational load, making the network adaptable to the limited computing power of the vehicle's infotainment system.

[0057] The embodiments of the present invention adopt a fusion strategy of "hierarchical pruning and probability-based knowledge distillation", which is executed in three steps: "loss calculation and parameter update", "redundant neuron identification" and "pruning execution" until the target student network is obtained.

[0058] This invention uses the error between the output results of the teacher network and the initial student network to assess whether the output result of the initial student network is close to the output result of the teacher network. The model parameters of the initial student network are updated synchronously. Under the condition that the output result of the initial student network is close to the output result of the teacher network, the neurons in the initial student network that can be deleted are further determined based on the performance of neurons in the initial student network (for example, if the weight parameter of a certain neuron is lower than a preset threshold, it means that the neuron is not important). Then, the initial student network is pruned to obtain a target student network with smaller parameter size.

[0059] The technical solution provided by this invention integrates feature-based knowledge distillation, probabilistic knowledge distillation, and hierarchical pruning techniques to generate a more accurate and flexible target student network for in-vehicle audio separation models. The student network is pruned from the teacher network, adjusting the number of channels and intermediate hierarchical structure based on the teacher network. It learns the intermediate thinking methods of the teacher network without needing to fit and approximate intermediate features, breaking through the traditional knowledge distillation limitation that "the input and output channels of the teacher and student networks must be consistent." It allows for flexible adjustment based on the number of microphones (input channels) and sound zone requirements (output channels) in the actual vehicle. Only one high-precision teacher network is needed to support the training of student networks for multiple vehicle models, eliminating the need to pre-train a large number of teacher networks for different vehicle models, significantly reducing training costs and development cycles.

[0060] Secondly, the pruning process simultaneously considers the "output error of the teacher and student networks" and the "neuronal performance of the initial student network," integrating probabilistic knowledge distillation and pruning techniques. This ensures that the student network inherits the core audio region separation capability of the teacher network while accurately removing redundant neurons, making the model parameters more compact and adaptable to the limited computing power of in-vehicle systems, enabling real-time audio zoning. These techniques do not rely on specific hardware layouts and can be quickly adapted to vehicle models with different numbers of microphones and different audio region requirements, significantly improving the versatility and practicality of the technology and providing an efficient and flexible model training solution for in-vehicle voice interaction scenarios.

[0061] In some alternative implementations, the teacher network is created based on a network structure of an encoder, a voice region separation network, and a decoder, wherein the encoder includes a downsampling layer, the decoder includes an upsampling layer, and the features of the decoder before the upsampling layer are added to the features of the encoder after the downsampling layer.

[0062] Specifically, in this embodiment of the invention, the encoder module generally adopts a U-net network structure. The encoder performs downsampling on the data. After downsampling, the number of model parameters required in the subsequent region separation module is smaller, and downsampling also allows for better fusion of channel and different feature information. After the encoder outputs data, it is input to the region separation module. The region separation network module can use a mainstream time-frequency domain network model structure. To enhance the model's performance, an attention network module can also be added. The region separation network further processes channel information and information between different frequency bands, enabling the model to learn better latent representations. The last module is the decoder module, which mainly implements two functions: upsampling and residual linking. Residual linking refers to summing the output of the encoder after each downsampling to the output before upsampling.

[0063] The specific network architectures of the encoder, audio region separation network, and decoder have been explained in the foregoing embodiments and will not be repeated here. In this embodiment of the invention, the encoder includes a downsampling layer, and the decoder includes an upsampling layer. The features of the decoder before the upsampling layer are added to the features of the encoder after the downsampling layer. The downsampling layer is a functional layer in the encoder that reduces the temporal dimension of the features by increasing the convolution stride, which can reduce subsequent computation and expand the receptive field of the features. The upsampling layer is a functional layer in the decoder that increases the temporal dimension of the features by transposing convolutions, used to restore the signal to its original duration. In this embodiment of the invention, the features are added element-wise to the features of the corresponding downsampling layer of the encoder before the decoder upsampling.

[0064] The feature addition operation avoids gradient explosion or vanishing problems during training by using residual links. At the same time, it passes the original detailed features extracted by the encoder to the decoder to make up for the speech details lost by downsampling (such as the speaker's tone and intonation). This ensures that the final output partitioned audio meets the requirements of in-vehicle voice interaction (such as voice commands and in-vehicle calls) in terms of both duration and sound quality, and provides high-precision "teacher knowledge" support for the subsequent knowledge distillation of the student network.

[0065] In some optional implementations, step S205 above includes: Step a1: Locate the subnetworks that are repeatedly used in the initial student network, and remove the first target neurons that are repeatedly used in the subnetworks. Step a2: In this training, the first loss value is calculated based on the error between the output of the teacher network and the initial student network. Step a3: In this training, calculate the second loss value based on the error between the initial student network output and the corresponding labels of the training data; Step a4: Weightedly fuse the first loss value and the second loss value to obtain the first fused loss value; Step a5: Obtain the second fusion loss value from the previous training. Step a6: When the difference between the first fusion loss value and the second fusion loss value does not exceed a preset threshold, determine the second target neuron in the initial student network that does not affect the model result in the previous training, and delete the second target neuron. Step a7: Repeat the steps of calculating the first loss value based on the error between the output results of the teacher network and the initial student network, until the loss difference between the first fusion loss value and the second fusion loss value does not exceed the preset threshold, determine the second target neuron that will not affect the model results during training, and delete the second target neuron. Repeat the training multiple times to obtain the target student network.

[0066] Specifically, embodiments of the present invention provide a specific process for obtaining a target student network by pruning based on the output error between the teacher network and the initial student network and the neuron performance of the initial student network.

[0067] The initial student network inherits the architecture of the teacher network, which is "encoder-segmentation network-decoder". The segmentation network (such as DPRNN network, LSTM network) contains sub-networks that are used repeatedly. The sub-networks that are used repeatedly refer to the modular network structures that are repeatedly called in the initial student network. In the in-vehicle audio partitioning model, these are mainly the recurrent blocks in the segmentation network (such as the recurrent units of DPRNN). Such sub-networks are prone to generating redundant neurons due to repeated computation. The first target neuron refers to the redundant neuron in the sub-network that is used repeatedly and has a very low contribution to the model output. It is usually characterized by a persistently low activation value or a weight value close to zero.

[0068] Recurrent units in such subnetworks (such as the time-path recurrent layer and channel-path recurrent layer of DPRNN) are prone to generating redundant neurons with overlapping functions due to repeated computation when processing multi-channel audio data. These should be prioritized for reduction to decrease computational load. For example, by using a network structure analysis tool to traverse all modules of the initial student network, subnetworks that are called ≥3 times are identified. Finally, the first target neuron in the repeatedly used subnetworks is reduced, specifically by deleting the corresponding weight parameters (such as the weights of the input gate, forget gate, and output gate in a Bi-LSTM unit) and bias parameters. Simultaneously, the output dimension of the subnetwork is adjusted to ensure that the dimension of the subnetwork's output features matches the input dimension of subsequent modules.

[0069] The first loss value is calculated based on the "teacher network output" to quantify the difference between the initial student network and the teacher network in terms of sound region separation results, ensuring that the student network inherits the high-precision partitioning capability of the teacher network.

[0070] For example, audio training data is selected and input into both the trained teacher network and the initial student network after a step of reducing recurrent units. The data input to the teacher network is the original multi-channel data adapted to the teacher network's input channels, while the data input to the initial student network is the filtered data adapted to its input channels. Then, the outputs of both networks are obtained. The teacher network outputs a multi-region time-domain audio signal (e.g., 4 regions), while the initial student network outputs a time-domain audio signal corresponding to the target region (e.g., 2 regions). If the number of regions output by the two networks is inconsistent, the output regions of the teacher network need to be aggregated according to the region division rules of the initial student network to ensure consistent output dimensions. Finally, the mean squared error loss function is used to calculate the first loss value.

[0071] Meanwhile, in this training, a second loss value is calculated based on the error between the initial student network output and the corresponding labels of the training data. This quantifies the difference between the initial student network and the real sound region separation results, ensuring that the student network meets the actual task requirements of the vehicle-mounted scenario.

[0072] For example, first obtain the labeled labels corresponding to the audio training data input to the initial student network. These labels are manually labeled target audio region time-domain audio signals (e.g., the training data is a mixed audio of "driver's voice + passenger interference", and the label is a pure driver's voice audio signal). The number of audio regions and duration of the labels are consistent with the output of the initial student network. Then, the second loss value is calculated using the same loss function as the previous steps.

[0073] Subsequently, in this embodiment of the invention, the first loss value and the second loss value are weighted and fused to obtain the first fused loss value. The purpose of weighted fusion is to balance the two objectives of "inheriting the teacher's network knowledge" and "adapting to the needs of real tasks," avoiding network imbalance caused by a single loss (such as focusing only on the teacher's network output while deviating from the real annotation, or focusing only on the real annotation while losing the high-precision features of the teacher's network). This value comprehensively reflects the knowledge learning effect and task adaptation effect of the student network. The network parameters will be adjusted based on this value to determine whether to continue pruning.

[0074] The second fusion loss value is the fusion loss value calculated using the same process in the previous training round (i.e., the training before this one). It is used to compare with the first fusion loss value in this round to determine whether the network has converged. If the difference between the two fusion loss values ​​is small, it indicates that the network parameters have stabilized and redundant neurons can be further removed. If the difference is large, it indicates that the network is still in the learning process and the parameters need to be adjusted before determining whether to prune.

[0075] For example, when the difference between the first fusion loss value and the second fusion loss value does not exceed a preset threshold, a second target neuron in the initial student network that does not affect the model results during the previous training is identified and removed. Secondary pruning is only performed when the network converges to avoid accidentally removing effective neurons when network parameters are unstable, ensuring the safety and accuracy stability of the pruning process.

[0076] The second target neurons fall into two categories: neurons with weights below a preset weight threshold and neurons that were adjusted during the previous training iteration. The preset weight threshold can be set to 1 / 10 of the mean absolute value of all weights in the initial student network (e.g., if the mean absolute value of all weights is 0.05, the preset weight threshold is 0.005; this is just an example and not a limitation). "Neurons adjusted during the previous training iteration" refer to neurons whose parameters were updated significantly in the previous training round, such as neurons where the absolute difference between the current and previous parameter values ​​is greater than 0.01 (this is just an example and not a limitation). The role of these neurons is not yet stable, and since the network has converged in this training, they have no crucial impact on the final result and can be marked as second target neurons. Finally, the two types of marked second target neurons are reduced in the same way as the previous steps (removing weights and bias parameters, and adjusting the output dimension).

[0077] Finally, repeat steps a2 to a6. After each training round, check if the loss difference has converged. If it has converged, perform secondary pruning; otherwise, continue training until the iteration termination condition is met. For example, when the number of parameters in the initial student network is reduced by more than 50% compared to the original state, and the sound region separation accuracy remains above 89% on the validation set, stop the iteration. The network at this point is the target student network.

[0078] Based on the aforementioned technical methods, prioritizing the removal of the first target neuron in a repeatedly used subnetwork can quickly eliminate redundant parameters from repetitive calculations, avoiding damage to the network's core feature extraction capabilities and laying a lightweight foundation for subsequent iterative pruning. Through dual-loss weighted fusion, knowledge distillation and real-world task requirements are balanced, avoiding network imbalances caused by single losses and ensuring that the student network inherits the high-precision partitioning features of the teacher network while also meeting the real-world annotation requirements of in-vehicle scenarios. An iterative convergence judgment mechanism ensures that secondary pruning is performed only when network parameters are stable, preventing the accidental removal of effective neurons. Simultaneously, through multiple iterations to gradually simplify parameters, the final target student network not only has significantly reduced parameters, adapting to the limited computing power of in-vehicle systems, but also meets the accuracy requirements of voice region separation in in-vehicle voice command and call interaction scenarios, effectively solving the problems of "sudden drop in accuracy" and "difficulty in adapting to computing power" inherent in traditional pruning.

[0079] In some alternative implementations, prior to step a7 above, the following step is also included: Step b1: Based on the first fusion loss value, update the remaining model parameters of the initial student network after this neuron deletion.

[0080] Specifically, after each neuron reduction, this embodiment of the invention also updates the remaining model parameters of the initial student network based on the first fusion loss value to compensate for the accuracy loss caused by neuron reduction and ensure that the network maintains stable learning ability during the lightweighting process. For example, a stochastic gradient descent optimizer is used, with the first fusion loss value as the optimization objective, to update the remaining model parameters (such as convolutional kernel weights, recurrent layer biases, etc.) of the initial student network after reduction in steps a1 or a6. After the update is completed, the network output accuracy is verified. If the accuracy decreases by more than 1%, the learning rate is halved and updated again until the accuracy decrease is ≤1%, ensuring that the remaining parameters can effectively compensate for the loss of feature extraction ability caused by neuron reduction and provide a stable parameter basis for the iterative training in step a7.

[0081] In some optional implementations, the training data includes first training data and second training data, wherein the microphone position of the first training data is the same as the microphone position of the data used when training the teacher network, and the microphone position of the second training data is different from the microphone position of the data used when training the teacher network.

[0082] Specifically, the training data provided in this embodiment of the invention includes first training data and second training data. The core difference between the two types of data lies in whether the microphone positions are consistent with those used in the teacher network training data. By using both types of data in combination, the adaptability of the target student network to different microphone layouts is improved. The microphone positions of the first training data are completely consistent with those of the data used when training the teacher network. Taking an in-vehicle scenario as an example, if the teacher network training uses microphone data from four fixed positions: the driver's side A-pillar, the passenger side A-pillar, and the left and right rear doors, then the first training data is also collected from these four positions, ensuring that the spatial distribution characteristics of the data match those of the teacher network training data, providing a consistent feature benchmark for knowledge distillation. The microphone positions of the second training data differ from those of the teacher network training data. For example, in the above scenario, the second training data can be collected from non-benchmark positions such as the center of the center console and the back of the rear seats, or the number of microphones can be reduced or increased (e.g., only two microphones are retained in the driver's seat and the center of the rear seats) to simulate the diversity of microphone layouts in actual vehicle installations, enhancing the robustness of the student network to positional deviations. In practical use, the first training data and the second training data can be mixed in a certain proportion to ensure that the student network can stably inherit the knowledge of the teacher network and to enable it to generalize to different microphone layouts, thus meeting the needs of multiple vehicle models.

[0083] This embodiment also provides an audio partitioning method, such as Figure 3 As shown, the process includes the following steps: Step S301: Receive audio data collected by several microphones; Step S302: Input the audio data into the trained target student network. The number of input channels of the target student network is the same as the number of microphones. Step S303: Obtain the partitioned audio based on the output of the target student network.

[0084] Specifically, in the vehicle environment, the microphone array is installed in different locations inside the vehicle according to a preset layout (e.g., one on the A-pillar of the driver's seat, one on the A-pillar of the passenger seat, one on the left rear door, and one on the right rear door, for a total of four microphones). Each microphone synchronously collects mixed audio signals inside the vehicle (including voices from multiple speakers, external noise, and noise from in-vehicle equipment), and transmits them to the vehicle's computer processor via the in-vehicle bus to complete the audio data reception.

[0085] The target student network is a lightweight model trained using the aforementioned method, with the number of input channels matching the actual number of microphones installed (e.g., 4 microphones corresponding to 4 input channels). The target student network outputs multi-channel partitioned audio data with the same duration as the input, with each output channel corresponding to a target audio region. By directly processing multi-channel audio through a target student network that matches the number of microphones, no additional channel adaptation or conversion is required, reducing data processing latency. The partitioning results inherit the high-precision characteristics of the teacher network while also possessing lightweight features, enabling real-time processing even with limited computing power in the vehicle, thus meeting the dual requirements of response speed and partitioning accuracy in in-vehicle scenarios.

[0086] This embodiment also provides a training device for an audio partitioning model, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0087] This embodiment provides a training device for an audio partitioning model, such as... Figure 4 As shown, it includes: The teacher network acquisition module 401 is used to acquire the trained teacher network. The teacher network is used to partition the audio based on the audio data input from multiple microphones and output multiple partitioned audio. The channel determination module 402 is used to determine the target number of input channels and the target number of output channels required by the student network, wherein the target number of output channels of the student network is not greater than the number of output channels of the teacher network. The student network acquisition module 403 is used to adjust the intermediate structure of the teacher network according to the target number of input channels and the target number of output channels to obtain the initial student network; Training module 404 is used to acquire audio training data and input the audio training data into the initial student network and teacher network; The pruning module 405 is used to prune the initial student network based on the error between the output results of the teacher network and the initial student network, and based on the neuron performance in the initial student network, to obtain the target student network.

[0088] The audio partitioning model training apparatus provided in this embodiment of the invention can execute the audio partitioning model training method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as in the corresponding embodiments described above, and will not be repeated here.

[0089] This embodiment also provides an audio partitioning device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0090] This embodiment provides an audio partitioning device, such as... Figure 5 As shown, it includes: The data acquisition module 501 is used to receive audio data collected by several microphones; The detection module 502 is used to input audio data into the target student network obtained by the above model training method. The number of input channels of the target student network is the same as the number of microphones. The output module 503 is used to obtain the partitioned audio based on the output of the target student network.

[0091] The audio partitioning apparatus provided in this embodiment of the invention can execute the audio partitioning method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as in the corresponding embodiments described above, and will not be repeated here.

[0092] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0093] The following is a detailed reference. Figure 6This diagram illustrates a suitable structural design for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 601, which can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) 602 or a program loaded from memory 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of the electronic device. The processor 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0094] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0095] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a memory 608, or installed from a ROM 602. When the computer program is executed by the processor 601, it performs the functions defined in the methods of the embodiments of the present invention.

[0096] Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0097] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0098] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0099] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A training method for an audio partitioning model, characterized in that, The method includes: Obtain the trained teacher network, which is used to partition the audio based on audio data input from multiple microphones and output multiple partitioned audio; Determine the target number of input channels and the target number of output channels for the student network, wherein the target number of output channels for the student network is not greater than the number of output channels for the teacher network; The intermediate structure of the teacher network is adjusted according to the target number of input channels and the target number of output channels to obtain the initial student network; Acquire audio training data and input the audio training data into the initial student network and the teacher network; Based on the error between the output results of the teacher network and the initial student network, and based on the neuron performance in the initial student network, the initial student network is pruned to obtain the target student network.

2. The method according to claim 1, characterized in that, The teacher network is created based on a network structure of an encoder, a voice region separation network, and a decoder. The encoder includes a downsampling layer, and the decoder includes an upsampling layer. The features of the decoder before the upsampling layer are added to the features of the encoder after the downsampling layer.

3. The method according to claim 1, characterized in that, The step of pruning the initial student network based on the error between the output results of the teacher network and the initial student network, and based on the neuron performance in the initial student network, to obtain the target student network, includes: Identify the subnetworks that are repeatedly used in the initial student network, and remove the first target neurons that are repeatedly used in the subnetworks; In this training, a first loss value is calculated based on the error between the outputs of the teacher network and the initial student network. In this training, a second loss value is calculated based on the error between the initial student network output and the corresponding labels of the training data; The first loss value and the second loss value are weighted and fused to obtain the first fused loss value; Obtain the second fusion loss value from the previous training iteration; When the difference between the first fusion loss value and the second fusion loss value does not exceed a preset threshold, the second target neuron in the initial student network that does not affect the model result in the previous training is determined, and the second target neuron is deleted. The process of calculating a first loss value based on the error between the outputs of the teacher network and the initial student network is repeated until the difference between the first fusion loss value and the second fusion loss value does not exceed a preset threshold. Then, a second target neuron that does not affect the model results during training is determined and the second target neuron is removed. This process is repeated multiple times to obtain the target student network.

4. The method according to claim 3, characterized in that, The second target neuron includes neurons with weight values ​​lower than a preset weight threshold, as well as neurons that were adjusted during the last training.

5. The method according to claim 3, characterized in that, Before repeating the steps of calculating a first loss value based on the error between the output results of the teacher network and the initial student network, and determining a second target neuron that does not affect the model results during training and deleting the second target neuron when the loss difference between the first fusion loss value and the second fusion loss value does not exceed a preset threshold, and performing multiple training runs to obtain the target student network, the method further includes: Based on the first fusion loss value, the remaining model parameters of the initial student network are updated after this neuron deletion.

6. The method according to claim 1, characterized in that, The training data includes first training data and second training data. The microphone position of the first training data is the same as the microphone position of the data used when training the teacher network, while the microphone position of the second training data is different from the microphone position of the data used when training the teacher network.

7. An audio partitioning method, characterized in that, The method includes: Receives audio data collected by several microphones; The audio data is input into the target student network trained by any one of claims 1-6, wherein the number of input channels of the target student network is the same as the number of microphones; The partitioned audio is obtained based on the output of the target student network.

8. A training device for an audio partitioning model, characterized in that, The device includes: The teacher network acquisition module is used to acquire the trained teacher network, which is used to partition the audio based on the audio data input from multiple microphones and output multiple partitioned audio. The channel determination module is used to determine the target number of input channels and the target number of output channels required by the student network, wherein the target number of output channels of the student network is not greater than the number of output channels of the teacher network; The student network acquisition module is used to adjust the intermediate structure of the teacher network according to the target number of input channels and the target number of output channels to obtain the initial student network; The training module is used to acquire audio training data and input the audio training data into the initial student network and the teacher network; The pruning module is used to prune the initial student network based on the error between the output results of the teacher network and the initial student network, and based on the neuron performance in the initial student network, to obtain the target student network.

9. An audio partitioning device, characterized in that, The device includes: The data acquisition module is used to receive audio data collected by several microphones; The detection module is used to input the audio data into a target student network trained by any one of claims 1-6, wherein the number of input channels of the target student network is the same as the number of microphones; The result output module is used to obtain the partitioned audio based on the output results of the target student network.

10. An electronic device, characterized in that, include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the method of any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method of any one of claims 1 to 7.

Images