Self-adaptive pickup method and system based on multi-sound-source localization

By performing feature extraction and sound source positioning model processing on multi-sound source sound data, the optimal sound pickup device parameters are predicted, and the problem of inaccurate sound positioning in complex environments is solved, and the accuracy and adaptation of multi-sound source sound pickup is achieved. sex.

CN119094937AActive Publication Date: 2024-12-06GUANGZHOU BAOLUN ELECTRONICS CO LTD

Patent Information

Application Number
CN202411277624.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-12
Publication Date
2024-12-06
Estimated Expiration
2044-09-12

AI Technical Summary

Technical Problem

The existing multi-microphone sound pickup technology is not accurate enough in complex environments, making it difficult to distinguish between multiple sound sources, resulting in sound confusion and signal overlap problems, and lacks the ability to learn and adapt to complex sound characteristics.

Method used

An adaptive sound pickup method based on multi-sound source positioning is proposed. By extracting the sound data of multiple sound sources, a comprehensive feature vector set is obtained, and the sound source positioning model is input to distinguish the target sound source, and the most predicted based on the target sound source characteristics is predicted. Optimize the parameters of the sound pickup device and adjust the parameters of the sound pickup device to achieve more accurate sound pickup.

Benefits of technology

It realizes more accurate and quick sound pickup when multiple sound sources are sounded simultaneously, reduces sound confusion and improves the accuracy and adaptability of sound source positioning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119094937A_ABST
    Figure CN119094937A_ABST
Patent Text Reader

Abstract

The invention discloses a self-adaptive pickup method and system based on multi-sound-source localization, and the method comprises the steps: carrying out the feature extraction of collected multi-sound-source sound production data, and obtaining a comprehensive feature vector set; inputting the comprehensive feature vector set into a preset sound source localization model for processing to obtain target sound source features; according to the target sound source features, optimal pickup equipment parameters in the current environment are obtained through a preset pickup strategy prediction model; and according to the optimal pickup equipment parameters, setting adjustable parameters of the pickup equipment to carry out pickup. According to the invention, the target sound source is accurately positioned when the multiple sound sources produce sound at the same time, and the related parameters of the pickup equipment are set according to the information of the target sound source, so that more accurate and rapid pickup is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the technical field of microphone sound pickup, and in particular relates to an adaptive sound pickup method and system based on multi-sound source localization. Background Art

[0002] In today's multi-microphone sound pickup technology, the accuracy of sound source localization is often limited by many factors. For environmental interference, the existing processing methods may only compensate based on some basic acoustic models. In complex and changeable actual environments, this compensation effect is limited, resulting in inaccurate sound source localization, requiring the use of more complex sound source separation and tracking technologies.

[0003] In the case of multiple sound sources, existing separation and tracking technologies mainly rely on energy or spectrum features for simple separation, which makes it difficult to distinguish the main sound source, resulting in sound confusion. In actual complex sound scenes, fixed screening methods based on simple thresholds or rules are usually used. They lack the ability to learn and adapt to complex sound features and cannot adjust in real time to follow the dynamic changes of sound. As a result, when the sound intensity and frequency change rapidly, the screening results are inaccurate and effective audio cannot be captured and processed in time. Summary of the invention

[0004] The present application proposes an adaptive sound pickup method and system based on multi-sound source positioning, which accurately locates the target sound source when multiple sound sources emit sound at the same time, thereby achieving more accurate and rapid sound pickup.

[0005] A first aspect of the present application provides an adaptive sound pickup method based on multi-sound source localization, the method comprising:

[0006] Extracting features from the collected multi-source sound data to obtain a comprehensive feature vector set;

[0007] Inputting the comprehensive feature vector set into a preset sound source localization model for processing to obtain the target sound source features;

[0008] According to the characteristics of the target sound source, the optimal sound pickup device parameters in the current environment are obtained through the preset sound pickup strategy prediction model;

[0009] According to the optimal sound pickup device parameters, adjustable parameters of the sound pickup device are set to pick up sound.

[0010] The above scheme first extracts the sound source features from the sound data of multiple sound sources to obtain a comprehensive feature vector set, and then obtains the target sound source features through the sound source localization model to achieve the distinction and location of the target sound source, solving the problem of sound confusion. Then, based on the target sound source features, the optimal sound pickup device parameters in the current environment are obtained, so that the sound pickup device can capture the sound signal emitted by the target sound source more accurately and quickly.

[0011] In a possible implementation method of the first aspect, feature extraction is performed on the collected multi-source sound data to obtain a comprehensive feature vector set, specifically:

[0012] Extracting sound signals, environmental information and sound source location information from the multi-sound source sound data;

[0013] Extracting features from the sound signal to obtain sound features;

[0014] Extracting features of the sound source position information to obtain sound source spatial features and sound source temporal features;

[0015] Based on the same time dimension, the sound features, sound source spatial features, sound source temporal features and environmental information are aligned and spliced ​​in turn to obtain a comprehensive feature vector set.

[0016] The above scheme extracts sound signals, environmental information and sound source location information respectively, and obtains sound features and the characteristics of the sound source in time and space. These features are then aligned and spliced ​​in time to provide data support for subsequent sound source positioning.

[0017] In a possible implementation method of the first aspect, feature extraction is performed on the sound signal to obtain the sound feature, specifically:

[0018] According to a preset time length, the sound signal is divided into a plurality of frame signals;

[0019] Convert the frame signal into a Mel spectrum, and extract a first threshold number of first coefficients through the Mel spectrum to obtain energy distribution of the sound signal within a preset frequency range;

[0020] Calculating the short-time energy of the sound signal according to the frame signal to obtain the intensity change of the sound signal within the time length;

[0021] Calculating the zero-crossing rate of each frame signal to obtain the frequency change of the sound signal within the time length;

[0022] The sound feature is obtained according to the first coefficient, the short-time energy and the zero-crossing rate.

[0023] The above schemes use three different algorithms to quantify the changes in the energy, intensity and frequency of sound signals over time, so as to obtain the characteristics of the sound signals. Moreover, based on the sound characteristics, it can provide data support for the subsequent sound source positioning.

[0024] In a possible implementation method of the first aspect, the short-term energy is specifically:

[0025] The calculation formula of the short-time energy is:

[0026]

[0027] Wherein, x() is the sound signal, N is the frame length of the frame signal, n is the sequence number of the frame signal, m is the index variable of the sample point in the current frame signal, and E(n) is the short-time energy of the nth sample point in the frame signal.

[0028] In a possible implementation method of the first aspect, feature extraction is performed on the sound source position information to obtain a sound source spatial feature and a sound source temporal feature, specifically:

[0029] Extracting the sound source spatial features from the sound source position information according to the time difference between each sound source reaching the sound pickup device;

[0030] According to the start time and duration of each sound source, a sound source time feature is extracted from the sound source position information.

[0031] The sound source spatial characteristics and sound source temporal characteristics obtained in the above scheme can accurately distinguish each sound source when multiple sound sources occur simultaneously and have the same sound characteristics, providing data support for sound source positioning.

[0032] In a possible implementation method of the first aspect, the comprehensive feature vector set is input into a preset sound source localization model for processing to obtain target sound source features, specifically:

[0033] Based on the comprehensive feature vector set, the sound source localization information, as well as the sound source distortion degree and the sound source confusion degree of the target sound source are obtained through the sound source localization model;

[0034] According to the sound source localization information, a target signal corresponding to a target sound source is extracted from the multi-sound source sound data, and the target signal is optimized according to the sound source distortion degree and the sound source confusion degree of the target sound source to obtain the target sound source feature.

[0035] The above scheme first obtains the sound source localization information, as well as the sound source distortion and the sound source confusion that affect the sound source localization through the sound source localization model. Then, the signal of the target sound source is extracted based on the sound source localization information, and then the signal is further optimized using the sound source distortion and the sound source confusion of the target sound source, so as to reduce the target sound source characteristics of the error and effectively and accurately locate the target sound source.

[0036] In a possible implementation method of the first aspect, the sound source localization model is specifically:

[0037] Among them, the loss function of the sound source localization model is expressed as:

[0038] L total =

[0039] Wlocation L location +W distortion L distortion +W classification L classification ;

[0040] Where, L total is the loss function of the sound source localization model, W location is the weight of the sound source localization loss, L location is the sound source localization loss, W distortion is the weight of the sound source distortion loss, L distortion is the distortion loss of the sound source, W classification is the weight of the loss of sound source confusion, L classification It is the loss of sound source confusion.

[0041] In a possible implementation method of the first aspect, according to the target sound source characteristics, the optimal sound pickup device parameters in the current environment are obtained through a preset sound pickup strategy prediction model, specifically:

[0042] According to the characteristics of the target sound source, the sound pickup strategy prediction model selects the optimal sound pickup strategy that meets the current environment from a preset sound pickup strategy library;

[0043] The optimal sound pickup device parameters are determined according to the hardware parameters of the sound pickup device and the optimal sound pickup strategy.

[0044] The above scheme selects the optimal sound pickup strategy that can most accurately pick up the target sound source from the existing sound pickup strategy library based on the characteristics of the target sound source. Moreover, the optimal sound pickup strategy can also adapt to the current environment and minimize the impact of the environment on the sound pickup.

[0045] In a possible implementation method of the first aspect, according to the optimal sound pickup device parameters, setting adjustable parameters of the sound pickup device to perform sound pickup is specifically as follows:

[0046] According to the optimal sound pickup device parameters, the sound pickup gain parameters, frequency response parameters, sampling rate and noise threshold value of the sound pickup device are set.

[0047] The second aspect of the present application provides an adaptive sound pickup system based on multi-sound source localization, the system comprising: a data preprocessing module, a sound source localization module, an optimal sound pickup parameter acquisition module and a sound pickup device setting module;

[0048] The data preprocessing module is used to extract features from the collected multi-source sound data to obtain a comprehensive feature vector set;

[0049] The sound source localization module is used to input the comprehensive feature vector set into a preset sound source localization model for processing to obtain sound source localization information and sound source error information;

[0050] The optimal sound pickup parameter acquisition module is used to obtain the optimal sound pickup device parameters in the current environment through a preset sound pickup strategy prediction model according to the sound source positioning information and the sound source error information;

[0051] The sound pickup device setting module is used to set the sound pickup device to pick up sound according to the optimal sound pickup device parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the implementation manner will be briefly introduced below. Obviously, the drawings described below are only some implementation manners of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 This is a schematic diagram of a specific process of an adaptive sound pickup method based on multi-sound source localization provided by a certain embodiment of the present application;

[0054] Figure 2 A structural diagram of an adaptive sound pickup system based on multi-sound source localization is provided in a certain embodiment of the present application. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0056] It should be understood that the step numbers used in this article are only for the convenience of description and are not intended to limit the order in which the steps are executed.

[0057] First embodiment

[0058] Existing multi-sound source localization methods rely only on simple geometric acoustic models and cannot accurately simulate complex sound propagation in actual environments. In complex acoustic environments, sound source localization is prone to large deviations. Moreover, when multiple sound sources emit sound at the same time and the sound characteristics are similar, it is difficult to accurately distinguish and track each sound source, resulting in confusion in sound source localization and inability to accurately locate the sound source, leading to problems such as sound confusion, signal overlap, and sound distortion. Therefore, how to adjust the sound pickup strategy in accordance with the dynamic changes of the sound, cope with the diversity and complexity of the sound scene, and more accurately locate multiple sound sources is an important research direction of the embodiments of this application.

[0059] like Figure 1 As shown, Figure 1 A specific flow chart of an adaptive sound pickup method based on multi-sound source localization is provided for a certain embodiment of the present application. The adaptive sound pickup method based on multi-sound source localization of this embodiment includes steps S1 to S4, which are described in detail as follows:

[0060] Step S1, extracting features from the collected multi-sound source sound data to obtain a comprehensive feature vector set.

[0061] In an embodiment of the present application, a large amount of multi-source sound data in which multiple sound sources emit sound simultaneously is first collected through a microphone array, and the multi-source sound data is annotated with detailed environmental parameters and sound source position information corresponding to the environment.

[0062] Among them, the labeled multi-sound source sound data includes data such as ambient temperature, ambient humidity, room size, related influencing materials (such as wall material, floor material and ceiling material, etc.), time difference between the sound source reaching different microphones, intensity difference between the sound source reaching different microphones and quantitative value of distortion degree.

[0063] The annotated multi-source sound data is preprocessed to check and remove outliers and missing values ​​in the data, and then feature extraction is performed on the preprocessed multi-source sound data to obtain sound signals, environmental information, and sound source location information, respectively. The sound signal is the original sound signal collected by the microphone array.

[0064] Next, the sound signal is analyzed to obtain the changes in the energy, intensity and frequency of the sound signal over time, so as to obtain the characteristics of the sound signal. There are three main methods used to extract the characteristics of the sound signal: Mel spectrum method, short-time energy calculation and zero-crossing rate calculation. Before feature extraction, the sound signal needs to be divided into several frame signals according to the preset time length.

[0065] The first feature extraction method for sound signals is the Mel spectrum method, which is mainly used to determine the energy distribution of sound signals within a preset frequency range and can effectively describe the key information of the sound. The specific process is as follows: First, each frame signal is optimized using an applied window function to reduce the discontinuity at both ends of each frame signal. Then, the windowed frame signal is subjected to a fast Fourier transform to convert the time domain signal into a frequency domain signal to obtain a spectrum. Then, the linear frequency scale in the spectrum is converted into a Mel frequency scale, and the spectrum is filtered using a set of triangular filters uniformly distributed on the Mel frequency scale to obtain a Mel spectrum. Next, the Mel spectrum is logarithmized to simulate the nonlinear perception of sound intensity by human hearing to obtain a logarithmic Mel spectrum. Finally, the logarithmic Mel spectrum is subjected to a discrete cosine transform to obtain a number of Mel frequency cepstral coefficients, and the first threshold coefficients are extracted from the Mel frequency cepstral coefficients as sound features.

[0066] It is worth mentioning that, because in general, the first 12 to 16 coefficients in the Mel frequency cepstral coefficients can contain the main features and most of the energy of the sound signal, can effectively describe the key information of the sound, and retaining fewer coefficients can also reduce the data dimension and calculation complexity without losing too much information. Therefore, in the embodiment of the present application, at most the first 16 Mel frequency cepstral coefficients are taken as sound features. In addition, these Mel frequency cepstral coefficients are sorted from large to small according to the energy size.

[0067] Optionally, in simple scenarios, such as in small conference rooms, the first 12 Mel-frequency cepstral coefficients can be used. 12 Mel-frequency cepstral coefficients can better capture key features and accurately identify them, reducing calculation time. In complex scenarios, such as abnormal multilingual academic conferences, speakers may speak with various accents and speeds. In order to more accurately capture these complex speech features and improve recognition accuracy, 16 Mel-frequency cepstral coefficients are selected.

[0068] Exemplarily, the application window function used in the embodiment of the present application is a Hamming window.

[0069] The second feature extraction method for sound signals is to calculate the short-time energy of the sound signal to obtain the intensity change of the sound signal within a certain time length. The specific calculation formula is as follows:

[0070]

[0071] Wherein, x() is the sound signal, N is the frame length of the frame signal, n is the sequence number of the frame signal, m is the index variable of the sample point in the current frame signal, and E(n) is the short-time energy of the nth sample point in the frame signal.

[0072] Among them, short-time energy is an independent sound characteristic parameter, which is mainly used in monitoring the start and end of sound signals, strength evaluation, etc. By traversing from 0 to N-1, a frame signal is traversed, so that the formula can sum the squares of each sample point in the current frame to calculate the short-time energy of this frame.

[0073] The third feature extraction method for sound signals is to calculate the zero-crossing rate of each frame signal to obtain the frequency change of the sound signal within a certain time length. The specific process is: for each frame signal, the number of times the frame signal crosses the zero level from positive to negative or from negative to positive is counted, that is, for a frame signal, if the signs of two adjacent sampling points are different, it is considered that a zero crossing has occurred. The zero-crossing rate can be expressed as the ratio of the number of zero crossings to the frame length. By calculating the zero-crossing rate, information about the frequency change of the sound signal and the distinction between unvoiced and voiced sounds can be obtained.

[0074] Based on the above three sound feature extraction methods, the sound features corresponding to the sound signal can be obtained.

[0075] Then, feature extraction is performed on the sound source position information to obtain the sound source spatial feature and the sound source temporal feature, specifically: the sound source spatial feature is extracted from the sound source position information according to the time difference of each sound source reaching the microphone array. The sound source temporal feature is extracted from the sound source position information according to the start time and duration of each sound source.

[0076] After obtaining the sound features, sound source spatial features, sound source temporal features, and environmental information, these data are first aligned in time, and then aligned in other data dimensions. After obtaining the aligned data, based on the same time dimension, these different types of feature data are spliced ​​into several unified vectors according to the alignment results to obtain a comprehensive feature vector set.

[0077] Exemplarily, the sound feature is a vector of length n, the sound source spatial feature is a vector of length m, the sound source temporal feature is a vector of length p, and the environmental information is a vector of length q. These data are sequentially concatenated into a comprehensive feature vector of length n+m+p+q.

[0078] Step S2: input the comprehensive feature vector set into a preset sound source localization model for processing to obtain target sound source features.

[0079] In the embodiment of the present application, the comprehensive feature vector set is first input into the trained sound source localization model to obtain the sound source localization information, as well as the sound source distortion degree and the sound source confusion degree of various sound sources. An additional fully connected layer is added after the TCN layer of the sound source localization model to realize the multiple data outputs of the model.

[0080] Among them, the loss function of the sound source localization model is expressed as:

[0081] L total =W location L location +W distortion L distortion +W classification L classification ;

[0082] Where, L total is the loss function of the sound source localization model, W location is the weight of the sound source localization loss, L location is the sound source localization loss, W distortion is the weight of the sound source distortion loss, L distortion is the distortion loss of the sound source, W classification is the weight of the loss of sound source confusion, L classification It is the loss of sound source confusion.

[0083]

[0084] Where N is the total number of samples, x pred,i ,y pred,i 、z pred,i are the position coordinates of the sound source in the x, y and z directions predicted by the model in the ith sample, respectively. true,i ,y true,i 、z true,i are the actual position coordinates of the sound source in the i-th sample in the x, y and z directions respectively.

[0085]

[0086] In the formula, distortion pred,i is the degree of distortion of the sound source predicted by the model in the i-th sample, distortion true,i is the true value of the sound source distortion in the i-th sample.

[0087]

[0088] In the formula, C is the total number of categories classified by the model, y true,i,c is the true probability that the i-th sample belongs to category c. If the sample belongs to category c, the value is 1, otherwise it is 0; y pred,i,c The model predicts the probability c that the i-th sample belongs to the category.

[0089] Finally, according to the output results of the model, the sound source localization information, the sound source distortion degree and the sound source confusion degree of the target sound source are obtained, and then according to the sound source localization information, the target signal corresponding to the target sound source is extracted from the multi-sound source sound data, and the target signal is optimized according to the sound source distortion degree and the sound source confusion degree of the target sound source to obtain the target sound source characteristics.

[0090] Optionally, an embodiment of the present application uses beamforming technology to extract sound source localization information of the target sound source from the sound source localization information output by the model, and then performs fast Fourier transform on the sound source localization information of the target sound source to obtain the corresponding target spectrum, and uses the target spectrum to normalize the sound source distortion degree and the sound source confusion degree to obtain the sound source distortion degree and the sound source confusion degree of the target sound source.

[0091] In addition, the embodiment of the present application collects a large amount of historical multi-sound source sound data under different environmental conditions, and records the corresponding environmental parameters and the corresponding sound source location information for each data sample in detail, and converts the historical multi-sound source sound data into a historical feature vector set. The historical feature vector set is divided into a training set, a validation set, and a test set in a ratio of 8:1:1, the training set is input into the initial time domain convolution model and trained using the ReLu activation function, and the validation set and the test set are used to evaluate the model performance and optimize the model according to the training results to obtain a sound source localization model that can meet the prediction accuracy.

[0092] Step S3, according to the target sound source characteristics, the optimal sound pickup device parameters in the current environment are obtained through a preset sound pickup strategy prediction model.

[0093] In an embodiment of the present application, the target sound source features are first preprocessed, and then the preprocessed target sound source features are input into a sound pickup strategy prediction model, and the optimal sound pickup strategy that meets the current environment is screened out according to a provided sound pickup strategy library.

[0094] The data in the sound pickup strategy library is extracted through relevant historical data or based on the professional knowledge and experience of relevant professionals, and is formed by making the optimal sound pickup strategy for different sound source positioning information, distortion degree quantification values ​​and sound source confusion degree related situations.

[0095] Optionally, the embodiment of the present application collects a large amount of historical sound source features, divides the historical sound source features into a training set, a test set, and a validation set in a ratio of 7:2:1, trains a random forest model, and obtains the sound pickup strategy prediction model.

[0096] Finally, based on the hardware parameters of the sound pickup device and the optimal sound pickup strategy, the optimal sound pickup device parameters are determined.

[0097] Step S4: according to the optimal sound pickup device parameters, setting adjustable parameters of the sound pickup device to pick up sound.

[0098] In an embodiment of the present application, according to the obtained optimal sound pickup device parameters, the sound pickup gain parameters, frequency response parameters, sampling rate, noise threshold value, compression ratio and equalizer parameters of the sound pickup device are adjusted to achieve adaptive sound pickup of the sound pickup device, accurately locate the target sound source when multiple sound sources sound at the same time, and achieve more accurate and faster sound pickup.

[0099] Implementing the embodiments of the present application has the following beneficial effects:

[0100] The embodiment of the present application obtains a comprehensive feature vector set by extracting sound source features from the sound data of multiple sound sources; wherein, sound features, spatial and temporal features of the sound source, and environmental information are extracted respectively, and the sound source features are determined based on these data, and each sound source is accurately distinguished when multiple sound sources occur simultaneously and the sound features are the same, providing data support for sound source positioning. Then, the target sound source features are obtained through the sound source positioning model, and the sound source positioning information obtained by predicting and correcting the degree of sound source distortion and the degree of sound source confusion is used to achieve the distinction and positioning of the target sound source, solving the problem of sound confusion. Then, based on the target sound source features, the optimal sound pickup device parameters in the current environment are obtained, so that the sound pickup device can more accurately and quickly capture the sound signal emitted by the target sound source.

[0101] Second embodiment

[0102] Further, in order to implement the adaptive sound pickup system based on multi-sound source localization corresponding to the above method embodiment to achieve the corresponding functions and technical effects, Figure 2 A structural diagram of an adaptive sound pickup system based on multi-sound source localization is provided. For ease of description, only the parts related to this embodiment are shown. The adaptive sound pickup system based on multi-sound source localization provided by the embodiment of the present application includes:

[0103] The data preprocessing module 201 is used to extract features from the collected multi-sound source sound data to obtain a comprehensive feature vector set.

[0104] In the embodiment of the present application, the sound signal, environmental information and sound source location information are first extracted from the sound data of multiple sound sources, and then the sound signal is feature extracted to obtain the sound feature; the sound source location information is feature extracted to obtain the sound source spatial feature and the sound source time feature. Finally, based on the same time dimension, the sound feature, the sound source spatial feature, the sound source time feature and the environmental information are sequentially aligned and spliced ​​in multiple dimensions to obtain a comprehensive feature vector set, which provides data support for subsequent sound source positioning.

[0105] The sound source localization module 202 is used to input the comprehensive feature vector set into a preset sound source localization model for processing to obtain sound source localization information and sound source error information.

[0106] In the embodiment of the present application, the comprehensive feature vector set is input into the sound source localization model for data prediction to obtain the sound source localization information, the sound source distortion degree and the sound source confusion degree. Then, the localization information of the target sound source is separated from the sound source localization information, and the sound source distortion degree and the sound source confusion degree of the target sound source are determined based on the localization information. Finally, the target signal corresponding to the target sound source is extracted from the multi-sound source sound data, and the target signal is optimized according to the sound source distortion degree and the sound source confusion degree of the target sound source to obtain the target sound source characteristics.

[0107] The optimal sound pickup parameter acquisition module 203 is used to obtain the optimal sound pickup device parameters in the current environment through a preset sound pickup strategy prediction model according to the sound source localization information and the sound source error information.

[0108] In an embodiment of the present application, according to the characteristics of the target sound source, the pickup strategy prediction model selects the optimal pickup strategy that meets the current environment from a preset pickup strategy library; and determines the optimal pickup device parameters according to the hardware parameters of the pickup device and the optimal pickup strategy.

[0109] Among them, the construction of the sound pickup strategy prediction model is achieved by collecting a large amount of historical sound source features, dividing the historical sound source features into a training set, a test set and a validation set in a ratio of 7:2:1, and training the random forest model.

[0110] The sound pickup device setting module 204 is used to set the sound pickup device to pick up sound according to the optimal sound pickup device parameters.

[0111] In an embodiment of the present application, according to the obtained optimal sound pickup device parameters, the sound pickup gain parameters, frequency response parameters, sampling rate, noise threshold value, compression ratio and equalizer parameters of the sound pickup device are adjusted to achieve adaptive sound pickup of the sound pickup device, accurately locate the target sound source when multiple sound sources sound at the same time, and achieve more accurate and faster sound pickup.

[0112] In some embodiments, the data preprocessing module 201 further includes:

[0113] A sound feature extraction unit is used to divide the sound signal into a plurality of frame signals according to a preset time length; convert the frame signal into a Mel spectrum, and extract a first threshold first coefficient through the Mel spectrum to obtain the energy distribution of the sound signal within a preset frequency range; calculate the short-time energy of the sound signal according to the frame signal to obtain the intensity change of the sound signal within the time length; calculate the zero-crossing rate of each frame signal to obtain the frequency change of the sound signal within the time length; and obtain the sound feature according to the first coefficient, the short-time energy and the zero-crossing rate.

[0114] Specifically, the sound signal is analyzed to obtain the changes in the energy, intensity and frequency of the sound signal over time, so as to obtain the characteristics of the sound signal. There are three main methods for extracting the characteristics of the sound signal: Mel spectrum method, short-time energy calculation and zero-crossing rate calculation. Before feature extraction, the sound signal needs to be divided into several frame signals according to the preset time length.

[0115] The first feature extraction method for sound signals is the Mel spectrum method, which is mainly used to determine the energy distribution of sound signals within a preset frequency range and can effectively describe the key information of the sound. The specific process is as follows: First, each frame signal is optimized using an applied window function to reduce the discontinuity at both ends of each frame signal. Then, the windowed frame signal is subjected to a fast Fourier transform to convert the time domain signal into a frequency domain signal to obtain a spectrum. Then, the linear frequency scale in the spectrum is converted into a Mel frequency scale, and the spectrum is filtered using a set of triangular filters uniformly distributed on the Mel frequency scale to obtain a Mel spectrum. Next, the Mel spectrum is logarithmized to simulate the nonlinear perception of sound intensity by human hearing to obtain a logarithmic Mel spectrum. Finally, the logarithmic Mel spectrum is subjected to a discrete cosine transform to obtain a number of Mel frequency cepstral coefficients, and the first threshold coefficients are extracted from the Mel frequency cepstral coefficients as sound features.

[0116] It is worth mentioning that, because in general, the first 12 to 16 coefficients in the Mel frequency cepstral coefficients can contain the main features and most of the energy of the sound signal, can effectively describe the key information of the sound, and retaining fewer coefficients can also reduce the data dimension and calculation complexity without losing too much information. Therefore, in the embodiment of the present application, at most the first 16 Mel frequency cepstral coefficients are taken as sound features. In addition, these Mel frequency cepstral coefficients are sorted from large to small according to the energy size.

[0117] Optionally, in simple scenarios, such as in small conference rooms, the first 12 Mel-frequency cepstral coefficients can be used. 12 Mel-frequency cepstral coefficients can better capture key features and accurately identify them, reducing calculation time. In complex scenarios, such as abnormal multilingual academic conferences, speakers may speak with various accents and speeds. In order to more accurately capture these complex speech features and improve recognition accuracy, 16 Mel-frequency cepstral coefficients are selected.

[0118] Exemplarily, the application window function used in the embodiment of the present application is a Hamming window.

[0119] The second feature extraction method for sound signals is to calculate the short-time energy of the sound signal to obtain the intensity change of the sound signal within a certain time length. The specific calculation formula is as follows:

[0120]

[0121] Wherein, x() is the sound signal, N is the frame length of the frame signal, n is the sequence number of the frame signal, m is the index variable of the sample point in the current frame signal, and E(n) is the short-time energy of the nth sample point in the frame signal.

[0122] Among them, short-time energy is an independent sound characteristic parameter, which is mainly used in monitoring the start and end of sound signals, strength evaluation, etc. By traversing from 0 to N-1, a frame signal is traversed, so that the formula can sum the squares of each sample point in the current frame to calculate the short-time energy of this frame.

[0123] The third feature extraction method for sound signals is to calculate the zero-crossing rate of each frame signal to obtain the frequency change of the sound signal within a certain time length. The specific process is: for each frame signal, the number of times the frame signal crosses the zero level from positive to negative or from negative to positive is counted, that is, for a frame signal, if the signs of two adjacent sampling points are different, it is considered that a zero crossing has occurred. The zero-crossing rate can be expressed as the ratio of the number of zero crossings to the frame length. By calculating the zero-crossing rate, information about the frequency change of the sound signal and the distinction between unvoiced and voiced sounds can be obtained.

[0124] Based on the above three sound feature extraction methods, the sound features corresponding to the sound signal can be obtained.

[0125] The sound source feature extraction unit is used to extract the sound source spatial feature from the sound source position information according to the time difference of each sound source reaching the sound pickup device; and to extract the sound source time feature from the sound source position information according to the start time and duration of each sound source.

[0126] According to the time difference of each sound source reaching the microphone array, the spatial feature of the sound source is extracted from the sound source position information. According to the start time and duration of each sound source, the time feature of the sound source is extracted from the sound source position information.

[0127] Specifically, after obtaining the sound features, sound source spatial features, sound source temporal features, and environmental information, these data are first aligned in time, and then aligned in other data dimensions. After obtaining the aligned data, based on the same time dimension, these different types of feature data are spliced ​​into several unified vectors according to the alignment results to obtain a comprehensive feature vector set.

[0128] Exemplarily, the sound feature is a vector of length n, the sound source spatial feature is a vector of length m, the sound source temporal feature is a vector of length p, and the environmental information is a vector of length q. These data are sequentially concatenated into a comprehensive feature vector of length n+m+p+q.

[0129] In some embodiments, the sound source localization module 202 further includes:

[0130] Beamforming technology can be used to extract the sound source localization information of the target sound source from the sound source localization information output by the model, and then the sound source localization information of the target sound source is fast Fourier transformed to obtain the corresponding target spectrum, and the target spectrum is used to normalize the sound source distortion degree and the sound source confusion degree to obtain the sound source distortion degree and the sound source confusion degree of the target sound source.

[0131] Implementing the embodiments of the present application has the following beneficial effects:

[0132] The embodiment of the present application first obtains relevant environmental data and sound source data in a multi-scenario environment, comprehensively considers environmental information such as temperature, humidity, space size, and material, and predicts the degree of distortion and confusion in the relevant environment through a sound source localization model to verify the predicted sound source localization, reduce prediction errors, and output accurate sound source localization information. Then, based on the sound pickup strategy prediction model and the preset sound pickup strategy library, the optimal sound pickup screening conditions are accurately predicted to achieve adaptive real-time adjustment, accurately locate the target sound source when multiple sound sources sound at the same time, and achieve more accurate and rapid sound pickup.

[0133] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of protection of the present application.

Claims

1. An adaptive sound pickup method based on multi-sound source localization, characterized in that: include: Extracting features from the collected multi-source sound data to obtain a comprehensive feature vector set; Inputting the comprehensive feature vector set into a preset sound source localization model for processing to obtain the target sound source features; According to the characteristics of the target sound source, the optimal sound pickup device parameters in the current environment are obtained through the preset sound pickup strategy prediction model; According to the optimal sound pickup device parameters, adjustable parameters of the sound pickup device are set to pick up sound.

2. The adaptive sound pickup method based on multi-sound source localization according to claim 1, characterized in that: The feature extraction of the collected multi-source sound data is performed to obtain a comprehensive feature vector set, which is specifically: Extracting sound signals, environmental information and sound source location information from the multi-sound source sound data; Extracting features from the sound signal to obtain sound features; Extracting features of the sound source position information to obtain sound source spatial features and sound source temporal features; Based on the same time dimension, the sound features, sound source spatial features, sound source temporal features and environmental information are aligned and spliced ​​in turn to obtain a comprehensive feature vector set.

3. The adaptive sound pickup method based on multi-sound source localization according to claim 2, characterized in that: The feature extraction of the sound signal to obtain the sound feature is specifically as follows: According to a preset time length, the sound signal is divided into a plurality of frame signals; Convert the frame signal into a Mel spectrum, and extract a first threshold number of first coefficients through the Mel spectrum to obtain energy distribution of the sound signal within a preset frequency range; Calculating the short-time energy of the sound signal according to the frame signal to obtain the intensity change of the sound signal within the time length; Calculating the zero-crossing rate of each frame signal to obtain the frequency change of the sound signal within the time length; The sound feature is obtained according to the first coefficient, the short-time energy and the zero-crossing rate.

4. The adaptive sound pickup method based on multi-sound source localization according to claim 3, characterized in that: The short-time energy is specifically: The calculation formula of the short-time energy is: Wherein, x() is the sound signal, N is the frame length of the frame signal, n is the sequence number of the frame signal, m is the index variable of the sample point in the current frame signal, and E(n) is the short-time energy of the nth sample point in the frame signal.

5. The adaptive sound pickup method based on multi-sound source localization according to claim 2, characterized in that: The feature extraction of the sound source position information to obtain the sound source spatial feature and the sound source temporal feature is specifically as follows: Extracting the sound source spatial features from the sound source position information according to the time difference between each sound source reaching the sound pickup device; According to the start time and duration of each sound source, a sound source time feature is extracted from the sound source position information.

6. The adaptive sound pickup method based on multi-sound source localization according to claim 1, characterized in that: The comprehensive feature vector set is input into a preset sound source localization model for processing to obtain the target sound source features, specifically: Based on the comprehensive feature vector set, the sound source localization information, as well as the sound source distortion degree and the sound source confusion degree of the target sound source are obtained through the sound source localization model; According to the sound source localization information, a target signal corresponding to a target sound source is extracted from the multi-sound source sound data, and the target signal is optimized according to the sound source distortion degree and the sound source confusion degree of the target sound source to obtain the target sound source feature.

7. The adaptive sound pickup method based on multi-sound source localization according to claim 6, characterized in that: The sound source localization model is specifically: Among them, the loss function of the sound source localization model is expressed as: L total = W location L location +W distortion L distortion +W classification L classification ; Where, L total is the loss function of the sound source localization model, W location is the weight of the sound source localization loss, L location is the sound source localization loss, W distortion is the weight of the sound source distortion loss, L distortion is the distortion loss of the sound source, W classification is the weight of the loss of sound source confusion, L classification It is the loss of sound source confusion.

8. The adaptive sound pickup method based on multi-sound source localization according to claim 1, characterized in that: According to the target sound source characteristics, the optimal sound pickup device parameters in the current environment are obtained through a preset sound pickup strategy prediction model, specifically: According to the characteristics of the target sound source, the sound pickup strategy prediction model selects the optimal sound pickup strategy that meets the current environment from a preset sound pickup strategy library; The optimal sound pickup device parameters are determined according to the hardware parameters of the sound pickup device and the optimal sound pickup strategy.

9. The adaptive sound pickup method based on multi-sound source localization according to claim 1, characterized in that: The step of setting the adjustable parameters of the sound pickup device to pick up sound according to the optimal sound pickup device parameters is specifically as follows: According to the optimal sound pickup device parameters, the sound pickup gain parameters, frequency response parameters, sampling rate and noise threshold value of the sound pickup device are set.

10. An adaptive sound pickup system based on multi-sound source localization, characterized in that: include: Data preprocessing module, sound source localization module, optimal sound pickup parameter acquisition module and sound pickup device setting module; The data preprocessing module is used to extract features from the collected multi-source sound data to obtain a comprehensive feature vector set; The sound source localization module is used to input the comprehensive feature vector set into a preset sound source localization model for processing to obtain sound source localization information and sound source error information; The optimal sound pickup parameter acquisition module is used to obtain the optimal sound pickup device parameters in the current environment through a preset sound pickup strategy prediction model according to the sound source positioning information and the sound source error information; The sound pickup device setting module is used to set the sound pickup device to pick up sound according to the optimal sound pickup device parameters.

Citation Information

Patent Citations

  • Signal generation method and voice identification method and device based on artificial intelligence

    CN110415718A

  • Anti-crosstalk method, device and equipment based on multi-pickup scene

    CN112151036A

  • Self-adaptive pickup method with voiceprint recognition

    CN117877491A

  • Noise suppression device, method and program

    JP2014029407A

  • Sound source information estimation device, sound source information estimation method and program

    JP2017107141A

Cited By

  • Intelligent regulation and control method and system for high-fidelity aluminum strip pickup device

    CN119562187A