Sound field reconstruction method and device, equipment and storage medium

Abstract

The invention discloses a sound field reconstruction method and device, equipment and a storage medium, and the method comprises the steps: obtaining expected sound pressure generated by a sound source to be reconstructed at a sound collection array, and a first transfer function set between a loudspeaker array and the sound collection array, the first transfer function set comprises a set representing signal transfer data between the loudspeaker array and the sound acquisition array; performing first optimization processing on the weight data of the loudspeaker array based on the expected sound pressure and the first transfer function set by using a target optimization model to obtain target weight data of the loudspeaker array; wherein when the loudspeaker array outputs the sound source signal according to the target weight data, the first difference between the target sound pressure generated by the loudspeaker array at the sound collection array and the expected sound pressure meets the preset threshold value requirement. In this way, the accuracy of sound field reconstruction can be improved.

Description

Technical Field

[0001] The present application relates to the field of acoustic technology, and in particular to a sound field reconstruction method, device, equipment and storage medium. Background Art

[0002] In recent years, with the continuous development of computer technology and multimedia technology, sound field reconstruction technology has been widely used in multimedia systems such as film, television, games, and theaters. Most of the sound field reconstruction technologies with high accuracy use the sound pressure value in the listening area of ​​the scene as the target sound field information, and use the speaker array to restore the sound pressure information to reconstruct the sound field. Among them, how to make the sound pressure information restored by the speaker array closer to the target sound field information to improve the accuracy of sound field reconstruction is of great significance. Summary of the invention

[0003] The main technical problem solved by the present application is to provide a sound field reconstruction method, device, equipment and storage medium, which can improve the accuracy of sound field reconstruction.

[0004] In order to solve the above technical problems, a technical solution adopted in the present application is: to provide a sound field reconstruction method, the method comprising: obtaining the expected sound pressure generated by the sound source to be reconstructed at the sound collection array, and a first transfer function set between the speaker array and the sound collection array, the first transfer function set being a collection of data including signal transmission between the speaker array and the sound collection array; using a target optimization model to perform a first optimization processing on the weight data of the speaker array based on the expected sound pressure and the first transfer function set to obtain the target weight data of the speaker array; wherein, when the speaker array outputs the sound source signal with the target weight data, the first difference between the target sound pressure generated by the speaker array at the sound collection array and the expected sound pressure meets the preset threshold requirement.

[0005] To solve the above technical problems, another technical solution adopted in the present application is: to provide a sound field reconstruction device, including: an acquisition module and an optimization module, the acquisition module is used to obtain the expected sound pressure generated by the sound source to be reconstructed at the sound collection array, and a first transfer function set between the speaker array and the sound collection array, the first transfer function set being data characterizing the signal transmission data between the speaker array and the sound collection array; the optimization module is used to optimize the weight data of the speaker array based on the expected sound pressure and the first transfer function set using a target optimization model to obtain the target weight data of the speaker array; wherein, when the speaker array outputs the sound source signal with the target weight data, the first difference between the target sound pressure generated by the speaker array at the sound collection array and the expected sound pressure meets the preset threshold requirement.

[0006] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, comprising a memory and a processor coupled to each other, the memory storing program instructions; the processor is used to execute the program instructions stored in the memory to implement the above method.

[0007] In order to solve the above technical problem, another technical solution adopted by the present application is: providing a computer-readable storage medium for storing program instructions, which can be executed to implement the above method.

[0008] In the above scheme, the expected sound pressure is the target of sound field reconstruction. When reconstructing the sound field, the present application uses a target optimization model to optimize the weight data of the speaker array based on the target of sound field reconstruction and the first transfer function set between the acoustic device array and the sound collection array to obtain the target weight data of the speaker array, so as to reconstruct the sound field using the target weight data, wherein the first transfer function set is a priori data obtained in advance, and the prior data is signal transmission data during the sound field propagation process. Compared with the method in which the model only uses the expected sound pressure to output the target weight data of the speaker array, the method in which the signal transmission data is used to output the target weight data of the speaker array in the present application takes into account the influence of signal transmission, can reduce the error caused by the uncertainty or complexity of signal transmission, and thus can improve the accuracy of sound field reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a flowchart of an embodiment of a sound field reconstruction method provided by the present application;

[0010] Figure 2 yes Figure 1 The flowchart of an embodiment before step S12 is shown;

[0011] Figure 3 It is a flow chart of an embodiment of iteratively optimizing an objective function using a preset optimization algorithm provided by the present application;

[0012] Figure 4 This is a flow chart of an embodiment of determining target weight data using a target optimization model provided by the present application;

[0013] Figure 5 It is a schematic diagram of the framework of an embodiment of a sound field reconstruction device provided by the present application;

[0014] Figure 6 It is a schematic diagram of a framework of an embodiment of an electronic device provided by the present application;

[0015] Figure 7 It is a schematic diagram of the framework of the computer-readable storage medium provided by this application. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail below with reference to the accompanying drawings and examples.

[0017] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0018] Among them, the general idea of ​​achieving sound field reconstruction is to determine the distribution weight of the target sound source signal on the speaker array set in the specified space for the target sound source signal to be reconstructed, and distribute the sound source signal to the speaker for playback according to the determined distribution weight, so as to restore the target sound source signal in the specified sound field reconstruction area, that is, to achieve sound field reconstruction in the sound field reconstruction area. In addition, most of the sound field reconstruction methods with higher accuracy use the sound pressure value in the listening area of ​​the scene as the target sound field information, and use the speaker array to restore the sound pressure information to achieve sound field reconstruction.

[0019] Therefore, if the sound field is to be reconstructed, the sound pressure reconstructed by the speaker array at the sound collection array must be equal to the expected sound pressure generated by the sound source to be reconstructed at the sound collection array. That is:

[0020] P s -HD

[0021] Where Ps represents the desired sound pressure, the result of HD represents the reconstructed sound pressure, H represents the transfer function set between the speaker array and the sound collection array, and D represents the distribution weight on the speaker array.

[0022] The sound field reconstruction method provided by the present application is: firstly obtain the expected sound pressure generated by the sound source to be reconstructed at the sound collection array, and the first transfer function set between the speaker array and the sound collection array; then use the target optimization model to determine the target weight data of the speaker array based on the expected sound pressure and the first transfer function set, that is, determine the target distribution weight of the sound source signal to be reconstructed on the speaker array. The target weight data of the speaker array can be represented in the form of a matrix.

[0023] Specifically, see Figure 1 , Figure 11 is a flow chart of an embodiment of the sound field reconstruction method provided by the present application. It should be noted that if there are substantially the same results, this embodiment does not necessarily use Figure 1 The process sequence shown is limited. Figure 1 As shown, this embodiment includes:

[0024] S11: obtaining the expected sound pressure generated by the sound source to be reconstructed at the sound collection array, and a first transfer function set between the speaker array and the sound collection array, wherein the first transfer function set is a set of data characterizing signal transfer between the speaker array and the sound collection array.

[0025] The sound source to be reconstructed is a sound source to be reconstructed or simulated. It can be an actual physical sound source, such as a musical instrument, human voice, machine noise, etc., or a virtual sound source, such as a sound source defined in a digital audio signal. It should be noted that, unless otherwise specified, the sound source to be reconstructed described in this application can refer to a sound source or a sound source signal emitted from the sound source.

[0026] In one embodiment, obtaining a first transfer function set between a speaker array and a sound collection array includes: controlling each speaker unit in the speaker array to sequentially send out a frequency sweep signal; measuring a response signal of each frequency sweep signal during transmission to the sound collection array; performing frequency domain analysis on the response signal to obtain a frequency domain analysis result; and comparing the frequency domain analysis results of the frequency sweep signal and the response signal to obtain a first transfer function set between the speaker array and the sound collection array. The first transfer function set is a set of data representing signal transmission between the speaker array and the sound collection array. The speaker array includes a plurality of speaker units, the sound collection array includes a plurality of sound collection units, and the first transfer function set includes a transfer function between the mth sound collection unit and the nth speaker unit.

[0027] In one embodiment, the expected sound pressure generated by the sound source to be reconstructed at the sound collection array can be obtained in the following manner: first, a third transfer function set between the sound source to be reconstructed and the sound collection array is obtained; then, the product of the third transfer function set and the sound source intensity of the sound source to be reconstructed is obtained, and the product is used as the expected sound pressure generated by the sound source to be reconstructed at the sound collection array.

[0028] The method for obtaining the third transfer function set may refer to the method for obtaining the first transfer function set, which will not be described in detail here.

[0029] S12: Performing a first optimization process on the weight data of the speaker array based on the expected sound pressure and the first transfer function set using the target optimization model to obtain the target weight data of the speaker array; wherein, when the speaker array outputs the sound source signal with the target weight data, the first difference between the target sound pressure generated by the speaker array at the sound collection array and the expected sound pressure meets the preset threshold requirement.

[0030] In this embodiment, the difference between the target sound pressure restored by the speaker array and the expected sound pressure meets the preset threshold requirement, which means that this embodiment can well restore the sound source signal to be reconstructed, thereby achieving accurate reconstruction of the sound field.

[0031] In one embodiment, the trained target optimization model may be directly used to perform a first optimization process on the weight data of the speaker array based on the expected sound pressure and the first transfer function set to obtain the target weight data of the speaker array.

[0032] In another embodiment, before step S12, the weight data of the speaker array may be preliminarily optimized using a related optimization algorithm to obtain a relatively good first weight data, and then the first weight data may be further optimized using a trained target optimization model based on the expected sound pressure and the first transfer function set to obtain the target weight data of the speaker array. To distinguish the above two different optimization processes, the optimization process preliminarily performed using the optimization algorithm may be referred to as the second optimization process, and the optimization process further performed using the target optimization model may be referred to as the first optimization process.

[0033] Among them, it is considered that the goal of sound field reconstruction is: the sound pressure reconstructed by the speaker array at the sound collection array is equal to the expected sound pressure generated by the sound source to be reconstructed at the sound collection array. That is:

[0034] P s -HD

[0035] Wherein, Ps represents the desired sound pressure, the result of HD represents the reconstructed sound pressure, H represents the transfer function set between the speaker array and the sound collection array, and D represents the distribution weight on the speaker array, that is, the weight data of the speaker array.

[0036] Therefore, if accurate sound field reconstruction is to be achieved, it is necessary to find D that makes the difference between the reconstructed sound pressure and the expected sound pressure close to 0.

[0037] Therefore, in one embodiment, before step S12, the weight data of the speaker array can be used as the optimization variable, and the difference between the expected sound pressure and the reconstructed sound pressure can be used as the optimization target to construct the objective function. Then, the constructed objective function is subjected to a second optimization process using a preset optimization algorithm to obtain the first weight data of the speaker array. The reconstructed sound pressure is obtained by performing the above preset operation on the optimization variable D and the first transfer function set H. For details about the implementation process of this embodiment, please refer to the following Figure 2 Related description of the illustrated embodiment.

[0038] Then, after determining the first weight data of the speaker array, a first optimization process is performed on the first weight data based on the expected sound pressure and the first transfer function set using the target optimization model to obtain target weight data.

[0039] That is, in this embodiment, step S12 includes: using the target optimization model to perform a first optimization process on the first weight data based on the expected sound pressure and the first transfer function set to obtain the target weight data. The target optimization model includes a plurality of weight optimization layers connected in sequence, and each weight optimization layer performs the first optimization process in sequence according to the connection order between the layers. The specific implementation process of performing the first optimization process on the first weight data using the target optimization model to obtain the target weight data can be referred to below Figure 4 Related description of the illustrated embodiment.

[0040] In the above scheme, the expected sound pressure is the target of sound field reconstruction. When reconstructing the sound field, the present application uses a target optimization model to optimize the weight data of the speaker array based on the target of sound field reconstruction and the first transfer function set between the acoustic device array and the sound collection array to obtain the target weight data of the speaker array, so as to reconstruct the sound field using the target weight data, wherein the first transfer function set is a priori data obtained in advance, and the prior data is signal transmission data during the sound field propagation process. Compared with the method in which the model only uses the expected sound pressure to output the target weight data of the speaker array, the method in which the signal transmission data is used to output the target weight data of the speaker array in the present application takes into account the influence of signal transmission, can reduce the error caused by the uncertainty or complexity of signal transmission, and thus can improve the accuracy of sound field reconstruction.

[0041] See also Figure 2 , Figure 2 yes Figure 1 The flowchart of an embodiment before step S12 is shown.

[0042] This embodiment includes:

[0043] S21: Using the weight data of the speaker array as an optimization variable and the second difference between the expected sound pressure and the reconstructed sound pressure as an optimization target, constructing an objective function; wherein the reconstructed sound pressure is obtained by performing a preset operation on the optimization variable and the first transfer function set.

[0044] In one embodiment, the constructed objective function only includes the square of the difference between the desired sound pressure and the reconstructed sound pressure as shown in the following formula.

[0045]

[0046] In another embodiment, the constructed objective function includes a constraint term in addition to the square of the difference between the desired sound pressure and the reconstructed sound pressure. The corresponding constructed objective function can be expressed as follows:

[0047]

[0048] In the formula, It represents the norm, which is a constraint term used to constrain the solution of the optimization variable D when solving the objective function. The norm of this embodiment can be selected from L1 norm, L2 norm, infinite norm (maximum norm) or Lp norm, etc., which is determined according to the optimization effect and is not specifically limited here.

[0049] Among them, λ in the constraint term is a regularization parameter, which can be solved in advance by using Tikhonov regularization method, generalized cross-validation method (GCV), and L-curve method.

[0050] S22: Performing a second optimization process on the objective function using a preset optimization algorithm to obtain first weight data of the speaker array.

[0051] In this embodiment, the preset optimization algorithm can be used to iteratively optimize the objective function until the third difference between the second weight data obtained in this optimization and the second weight data obtained in the previous optimization meets the preset difference requirement, or the number of iterative optimizations reaches a preset number, and then the second weight data obtained in the last optimization is used as the first weight data of the speaker array.

[0052] That is, the second weight data obtained from the last optimization when the number of iterative optimization reaches a preset number of times can be used as the first weight data; or, during the iterative process, if the third difference between the second weight data obtained from this optimization and the previous optimization meets the preset requirements, the second weight data obtained from the last optimization process of the two optimizations can be used as the first weight data of the speaker array.

[0053] In one embodiment, the preset difference requirement is that the third difference is less than the product of the second weight data obtained by the previous optimization and the second preset threshold. That is, when the following formula is satisfied, it is determined that the third difference meets the preset difference requirement:

[0054] ||D k D k-1 |≤δD k 1

[0055] Where D k Indicates the second weight data obtained in this optimization, D k-1 represents the second weight data obtained in the previous optimization, and δ represents the second preset threshold value. δ is an empirical value obtained. It is used to adjust the balance of weight difference and iteration time during the optimization process.

[0056] In another embodiment, without considering the iteration time, the preset difference requirement may also be set as the third difference being smaller than a preset difference threshold.

[0057] In some embodiments, see Figure 3 , Figure 3 1 is a flow chart of an embodiment of iteratively optimizing an objective function using a preset optimization algorithm provided by the present application. In this embodiment, iteratively optimizing an objective function using a preset optimization algorithm includes:

[0058] S31: Obtain the first reference weight data and the first auxiliary vector required for this optimization.

[0059] Among them, when this optimization is the first optimization, the first reference weight data and the first auxiliary vector required for this optimization are both preset initial data; when this optimization is not the first optimization, the first auxiliary vector required for this optimization is the result of a second preset operation on the fourth difference and the second weight data obtained from the previous optimization, the fourth difference is the difference between the second weight data obtained from the previous optimization and the first reference weight data required for the previous optimization, and the first reference weight data required for this optimization is the second weight data obtained from the previous optimization.

[0060] S32: Using a sound pressure obtained by performing a preset calculation on the first auxiliary vector and the first transfer function set as a first reference sound pressure.

[0061] S33: performing a first preset operation on a first difference between the expected sound pressure and the first reference sound pressure and the first auxiliary vector to determine a first residual between the expected sound pressure and the first reference sound pressure.

[0062] S34: Adjust the element values ​​in the first residual that are less than the first preset threshold to the first preset threshold, and retain the element values ​​in the first residual that are greater than the first preset threshold to obtain the adjusted first residual, and use the adjusted first residual as the second weight data obtained in this optimization.

[0063] S35: Repeat the steps of obtaining the first reference weight data and the first auxiliary vector required for this optimization and subsequent steps.

[0064] To facilitate understanding of the processing of steps S31-S35, the above iterative optimization process is briefly described below in conjunction with the following formula:

[0065]

[0066] D k =max(r k ,0)

[0067]

[0068] Among them, y k represents the first auxiliary vector required for the kth optimization process, Ps represents the desired sound pressure, H represents the first transfer function set, Hy k represents the first reference sound pressure, r k represents the first residual, D k represents the second weight data obtained by the kth optimization, D k-1 represents the second weight data obtained by the k-1th optimization, t k+1 represents the step size corresponding to the kth optimization, D k -D k-1 Indicates the fourth difference, D k-1 Represents the first reference weight data, y k+1 represents the first auxiliary vector required for the k+1th optimization.

[0069] Combined with the above formula, the kth optimization to be performed is regarded as this optimization. First, the first reference weight data D required for this optimization should be obtained. k-1 (i.e., the second weight data obtained from the previous optimization) and the first auxiliary vector y k Among them, when this optimization is the first optimization, the first reference weight data and the first auxiliary vector required for this optimization are both preset initial data (which can be expressed as D0); when this optimization is not the first optimization, the first auxiliary vector required for this optimization is the fourth difference and the second weight data D obtained in the previous optimization. k-1 The result of the second preset operation is the fourth difference, which is the second weight data D obtained from the previous optimization. k-1 The first reference weight data D required for the previous optimization k-2 The first reference weight data required for this optimization is the second weight data D obtained in the previous optimization. k-1 .

[0070] Furthermore, the first auxiliary vector y k The sound pressure Hy obtained by performing a preset calculation with the first transfer function set H k As the first reference sound pressure; then the desired sound pressure Ps and the first reference sound pressure Hy k The first difference between the first auxiliary vector y k Perform a first preset operation to determine a first residual r between the expected sound pressure and the first reference sound pressure k .

[0071] The parameter R used in the above-mentioned first preset operation process can be the matrix H T The maximum eigenvalue of H can of course also be determined as other values ​​according to actual conditions.

[0072] Furthermore, the first residual r k The element values ​​less than the first preset threshold are adjusted to the first preset threshold, and the element values ​​greater than the first preset threshold in the first residual are retained to obtain the adjusted first residual, and the adjusted first residual is used as the second weight data obtained in this optimization.

[0073] The first preset threshold in the formula is 0, that is, the processing in the formula is to convert the first residual r k The values ​​of elements less than 0 in are adjusted to, and the first residual r is retained. k The element value greater than 0 in is used to obtain the adjusted first residual, and the adjusted first residual is used as the second weight data obtained in this optimization. In this way, unnecessary weights in the weight data can be reduced, and sparsity can be promoted.

[0074] It should be noted that the first preset threshold in the formula is only used as an example. In other embodiments, the first preset threshold may be preset according to actual needs.

[0075] It should also be noted that the above formula can be used to continuously update the step size parameter during the iterative optimization process to promote rapid convergence. In the first iteration, the step size parameter of the current optimization can be adjusted according to the initialized step size parameter.

[0076] In addition, the above formulas are only used to illustrate the processing logic of each optimization process in the iterative optimization process, and the constants or weighted parameters recorded therein can be adjusted according to actual needs.

[0077] See also Figure 4 , Figure 4 1 is a flow chart of an embodiment of determining target weight data using a target optimization model provided by the present application. In this embodiment, the target optimization model is used to perform a first optimization process on the first weight data based on the expected sound pressure and the first transfer function set to obtain the target weight data, including:

[0078] S41: The weight optimization layer currently performing the first optimization process is used as the current optimization layer, and the second reference weight data and the second auxiliary vector of the current optimization layer are obtained, and the sound pressure obtained by performing the third preset operation on the second auxiliary vector and the first transfer function set is used as the second reference sound pressure.

[0079] S42: Perform a fourth preset operation on the second difference between the expected sound pressure and the second reference sound pressure and the second auxiliary vector to determine a second residual between the expected sound pressure and the second reference sound pressure.

[0080] S43: performing convolution processing on the second residual to obtain a processed second residual, and using the processed second residual as the third weight data obtained by optimizing the current optimization layer.

[0081] S44: Perform a fifth preset operation on the fifth difference between the third weight data and the second reference weight data and the third weight data to obtain a second operation result; wherein the second operation result is used as the second auxiliary vector of the next weight optimization layer of the current optimization layer, and the third weight data obtained by optimizing the current optimization layer is used as the second reference weight data of the next weight optimization layer of the current optimization layer; when the current optimization layer is the first weight optimization layer, the second reference weight data and the second auxiliary vector of the current optimization layer are both first weight data.

[0082] In this embodiment, the target optimization model includes a plurality of weight optimization layers connected in sequence, and each weight optimization layer performs the first optimization process in sequence according to the connection order between the layers.

[0083] To facilitate understanding of the specific processing process of each layer performing the first optimization processing, in this embodiment, the weight optimization layer currently performing the first optimization processing is taken as the current optimization layer, and the processing process of each optimization layer is understood by introducing the processing process of the current optimization layer in combination with the following formula.

[0084] r k -α k y k -β k H T {Hy k -P s ),

[0085]

[0086]

[0087] Among them, y k represents the second auxiliary vector of the kth optimization layer, Ps represents the desired sound pressure, H represents the first transfer function set, Hy k represents the second reference sound pressure, r k represents the second residual, D k represents the third weight data obtained by optimizing the k-th optimization layer, Indicates convolution processing, D k-1 represents the third weight data obtained by optimizing the k-1th optimization layer, and also represents the second reference weight data of the kth optimization layer, D k -D k-1 Indicates the fifth difference, y k+1 Represents the second auxiliary vector for performing the k+1th optimization layer.

[0088] First, obtain the second reference weight data D of the current optimization layer k-1 and the second auxiliary vector y k , and then the second auxiliary vector yk The sound pressure Hy obtained by performing the third preset operation on the first transfer function set H k as the second reference sound pressure.

[0089] Furthermore, the expected sound pressure Ps and the second reference sound pressure Hy k The second difference between the second auxiliary vector y k A fourth preset operation is performed to determine a second residual r between the desired sound pressure and the second reference sound pressure. k .

[0090] Then the second residual r k Perform convolution Get the processed second residual and convert the processed second residual As the third weight data D obtained by optimizing the current optimization layer k .

[0091] Furthermore, for the third weight data D k and the second reference weight data D k-1 The difference between the fifth and third weight data D k Perform the fifth preset operation to obtain the second operation result y k+1 Among them, the second operation result y k+1 The third weight data obtained by optimizing the current optimization layer is used as the second auxiliary vector of the next weight optimization layer of the current optimization layer, and the third weight data obtained by optimizing the current optimization layer is used as the second reference weight data of the next weight optimization layer of the current optimization layer.

[0092] It should be noted that, when the current optimization layer is the weight optimization layer located at the first position, the second reference weight data and the second auxiliary vector of the current optimization layer are both the first weight data.

[0093] It should also be noted that the above parameter α k , β k , μ k and They are all learnable parameters of the target optimization model, which are learned during the training phase of the target optimization model.

[0094] In some embodiments, before step S12, the following target optimization model training step is also included:

[0095] First, a number of sample data are obtained; each sample data includes a sample expected sound pressure generated by each sample sound source to be reconstructed at the sample sound collection array, and a second transfer function set between the sample speaker array and the sample sound collection array.

[0096] The sample sound collection array includes M sound collection units and is located in the target reconstruction sound field area, and the speaker array includes L speaker units. The speaker array is a ring array or a linear array, the ring array surrounds the target reconstruction sound field area, and the linear array points to the target reconstruction sound field area.

[0097] For the expected sound pressure of the target reconstructed sound field area, the expected sound pressure of the target area can be collected or calculated by using the positional relationship between the given target reconstructed sound field area and the sample sound source.

[0098] In addition, the transfer function set can be expressed in the form of a matrix. In addition, considering that the measurement of the transfer function is time-consuming and laborious, it is difficult to meet the number of sample data required for the target optimization model learning. Therefore, a large number of transfer function sets can also be obtained through simulation. Among them, for the free field, the Green function is generally used as the transfer matrix; the reverberation field uses the image sound source method or the sound line method to simulate the transfer function of the reverberation field.

[0099] Through the above method, a large number of transfer function representation matrices and expected pressures can be obtained as sample data. In addition, a large number of sample data for training can be generated by changing the position of the sample sound source, and the sample data is used as input data of the target optimization model.

[0100] Second, the target optimization model is used to optimize the weight data of the sample speaker array based on each sample data to obtain the sample weight data of the sample speaker array.

[0101] In this embodiment, the optimized weight data may be weight data that has not been optimized, or may be preliminary optimized weight data obtained after preliminary optimization, which may be selected according to actual needs.

[0102] Among them, it can be understood that the target optimization model is trained on the basis of the existing preliminary optimization weight data, which can effectively reduce the learning difficulty and model parameters of the target optimization model, and a better target optimization model can be obtained by training with fewer training samples.

[0103] In this embodiment, the target optimization model includes several weight optimization layers connected in sequence. Each weight optimization layer optimizes the weight data of the sample speaker array in turn based on the sample data according to the connection order between the layers, and uses the weight data obtained by the last layer of optimization as the sample weight data of the sample speaker array.

[0104] Third, determine the sample sound pressure corresponding to each sample weight data; wherein the sample sound pressure is the sound pressure generated by the sample speaker array at the sample sound collection array when the sample speaker array outputs the sound source signal with the sample weight data.

[0105] Fourth, based on a sixth difference between each sample sound pressure and the corresponding sample expected sound pressure, adjusting the network parameters of the target optimization model.

[0106] It can be understood that the goal of the target optimization model training is to make the sample sound pressure generated by the sample speaker array at the sample sound collection array infinitely close to the sample expected sound pressure when the sample speaker array outputs the sound source signal with sample weight data.

[0107] Therefore, when training the model, the network parameters of the target optimization model can be adjusted based on the sixth difference between the sound pressure of each sample and the expected sound pressure of the corresponding sample.

[0108] Since the target optimization model includes several weight optimization layers connected sequentially, in one embodiment, the update gradient can be calculated based on the sixth difference between each sample sound pressure and the corresponding sample expected sound pressure, and the network parameters of each weight optimization layer in the target optimization model can be adjusted using the gradient descent method to obtain the adjusted target optimization model.

[0109] The sixth difference between the sample sound pressure and the corresponding sample expected sound pressure may be calculated using a mean square error (MSE), a mean absolute error (MAE), or a Huber loss function.

[0110] Mean Squared Error (MSE) loss function: MSE is one of the most commonly used loss functions for regression tasks. It calculates the average square error between the reconstructed sample sound pressure and the sample expected sound pressure, that is, the square of the difference between the predicted value and the true value. The MSE loss function can train the target optimization model by minimizing the difference between the predicted value and the true value.

[0111] Mean Absolute Error (MAE) loss function: MAE is another commonly used loss function for regression tasks. It calculates the mean absolute error between the reconstructed sample sound pressure and the sample expected sound pressure, that is, the average of the absolute values ​​of the difference between the predicted value and the true value. The MAE loss function can train the target optimization model by minimizing the difference between the predicted value and the true value.

[0112] Huber loss function: Huber loss function is a loss function that balances MSE and MAE. It uses MSE loss function when the difference between the predicted value and the true value is small, and uses MAE loss function when the difference is large. Huber loss function can train the target optimization model by minimizing the difference between the predicted value and the true value.

[0113] In one embodiment, considering that the sound pressure of the sound field contains amplitude information and phase information, the sixth difference determined above includes the amplitude information difference and the phase information difference, and the amplitude information difference and the phase information difference are weighted using the weighting coefficients corresponding to the amplitude information difference and the phase information difference respectively to obtain the total loss.

[0114] It should be noted that, in the target model training process, since the network parameters of the target optimization model need to be adjusted based on the sixth difference between each sample sound pressure and the corresponding sample expected sound pressure, after determining the sample weight data, it is also necessary to determine the sample sound pressure generated by the sample speaker array at the sample sound collection array when the sample speaker array outputs the sound source signal with the sample weight data. However, in the model application stage, since the main purpose of optimizing using the target optimization model is to determine the target weight data of the speaker array, the target weight data of the speaker array can be obtained by optimizing using only the target optimization model.

[0115] See also Figure 5 , Figure 5 It is a schematic diagram of the framework of an embodiment of a sound field reconstruction device provided in the present application. In this embodiment, the sound field reconstruction device 50 includes an acquisition module 51 and an optimization module 52. The acquisition module 51 is used to obtain the expected sound pressure generated by the sound source to be reconstructed at the sound collection array, and the first transfer function set between the speaker array and the sound collection array, wherein the first transfer function set is data characterizing the signal transmission data between the speaker array and the sound collection array; the optimization module 52 is used to optimize the weight data of the speaker array based on the expected sound pressure and the first transfer function set using a target optimization model to obtain the target weight data of the speaker array; wherein, when the speaker array outputs the sound source signal with the target weight data, the first difference between the target sound pressure generated by the speaker array at the sound collection array and the expected sound pressure meets the preset threshold requirement.

[0116] In some embodiments, before the optimization module 52 uses the target optimization model to perform a first optimization process on the weight data of the speaker array based on the expected sound pressure and the first transfer function set to obtain the target weight data of the speaker array, it also includes: taking the weight data of the speaker array as the optimization variable, and taking the second difference between the expected sound pressure and the reconstructed sound pressure as the optimization target to construct a target function; wherein the reconstructed sound pressure is obtained by performing a preset operation on the optimization variable and the first transfer function set; performing a second optimization process on the target function using a preset optimization algorithm to obtain the first weight data of the speaker array; the optimization module 52 uses the target optimization model to perform a first optimization process on the weight data of the speaker array based on the expected sound pressure and the first transfer function set to obtain the target weight data of the speaker array, including: performing a first optimization process on the first weight data based on the expected sound pressure and the first transfer function set using the target optimization model to obtain the target weight data.

[0117] In some embodiments, a preset optimization algorithm is used to perform a second optimization process on the objective function to obtain first weight data for the speaker array, including: iteratively optimizing the objective function using a preset optimization algorithm until the third difference between the second weight data obtained from this optimization and the second weight data obtained from the previous optimization meets a preset difference requirement, or the number of iterative optimizations reaches a preset number; and using the second weight data obtained from the last optimization as the first weight data.

[0118] In some embodiments, the preset difference requirement includes that the third difference is less than the product of the second weight data obtained from the previous optimization and the second preset threshold; and / or, the objective function is iteratively optimized using a preset optimization algorithm, including: obtaining the first reference weight data and the first auxiliary vector required for this optimization; wherein, when this optimization is the first optimization, the first reference weight data and the first auxiliary vector required for this optimization are both preset initial data; when this optimization is not the first optimization, the first auxiliary vector required for this optimization is the result of a second preset operation performed on the fourth difference and the second weight data obtained from the previous optimization, and the fourth difference is the difference between the second weight data obtained from the previous optimization and the first reference weight data required for the previous optimization. , the first reference weight data required for this optimization is the second weight data obtained in the previous optimization; the sound pressure obtained by performing a preset calculation on the first auxiliary vector and the first transfer function set is used as the first reference sound pressure; the first difference between the expected sound pressure and the first reference sound pressure and the first auxiliary vector are subjected to a first preset operation to determine the first residual between the expected sound pressure and the first reference sound pressure; the element value in the first residual that is less than the first preset threshold is adjusted to the first preset threshold, and the element value in the first residual that is greater than the first preset threshold is retained to obtain the adjusted first residual, and the adjusted first residual is used as the second weight data obtained in this optimization; the first reference weight data and the first auxiliary vector required for this optimization and subsequent steps are repeatedly executed.

[0119] In some embodiments, the target optimization model includes a plurality of weight optimization layers connected in sequence, and each weight optimization layer performs the first optimization process in sequence according to the connection order between the layers.

[0120] In some embodiments, a first optimization process is performed on the first weight data based on the expected sound pressure and the first transfer function set using a target optimization model to obtain the target weight data, including: taking the weight optimization layer currently performing the first optimization process as the current optimization layer, obtaining the second reference weight data and the second auxiliary vector of the current optimization layer, and performing a third preset operation on the second auxiliary vector and the first transfer function set to obtain the sound pressure obtained as the second reference sound pressure; performing a fourth preset operation on the second difference between the expected sound pressure and the second reference sound pressure and the second auxiliary vector to determine the second residual between the expected sound pressure and the second reference sound pressure; performing a convolution process on the second residual to obtain a processed second residual, and using the processed second residual as the third weight data obtained by optimizing the current optimization layer; performing a fifth preset operation on the fifth difference between the third weight data and the second reference weight data and the third weight data to obtain a second operation result; wherein the second operation result is used as the second auxiliary vector of the next weight optimization layer of the current optimization layer, and the third weight data obtained by optimizing the current optimization layer is used as the second reference weight data of the next weight optimization layer of the current optimization layer; when the current optimization layer is the weight optimization layer in the first position, the second reference weight data and the second auxiliary vector of the current optimization layer are both first weight data.

[0121] In some embodiments, before using the target optimization model to perform a first optimization process on the first weight data based on the expected sound pressure and the first transfer function set to obtain the target weight data, the following training step of the target optimization model is also included: obtaining a number of sample data; each sample data includes a sample expected sound pressure generated by each sample sound source to be reconstructed at the sample sound collection array, and a second transfer function set between the sample speaker array and the sample sound collection array; using the target optimization model to optimize the weight data of the sample speaker array based on each sample data to obtain the sample weight data of the sample speaker array; determining the sample sound pressure corresponding to each sample weight data; wherein the sample sound pressure is the sound pressure generated by the sample speaker array at the sample sound collection array when the sample speaker array outputs the sound source signal with the sample weight data; based on the sixth difference between each sample sound pressure and the corresponding sample expected sound pressure, adjusting the network parameters of the target optimization model.

[0122] In some embodiments, the target optimization model includes a plurality of weight optimization layers connected in sequence, and each weight optimization layer optimizes the weight data of the sample speaker array in sequence based on the sample data in the order of connection between the layers; based on the sixth difference between each sample sound pressure and the corresponding sample expected sound pressure, the network parameters of the target optimization model are adjusted, including: adjusting the network parameters of each weight optimization layer in the target optimization model based on each sixth difference to obtain the adjusted target optimization model.

[0123] In some embodiments, the acquisition module 51 acquires the expected sound pressure generated by the sound source to be reconstructed at the sound collection array, including: acquiring a third transfer function set between the sound source to be reconstructed and the sound collection array; taking the product of the third transfer function set and the sound source intensity of the sound source to be reconstructed as the expected sound pressure; and / or, the acquisition module 51 acquires the first transfer function set between the speaker array and the sound collection array, including: controlling each speaker unit in the speaker array to send out a swept frequency signal in turn; measuring the response signal of each swept frequency signal in the process of being transmitted to the sound collection array; performing frequency domain analysis on the response signal to obtain a frequency domain analysis result; comparing the frequency domain analysis results of the swept frequency signal and the response signal to obtain the first transfer function set between the speaker array and the sound collection array; and / or, the weight data is represented in the form of a matrix.

[0124] See also Figure 6 , Figure 6 Schematic diagram of the framework of an electronic device according to an embodiment of the present application. In this embodiment, the electronic device 60 includes a memory 61 and a processor 62 coupled to each other.

[0125] The memory 61 stores program instructions, and the processor 62 is used to execute the program instructions stored in the memory 61 to implement the steps of any of the above method implementations. In a specific implementation scenario, the electronic device 60 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 60 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0126] Specifically, the processor 62 is used to control itself and the memory 61 to implement the steps of any of the above-mentioned embodiments. The processor 62 can also be called a CPU (Central Processing Unit). The processor 62 may be an integrated circuit chip with signal processing capabilities. The processor 62 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 62 can be implemented by an integrated circuit chip.

[0127] See also Figure 7 , Figure 7It is a schematic diagram of the framework of the computer-readable storage medium provided by the present application. The computer-readable storage medium 70 of the embodiment of the present application stores a program instruction 71, and when the program instruction 71 is executed, the method provided by any embodiment of the above method and any non-conflicting combination is implemented. Among them, the program instruction 71 can form a program file and be stored in the above-mentioned computer-readable storage medium 70 in the form of a software product, so that a computer device (which can be a personal computer, a server, or a network device, etc.) executes all or part of the steps of each implementation method of the present application. The aforementioned computer-readable storage medium 70 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, and a tablet.

[0128] In the above scheme, the expected sound pressure is the target of sound field reconstruction. When reconstructing the sound field, the present application uses a target optimization model to optimize the weight data of the speaker array based on the target of sound field reconstruction and the first transfer function set between the acoustic device array and the sound collection array to obtain the target weight data of the speaker array, so as to reconstruct the sound field using the target weight data, wherein the first transfer function set is a priori data obtained in advance, and the prior data is signal transmission data during the sound field propagation process. Compared with the method in which the model only uses the expected sound pressure to output the target weight data of the speaker array, the method in which the signal transmission data is used to output the target weight data of the speaker array in the present application takes into account the influence of signal transmission, can reduce the error caused by the uncertainty or complexity of signal transmission, and thus can improve the accuracy of sound field reconstruction.

[0129] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0130] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0131] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0132] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0133] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0134] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0135] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A sound field reconstruction method, characterized in that: The method comprises: Acquire an expected sound pressure generated by a sound source to be reconstructed at a sound collection array, and a first transfer function set between a speaker array and the sound collection array, wherein the first transfer function set is a set of data representing signal transfer between the speaker array and the sound collection array; A target optimization model is used to perform a first optimization process on the weight data of the speaker array based on the expected sound pressure and the first transfer function set to obtain the target weight data of the speaker array; wherein, when the speaker array outputs a sound source signal with the target weight data, a first difference between the target sound pressure generated by the speaker array at the sound collection array and the expected sound pressure meets a preset threshold requirement.

2. The method according to claim 1, characterized in that Before performing a first optimization process on the weight data of the speaker array based on the expected sound pressure and the first transfer function set using the target optimization model to obtain the target weight data of the speaker array, the method further includes: The objective function is constructed by taking the weight data of the speaker array as an optimization variable and taking the second difference between the desired sound pressure and the reconstructed sound pressure as an optimization target; wherein the reconstructed sound pressure is obtained by performing a preset operation on the optimization variable and the first transfer function set; Performing a second optimization process on the objective function using a preset optimization algorithm to obtain first weight data of the speaker array; The step of performing a first optimization process on the weight data of the speaker array based on the expected sound pressure and the first transfer function set using the target optimization model to obtain the target weight data of the speaker array includes: The target optimization model is used to perform a first optimization process on the first weight data based on the expected sound pressure and the first transfer function set to obtain the target weight data.

3. The method according to claim 2, characterized in that The performing a second optimization process on the objective function by using a preset optimization algorithm to obtain first weight data of the speaker array includes: Iteratively optimizing the objective function using the preset optimization algorithm until a third difference between the second weight data obtained by the current optimization and the second weight data obtained by the previous optimization meets a preset difference requirement, or the number of iterative optimizations reaches a preset number; The second weight data obtained from the last optimization is used as the first weight data.

4. The method according to claim 3, characterized in that The preset difference requirement includes that the third difference is smaller than the product of the second weight data obtained in the previous optimization and a second preset threshold; And / or, the iterative optimization of the objective function using the preset optimization algorithm includes: Obtaining first reference weight data and first auxiliary vector required for this optimization; wherein, when this optimization is the first optimization, the first reference weight data and first auxiliary vector required for this optimization are both preset initial data; when this optimization is not the first optimization, the first auxiliary vector required for this optimization is the result of a second preset operation performed on the fourth difference and the second weight data obtained in the previous optimization, the fourth difference is the difference between the second weight data obtained in the previous optimization and the first reference weight data required for the previous optimization, and the first reference weight data required for this optimization is the second weight data obtained in the previous optimization; using a sound pressure obtained by performing a preset calculation on the first auxiliary vector and the first transfer function set as a first reference sound pressure; performing a first preset operation on a first difference between the expected sound pressure and the first reference sound pressure and the first auxiliary vector to determine a first residual between the expected sound pressure and the first reference sound pressure; Adjusting the element values ​​of the first residual that are smaller than the first preset threshold to the first preset threshold, and retaining the element values ​​of the first residual that are larger than the first preset threshold, to obtain the adjusted first residual, and using the adjusted first residual as the second weight data obtained in this optimization; Repeat the steps of obtaining the first reference weight data and the first auxiliary vector required for this optimization and subsequent steps.

5. The method according to claim 1 or 2, characterized in that: The target optimization model includes a plurality of weight optimization layers connected in sequence, and each of the weight optimization layers performs the first optimization process in sequence according to the connection order between the layers.

6. The method according to claim 5, characterized in that The step of performing a first optimization process on the first weight data based on the expected sound pressure and the first transfer function set using the target optimization model to obtain the target weight data includes: The weight optimization layer currently performing the first optimization process is used as the current optimization layer, and second reference weight data and a second auxiliary vector of the current optimization layer are obtained, and a sound pressure obtained by performing a third preset operation on the second auxiliary vector and the first transfer function set is used as a second reference sound pressure; performing a fourth preset operation on a second difference between the expected sound pressure and the second reference sound pressure and the second auxiliary vector to determine a second residual between the expected sound pressure and the second reference sound pressure; Performing convolution processing on the second residual to obtain a processed second residual, and using the processed second residual as third weight data obtained by optimizing the current optimization layer; performing a fifth preset operation on a fifth difference between the third weight data and the second reference weight data and the third weight data to obtain a second operation result; Among them, the second operation result is used as the second auxiliary vector of the next weight optimization layer of the current optimization layer, and the third weight data obtained by optimizing the current optimization layer is used as the second reference weight data of the next weight optimization layer of the current optimization layer; when the current optimization layer is the first weight optimization layer, the second reference weight data and the second auxiliary vector of the current optimization layer are both the first weight data.

7. The method according to claim 1, characterized in that Before performing a first optimization process on the weight data of the speaker array based on the expected sound pressure and the first transfer function set using the target optimization model to obtain the target weight data of the speaker array, the method further includes the following training step of the target optimization model: Acquire a number of sample data; each of the sample data includes a sample expected sound pressure generated by each sample sound source to be reconstructed at the sample sound collection array, and a second transfer function set between the sample speaker array and the sample sound collection array; Optimizing the weight data of the sample speaker array based on each of the sample data using a target optimization model to obtain the sample weight data of the sample speaker array; Determine the sample sound pressure corresponding to each of the sample weight data; wherein the sample sound pressure is the sound pressure generated by the sample speaker array at the sample sound collection array when the sample speaker array outputs the sound source signal with the sample weight data; Based on a sixth difference between each of the sample sound pressures and the corresponding sample expected sound pressure, a network parameter of the target optimization model is adjusted.

8. The method according to claim 7, characterized in that The target optimization model includes a plurality of weight optimization layers connected in sequence, and each of the weight optimization layers sequentially optimizes the weight data of the sample speaker array based on the sample data in sequence according to the connection order between the layers; The adjusting the network parameters of the target optimization model based on the sixth difference between each of the sample sound pressures and the corresponding sample expected sound pressure comprises: Based on each of the sixth differences, the network parameters of each of the weight optimization layers in the target optimization model are adjusted to obtain the adjusted target optimization model.

9. The method according to claim 1, characterized in that: The obtaining of the expected sound pressure generated by the sound source to be reconstructed at the sound collection array includes: Acquire a third transfer function set between the sound source to be reconstructed and the sound collection array; taking the product of the third transfer function set and the sound source intensity of the sound source to be reconstructed as the expected sound pressure; And / or, obtaining the first transfer function set between the speaker array and the sound collection array comprises: Controlling each speaker unit in the speaker array to send out a sweep frequency signal in sequence; Measuring the response signal of each of the frequency sweep signals during the process of being transmitted to the sound collection array; Performing frequency domain analysis on the response signal to obtain a frequency domain analysis result; Comparing the frequency domain analysis results of the frequency sweep signal and the response signal to obtain the first transfer function set between the speaker array and the sound collection array; And / or, the weight data is represented in a matrix.

10. A sound field reconstruction device, characterized in that: The device comprises: An acquisition module, configured to acquire an expected sound pressure generated by a sound source to be reconstructed at a sound collection array, and a first transfer function set between a speaker array and the sound collection array, wherein the first transfer function set is data characterizing signal transfer data between the speaker array and the sound collection array; An optimization module is used to optimize the weight data of the speaker array based on the expected sound pressure and the first transfer function set using a target optimization model to obtain target weight data of the speaker array; wherein, when the speaker array outputs a sound source signal with the target weight data, a first difference between the target sound pressure generated by the speaker array at the sound collection array and the expected sound pressure meets a preset threshold requirement.

11. An electronic device, characterized in that: comprising a memory and a processor coupled to each other, The memory stores program instructions; The processor is used to execute the program instructions stored in the memory to implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions that can be run by a processor, and the program instructions can be executed by the processor to implement the method according to any one of claims 1 to 9.

Images