Audio data mixing method and system based on sound source parameters

Abstract

The present invention discloses a method and system for mixing audio data based on sound source parameters, the method comprising: obtaining multiple audio data to be mixed and corresponding transmission parameters corresponding to a target playback device; determining source device parameters corresponding to each of the audio data based on the transmission parameters based on a prediction algorithm; determining processing parameters corresponding to each of the audio data based on the device parameters of the target playback device and the source device parameters; determining mixing strategies corresponding to at least two of the audio data based on the processing parameters and the source device parameters based on a dynamic programming algorithm; the mixing strategy is used to indicate a way to mix the audio data that has been preprocessed based on the processing parameters. It can be seen that the present invention can fully combine the prediction of the source device parameters of the audio data to improve the accuracy and efficiency of audio mixing, and improve the intelligence of audio technology and the user's auditory experience.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to an audio data mixing method and system based on sound source parameters. Background Art

[0002] The accurate mixing and display of audio data can effectively improve the accuracy of information dissemination and the user's sound environment experience. Therefore, how to effectively and accurately mix different audio data is an important technical issue. However, most of the existing audio data mixing technologies still use pre-set rules to directly mix audio data from different sources, without using prediction algorithms to more directly and accurately predict the source device parameters corresponding to the audio data and further determine the mixing strategy. Therefore, the effect and efficiency of its audio mixing are poor, and the user experience is also relatively general. It can be seen that the existing technology has defects that need to be solved urgently. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide an audio data mixing method and system based on sound source parameters, which can fully combine the prediction of the source device parameters of the audio data to improve the accuracy and efficiency of audio mixing, and improve the intelligence level of audio technology and the user's auditory experience.

[0004] In order to solve the above technical problems, the first aspect of the present invention discloses an audio data mixing method based on sound source parameters, the method comprising:

[0005] Acquire multiple audio data to be mixed and corresponding transmission parameters corresponding to the target playback device;

[0006] Based on the prediction algorithm, according to the transmission parameters, determining the source device parameters corresponding to each of the audio data;

[0007] Determining processing parameters corresponding to each of the audio data according to device parameters of the target playback device and parameters of the source device;

[0008] Based on the processing parameters and the source device parameters, a mixing strategy corresponding to at least two of the audio data is determined based on a dynamic programming algorithm; the mixing strategy is used to indicate a method for mixing the audio data that has been preprocessed based on the processing parameters.

[0009] As an optional implementation, in the first aspect of the present invention, the transmission parameters include at least one of a source IP address, a transmission medium type, a transmission speed, a transmission verification character, and a transmission encryption parameter.

[0010] As an optional implementation, in the first aspect of the present invention, determining the source device parameter corresponding to each of the audio data based on the prediction algorithm and the transmission parameter includes:

[0011] For each of the audio data, determining a source device prediction neural network corresponding to the audio data; the source device prediction neural network is trained by a training data set including a plurality of training audio data and corresponding transmission parameter annotations and source device parameter annotations;

[0012] Inputting the transmission parameters corresponding to the audio data into the source device prediction neural network to obtain the prediction device parameters corresponding to the audio data;

[0013] The source device parameters corresponding to the audio data are determined according to the predicted device parameters corresponding to the audio data.

[0014] As an optional implementation, in the first aspect of the present invention, determining the source device prediction neural network corresponding to the audio data includes:

[0015] For each candidate prediction neural network, obtaining a training data set and a validation data set of the candidate prediction neural network;

[0016] Calculate the proportion of all training audio data in the training data set whose similarity with the audio data is greater than a first threshold, and obtain a proportion parameter;

[0017] Calculate the prediction accuracy corresponding to all verification audio data in the verification data set whose similarity with the audio data is greater than a second threshold, and obtain an accuracy parameter;

[0018] Calculate the weighted average of the proportion parameter and the accuracy parameter to obtain the network priority parameter of the candidate prediction neural network;

[0019] The neural network with the highest network priority parameter among all the candidate prediction neural networks is determined as the source device prediction neural network corresponding to the audio data.

[0020] As an optional implementation, in the first aspect of the present invention, determining the source device parameter corresponding to the audio data according to the predicted device parameter corresponding to the audio data includes:

[0021] Screening out a plurality of similar audio data having a similarity with the audio data greater than a third threshold from all other audio data;

[0022] Calculating parameter similarity between the predicted device parameters of each of the similar audio data and the predicted device parameters of the audio data;

[0023] Filter out the predicted device parameters of all the similar audio data whose parameter similarity is greater than a fourth threshold, to obtain a similar device parameter set;

[0024] Calculating the intersection of all the predicted device parameters in the similar device parameter set to obtain intersection device parameters;

[0025] The union of the intersection device parameters and the predicted device parameters corresponding to the audio data is calculated to obtain the source device parameters corresponding to the audio data.

[0026] As an optional implementation, in the first aspect of the present invention, the source device parameters include at least one of a source user type, a source device type, a source biological type, a source device geographic location, and a source device current activity type.

[0027] As an optional implementation, in the first aspect of the present invention, determining the processing parameters corresponding to each of the audio data according to the device parameters of the target playback device and the source device parameters includes:

[0028] For each of the audio data, obtaining the source device parameter corresponding to the audio data;

[0029] Input the device parameters of the target playback device and the source device parameters corresponding to the audio data into a trained collaborative relationship prediction neural network to obtain the device collaborative relationship corresponding to the audio data and the target playback device; the device collaborative relationship includes background music relationship, same-camp user interaction relationship, hostile-camp user interaction relationship, same-environment mixed relationship or different-environment mixed relationship;

[0030] According to the correspondence between the preset collaborative relationship and the processing parameters, the processing parameters corresponding to the device collaborative relationship are determined to obtain the processing parameters corresponding to the audio data; the processing parameters include at least one of sampling parameters, encoding parameters, decoding parameters, compression parameters, noise reduction parameters and echo cancellation parameters.

[0031] As an optional implementation, in the first aspect of the present invention, the determining, based on the processing parameters and the source device parameters and based on a dynamic programming algorithm, a mixing strategy corresponding to at least two of the audio data comprises:

[0032] The objective function is set to maximize the number of audio data included in the mixed scheme and minimize the average value of the processing degree corresponding to all the audio data in the mixed scheme; the processing degree value is obtained by predicting the processing parameters of the audio data through a trained degree prediction neural network;

[0033] Setting restrictions includes:

[0034] The audio data whose processing degree value in the mixing scheme is less than the fifth threshold value is eliminated;

[0035] The mixing weight of each audio data in the mixing scheme is the product of the first weight and the second weight; the first weight is proportional to the processing degree value of the corresponding audio data; the second weight is proportional to the number of all the audio data belonging to the same device cooperation relationship corresponding to the corresponding audio data;

[0036] The difference between the mixing weights of any two audio data in the mixing scheme that belong to the same device cooperation relationship with the target playback device is less than a sixth threshold;

[0037] The difference between the mixing weights of any two audio data in different device cooperation relationships with the target playback device in the mixing scheme is greater than a seventh threshold;

[0038] Based on the dynamic programming algorithm, all the audio data are iteratively mixed according to the objective function and the constraint conditions until convergence, and an optimal mixing solution is obtained, which is determined as the corresponding mixing strategy.

[0039] A second aspect of an embodiment of the present invention discloses an audio data mixing system based on sound source parameters, the system comprising:

[0040] An acquisition module, used for acquiring a plurality of audio data to be mixed and corresponding transmission parameters corresponding to a target playback device;

[0041] A prediction module, used for determining source device parameters corresponding to each of the audio data according to the transmission parameters based on a prediction algorithm;

[0042] A determination module, used for determining a processing parameter corresponding to each of the audio data according to the device parameters of the target playback device and the source device parameters;

[0043] A mixing module is used to determine a mixing strategy corresponding to at least two of the audio data based on the processing parameters and the source device parameters and based on a dynamic programming algorithm; the mixing strategy is used to indicate a way to mix the audio data that has been preprocessed based on the processing parameters.

[0044] As an optional implementation, in the second aspect of the present invention, the transmission parameters include at least one of a source IP address, a transmission medium type, a transmission speed, a transmission verification character, and a transmission encryption parameter.

[0045] As an optional implementation, in the second aspect of the present invention, the prediction module determines the specific manner of the source device parameter corresponding to each of the audio data based on the prediction algorithm and the transmission parameters, including:

[0046] For each of the audio data, determining a source device prediction neural network corresponding to the audio data; the source device prediction neural network is trained by a training data set including a plurality of training audio data and corresponding transmission parameter annotations and source device parameter annotations;

[0047] Inputting the transmission parameters corresponding to the audio data into the source device prediction neural network to obtain the prediction device parameters corresponding to the audio data;

[0048] The source device parameters corresponding to the audio data are determined according to the predicted device parameters corresponding to the audio data.

[0049] As an optional implementation, in the second aspect of the present invention, the prediction module determines the specific manner in which the source device corresponding to the audio data predicts the neural network, including:

[0050] For each candidate prediction neural network, obtaining a training data set and a validation data set of the candidate prediction neural network;

[0051] Calculate the proportion of all training audio data in the training data set whose similarity with the audio data is greater than a first threshold, and obtain a proportion parameter;

[0052] Calculate the prediction accuracy corresponding to all verification audio data in the verification data set whose similarity with the audio data is greater than a second threshold, and obtain an accuracy parameter;

[0053] Calculate the weighted average of the proportion parameter and the accuracy parameter to obtain the network priority parameter of the candidate prediction neural network;

[0054] The neural network with the highest network priority parameter among all the candidate prediction neural networks is determined as the source device prediction neural network corresponding to the audio data.

[0055] As an optional implementation, in the second aspect of the present invention, the prediction module determines the specific manner of the source device parameter corresponding to the audio data according to the prediction device parameter corresponding to the audio data, including:

[0056] Screening out a plurality of similar audio data having a similarity with the audio data greater than a third threshold from all other audio data;

[0057] Calculating parameter similarity between the predicted device parameters of each of the similar audio data and the predicted device parameters of the audio data;

[0058] Filter out the predicted device parameters of all the similar audio data whose parameter similarity is greater than a fourth threshold, to obtain a similar device parameter set;

[0059] Calculating the intersection of all the predicted device parameters in the similar device parameter set to obtain intersection device parameters;

[0060] The union of the intersection device parameters and the predicted device parameters corresponding to the audio data is calculated to obtain the source device parameters corresponding to the audio data.

[0061] As an optional implementation, in the second aspect of the present invention, the source device parameters include at least one of a source user type, a source device type, a source biological type, a source device geographic location, and a source device current activity type.

[0062] As an optional implementation, in the second aspect of the present invention, the specific manner in which the determination module determines the processing parameters corresponding to each of the audio data according to the device parameters of the target playback device and the source device parameters includes:

[0063] For each of the audio data, obtaining the source device parameter corresponding to the audio data;

[0064] Input the device parameters of the target playback device and the source device parameters corresponding to the audio data into a trained collaborative relationship prediction neural network to obtain the device collaborative relationship corresponding to the audio data and the target playback device; the device collaborative relationship includes background music relationship, same-camp user interaction relationship, hostile-camp user interaction relationship, same-environment mixed relationship or different-environment mixed relationship;

[0065] According to the correspondence between the preset collaborative relationship and the processing parameters, the processing parameters corresponding to the device collaborative relationship are determined to obtain the processing parameters corresponding to the audio data; the processing parameters include at least one of sampling parameters, encoding parameters, decoding parameters, compression parameters, noise reduction parameters and echo cancellation parameters.

[0066] As an optional implementation, in the second aspect of the present invention, the mixing module determines a specific manner of mixing strategies corresponding to at least two of the audio data based on the processing parameters and the source device parameters and based on a dynamic programming algorithm, including:

[0067] The objective function is set to maximize the number of audio data included in the mixed scheme and minimize the average value of the processing degree corresponding to all the audio data in the mixed scheme; the processing degree value is obtained by predicting the processing parameters of the audio data through a trained degree prediction neural network;

[0068] Setting restrictions includes:

[0069] The audio data whose processing degree value in the mixing scheme is less than the fifth threshold value is eliminated;

[0070] The mixing weight of each audio data in the mixing scheme is the product of the first weight and the second weight; the first weight is proportional to the processing degree value of the corresponding audio data; the second weight is proportional to the number of all the audio data belonging to the same device cooperation relationship corresponding to the corresponding audio data;

[0071] The difference between the mixing weights of any two audio data in the mixing scheme that belong to the same device cooperation relationship with the target playback device is less than a sixth threshold;

[0072] The difference between the mixing weights of any two audio data in different device cooperation relationships with the target playback device in the mixing scheme is greater than a seventh threshold;

[0073] Based on the dynamic programming algorithm, all the audio data are iteratively mixed according to the objective function and the constraint conditions until convergence, and an optimal mixing solution is obtained, which is determined as the corresponding mixing strategy.

[0074] The third aspect of the present invention discloses another audio data mixing system based on sound source parameters, the system comprising:

[0075] A memory storing executable program code;

[0076] a processor coupled to the memory;

[0077] The processor calls the executable program code stored in the memory to execute part or all of the steps in the audio data mixing method based on sound source parameters disclosed in the first aspect of the present invention.

[0078] The fourth aspect of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute some or all of the steps in the audio data mixing method based on sound source parameters disclosed in the first aspect of the present invention.

[0079] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0080] The present invention can determine the source device parameters corresponding to each audio data according to the transmission parameters based on a prediction algorithm, and then determine the processing parameters corresponding to the audio data according to the device parameters of the target playback device and the source device parameters, so as to finally determine the mixing strategy corresponding to the audio data based on a dynamic programming algorithm, thereby being able to fully combine the prediction of the source device parameters of the audio data to improve the accuracy and efficiency of audio mixing, and improve the intelligence of audio technology and the user's auditory experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0082] Figure 1 It is a flowchart of an audio data mixing method based on sound source parameters disclosed in an embodiment of the present invention.

[0083] Figure 2 It is a structural schematic diagram of an audio data mixing system based on sound source parameters disclosed in an embodiment of the present invention.

[0084] Figure 3 It is a structural schematic diagram of another audio data mixing system based on sound source parameters disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0085] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0086] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or equipment.

[0087] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0088] The present invention discloses an audio data mixing method and system based on sound source parameters, which can determine the source device parameters corresponding to each audio data according to the transmission parameters based on the prediction algorithm, and then determine the processing parameters corresponding to the audio data according to the device parameters of the target playback device and the source device parameters, so as to finally determine the mixing strategy corresponding to the audio data based on the dynamic programming algorithm, so as to fully combine the prediction of the source device parameters of the audio data to improve the accuracy and efficiency of audio mixing, improve the intelligence of audio technology and the user's auditory experience. The following are detailed descriptions.

[0089] Embodiment 1

[0090] See also Figure 1 , Figure 1 is a flow chart of an audio data mixing method based on sound source parameters disclosed in an embodiment of the present invention. Figure 1 The audio data mixing method based on sound source parameters described above can be applied to a data processing system / data processing device / data processing server (wherein the server includes a local processing server or a cloud processing server). Figure 1 As shown, the audio data mixing method based on sound source parameters may include the following operations:

[0091] 101. Acquire multiple audio data to be mixed and corresponding transmission parameters corresponding to a target playback device.

[0092] 102. Based on the prediction algorithm and according to the transmission parameters, the source device parameters corresponding to each audio data are determined.

[0093] 103. Determine processing parameters corresponding to each audio data according to device parameters of the target playback device and source device parameters.

[0094] 104. Determine a mixing strategy corresponding to at least two audio data based on the processing parameters and the source device parameters and based on a dynamic programming algorithm.

[0095] Optionally, the mixing strategy is used to indicate a manner of mixing the audio data that has been pre-processed based on the processing parameters.

[0096] It can be seen that the above-mentioned embodiments of the invention can determine the source device parameters corresponding to each audio data based on the transmission parameters based on the prediction algorithm, and then determine the processing parameters corresponding to the audio data based on the device parameters of the target playback device and the source device parameters, so as to finally determine the mixing strategy corresponding to the audio data based on the dynamic programming algorithm, so as to fully combine the prediction of the source device parameters of the audio data to improve the accuracy and efficiency of audio mixing, improve the intelligence level of audio technology and the user's auditory experience.

[0097] As an optional embodiment, in the above steps, the transmission parameters include at least one of a source IP address, a transmission medium type, a transmission speed, a transmission verification character, and a transmission encryption parameter.

[0098] It can be seen that through the above-mentioned optional embodiments, the content of the transmission parameters is limited to comprehensively characterize the transmission characteristics of the audio data, facilitate the subsequent prediction of the source device parameters and the determination of the mixing strategy, and assist in fully combining the prediction of the source device parameters of the audio data to improve the accuracy and efficiency of audio mixing, thereby improving the intelligence of audio technology and the user's auditory experience.

[0099] As an optional embodiment, in the above steps, determining the source device parameters corresponding to each audio data according to the transmission parameters based on the prediction algorithm includes:

[0100] For each audio data, determining a source device prediction neural network corresponding to the audio data; optionally, the source device prediction neural network is trained by a training data set including a plurality of training audio data and corresponding transmission parameter annotations and source device parameter annotations;

[0101] Inputting the transmission parameters corresponding to the audio data into the source device prediction neural network to obtain the prediction device parameters corresponding to the audio data;

[0102] The source device parameters corresponding to the audio data are determined according to the predicted device parameters corresponding to the audio data.

[0103] It can be seen that through the above-mentioned optional embodiments, the transmission parameters can be predicted based on the source device prediction neural network corresponding to the audio data to obtain accurate source device parameters of the audio data, which is convenient for determining the subsequent mixing strategy, and assists in fully combining the prediction of the source device parameters of the audio data to improve the accuracy and efficiency of audio mixing, thereby improving the intelligence of audio technology and the user's auditory experience.

[0104] As an optional embodiment, in the above step, determining the source device prediction neural network corresponding to the audio data includes:

[0105] For each candidate prediction neural network, obtaining a training data set and a validation data set of the candidate prediction neural network;

[0106] Calculate the proportion of all training audio data in the training data set whose similarity with the audio data is greater than a first threshold, and obtain a proportion parameter;

[0107] Calculate the prediction accuracy corresponding to all verification audio data in the verification data set whose similarity with the audio data is greater than a second threshold, and obtain an accuracy parameter;

[0108] Calculate the weighted average of the proportion parameter and the accuracy parameter to obtain the network priority parameter of the candidate prediction neural network;

[0109] The neural network with the highest network priority parameter among all candidate prediction neural networks is determined as the source device prediction neural network corresponding to the audio data.

[0110] It can be seen that through the above-mentioned optional embodiments, the priority of the candidate neural network can be determined based on parameters such as the proportion and accuracy in the training data set and the verification data set, so as to screen out a suitable neural network to accurately predict the source device parameters, facilitate the subsequent determination of the mixing strategy, and assist in fully combining the prediction of the source device parameters of the audio data to improve the accuracy and efficiency of audio mixing, thereby improving the intelligence of audio technology and the user's auditory experience.

[0111] As an optional embodiment, in the above step, determining the source device parameter corresponding to the audio data according to the predicted device parameter corresponding to the audio data includes:

[0112] Screening out a plurality of similar audio data having a similarity with the audio data greater than a third threshold from all other audio data;

[0113] Calculating parameter similarity between the predicted device parameters of each similar audio data and the predicted device parameters of the audio data;

[0114] Filter out predicted device parameters of all similar audio data whose parameter similarity is greater than a fourth threshold, and obtain a similar device parameter set;

[0115] Calculate the intersection of all predicted device parameters in the similar device parameter set to obtain the intersection device parameters;

[0116] The union of the intersection device parameters and the predicted device parameters corresponding to the audio data is calculated to obtain the source device parameters corresponding to the audio data.

[0117] It can be seen that through the above-mentioned optional embodiments, it is possible to determine the device parameters that may be ignored or misjudged in the current prediction based on the screening and intersection calculation of the predicted device parameter results of other audio data to be mixed in the same batch, so as to obtain more accurate source device parameters through union calculation, thereby facilitating the determination of subsequent mixing strategies and assisting in fully combining the prediction of the source device parameters of the audio data to improve the accuracy and efficiency of audio mixing, thereby improving the intelligence of audio technology and the user's auditory experience.

[0118] As an optional embodiment, in the above steps, the source device parameter includes at least one of a source user type, a source device type, a source biological type, a source device geographic location, and a source device current activity type.

[0119] It can be seen that through the above-mentioned optional embodiments, the content of the source device parameters is limited to comprehensively characterize the characteristics of the source device of the audio data, facilitate the subsequent determination of the mixing strategy, and assist in fully combining the prediction of the source device parameters of the audio data to improve the accuracy and efficiency of audio mixing, thereby improving the intelligence of audio technology and the user's auditory experience.

[0120] As an optional embodiment, in the above steps, determining the processing parameters corresponding to each audio data according to the device parameters of the target playback device and the source device parameters includes:

[0121] For each audio data, obtain the source device parameter corresponding to the audio data;

[0122] Input the device parameters of the target playback device and the source device parameters corresponding to the audio data into the trained collaborative relationship prediction neural network to obtain the device collaborative relationship corresponding to the audio data and the target playback device; optionally, the device collaborative relationship includes background music relationship, same-camp user interaction relationship, hostile-camp user interaction relationship, same-environment mixed relationship or different-environment mixed relationship;

[0123] According to the correspondence between the preset collaborative relationship and the processing parameters, the processing parameters corresponding to the device collaborative relationship are determined to obtain the processing parameters corresponding to the audio data; the processing parameters include at least one of sampling parameters, encoding parameters, decoding parameters, compression parameters, noise reduction parameters and echo cancellation parameters.

[0124] It can be seen that through the above-mentioned optional embodiments, the device cooperation relationship can be predicted according to the device parameters based on the trained cooperation relationship prediction neural network, and then the corresponding processing parameters can be determined based on the correspondence between the preset cooperation relationship and the processing parameters, so as to facilitate the subsequent reasonable and effective audio mixing, and assist in fully combining the prediction of the source device parameters of the audio data to improve the accuracy and efficiency of the audio mixing, thereby improving the intelligence of the audio technology and the user's auditory experience.

[0125] As an optional embodiment, in the above steps, determining the mixing strategy corresponding to at least two audio data based on the processing parameters and the source device parameters and based on the dynamic programming algorithm includes:

[0126] The objective function is set to maximize the number of audio data included in the mixed scheme and minimize the average value of the processing degree values ​​corresponding to all audio data in the mixed scheme; optionally, the processing degree value is obtained by predicting the processing parameters of the audio data through a trained degree prediction neural network;

[0127] Setting restrictions includes:

[0128] In the hybrid scheme, audio data with a processing degree value less than a fifth threshold value is eliminated;

[0129] The mixing weight of each audio data in the mixing scheme is the product of the first weight and the second weight; optionally, the first weight is proportional to the processing degree value of the corresponding audio data; the second weight is proportional to the number of all audio data belonging to the same device cooperation relationship corresponding to the corresponding audio data;

[0130] The difference between the mixing weights of any two audio data in the mixing scheme that belong to the same device cooperation relationship with the target playback device is less than a sixth threshold;

[0131] The difference between the mixing weights of any two audio data in different device cooperation relationships with the target playback device in the mixing scheme is greater than the seventh threshold;

[0132] Based on the dynamic programming algorithm, all audio data are iteratively mixed according to the objective function and constraints until convergence, and the optimal mixing solution is obtained and determined as the corresponding mixing strategy.

[0133] It can be seen that through the above-mentioned optional embodiments, the weights and processing parameters in the audio mixing can be restricted according to the preset objective function and constraints, so as to calculate the optimal and most suitable audio mixing strategy, and fully combine the prediction of the source device parameters of the audio data to improve the accuracy and efficiency of the audio mixing, thereby improving the intelligence of the audio technology and the user's auditory experience.

[0134] Embodiment 2

[0135] See also Figure 2 , Figure 2 is a structural diagram of an audio data mixing system based on sound source parameters disclosed in an embodiment of the present invention. Figure 2 The audio data mixing system based on sound source parameters described above can be applied to a data processing system / data processing device / data processing server (wherein the server includes a local processing server or a cloud processing server). Figure 2 As shown, the audio data mixing system based on sound source parameters may include:

[0136] The acquisition module 201 is used to acquire a plurality of audio data to be mixed and corresponding transmission parameters corresponding to the target playback device.

[0137] The prediction module 202 is used to determine the source device parameters corresponding to each audio data according to the transmission parameters based on a prediction algorithm.

[0138] The determination module 203 is used to determine the processing parameters corresponding to each audio data according to the device parameters of the target playback device and the source device parameters.

[0139] The mixing module 204 is used to determine a mixing strategy corresponding to at least two audio data based on the processing parameters and the source device parameters and based on a dynamic programming algorithm.

[0140] Optionally, the mixing strategy is used to indicate a manner of mixing the audio data that has been pre-processed based on the processing parameters.

[0141] It can be seen that the above-mentioned embodiments of the invention can determine the source device parameters corresponding to each audio data based on the transmission parameters based on the prediction algorithm, and then determine the processing parameters corresponding to the audio data based on the device parameters of the target playback device and the source device parameters, so as to finally determine the mixing strategy corresponding to the audio data based on the dynamic programming algorithm, so as to fully combine the prediction of the source device parameters of the audio data to improve the accuracy and efficiency of audio mixing, improve the intelligence level of audio technology and the user's auditory experience.

[0142] As an optional embodiment, the transmission parameters include at least one of a source IP address, a transmission medium type, a transmission speed, a transmission verification character, and a transmission encryption parameter.

[0143] It can be seen that through the above-mentioned optional embodiments, the content of the transmission parameters is limited to comprehensively characterize the transmission characteristics of the audio data, facilitate the subsequent prediction of the source device parameters and the determination of the mixing strategy, and assist in fully combining the prediction of the source device parameters of the audio data to improve the accuracy and efficiency of audio mixing, thereby improving the intelligence of audio technology and the user's auditory experience.

[0144] As an optional embodiment, the prediction module determines the specific method of the source device parameter corresponding to each audio data based on the prediction algorithm and the transmission parameter, including:

[0145] For each audio data, determining a source device prediction neural network corresponding to the audio data; optionally, the source device prediction neural network is trained by a training data set including a plurality of training audio data and corresponding transmission parameter annotations and source device parameter annotations;

[0146] Inputting the transmission parameters corresponding to the audio data into the source device prediction neural network to obtain the prediction device parameters corresponding to the audio data;

[0147] The source device parameters corresponding to the audio data are determined according to the predicted device parameters corresponding to the audio data.

[0148] It can be seen that through the above-mentioned optional embodiments, the transmission parameters can be predicted based on the source device prediction neural network corresponding to the audio data to obtain accurate source device parameters of the audio data, which is convenient for determining the subsequent mixing strategy, and assists in fully combining the prediction of the source device parameters of the audio data to improve the accuracy and efficiency of audio mixing, thereby improving the intelligence of audio technology and the user's auditory experience.

[0149] As an optional embodiment, the prediction module determines the specific manner in which the source device corresponding to the audio data predicts the neural network, including:

[0150] For each candidate prediction neural network, obtaining a training data set and a validation data set of the candidate prediction neural network;

[0151] Calculate the proportion of all training audio data in the training data set whose similarity with the audio data is greater than a first threshold, and obtain a proportion parameter;

[0152] Calculate the prediction accuracy corresponding to all verification audio data in the verification data set whose similarity with the audio data is greater than a second threshold, and obtain an accuracy parameter;

[0153] Calculate the weighted average of the proportion parameter and the accuracy parameter to obtain the network priority parameter of the candidate prediction neural network;

[0154] The neural network with the highest network priority parameter among all candidate prediction neural networks is determined as the source device prediction neural network corresponding to the audio data.

[0155] It can be seen that through the above-mentioned optional embodiments, the priority of the candidate neural network can be determined based on parameters such as the proportion and accuracy in the training data set and the verification data set, so as to screen out a suitable neural network to accurately predict the source device parameters, facilitate the subsequent determination of the mixing strategy, and assist in fully combining the prediction of the source device parameters of the audio data to improve the accuracy and efficiency of audio mixing, thereby improving the intelligence of audio technology and the user's auditory experience.

[0156] As an optional embodiment, the prediction module determines the specific manner of the source device parameter corresponding to the audio data according to the prediction device parameter corresponding to the audio data, including:

[0157] Screening out a plurality of similar audio data having a similarity with the audio data greater than a third threshold from all other audio data;

[0158] Calculating parameter similarity between the predicted device parameters of each similar audio data and the predicted device parameters of the audio data;

[0159] Filter out predicted device parameters of all similar audio data whose parameter similarity is greater than a fourth threshold, and obtain a similar device parameter set;

[0160] Calculate the intersection of all predicted device parameters in the similar device parameter set to obtain the intersection device parameters;

[0161] The union of the intersection device parameters and the predicted device parameters corresponding to the audio data is calculated to obtain the source device parameters corresponding to the audio data.

[0162] It can be seen that through the above-mentioned optional embodiments, it is possible to determine the device parameters that may be ignored or misjudged in the current prediction based on the screening and intersection calculation of the predicted device parameter results of other audio data to be mixed in the same batch, so as to obtain more accurate source device parameters through union calculation, thereby facilitating the determination of subsequent mixing strategies and assisting in fully combining the prediction of the source device parameters of the audio data to improve the accuracy and efficiency of audio mixing, thereby improving the intelligence of audio technology and the user's auditory experience.

[0163] As an optional embodiment, the source device parameter includes at least one of a source user type, a source device type, a source biological type, a source device geographic location, and a current activity type of the source device.

[0164] It can be seen that through the above-mentioned optional embodiments, the content of the source device parameters is limited to comprehensively characterize the characteristics of the source device of the audio data, facilitate the subsequent determination of the mixing strategy, and assist in fully combining the prediction of the source device parameters of the audio data to improve the accuracy and efficiency of audio mixing, thereby improving the intelligence of audio technology and the user's auditory experience.

[0165] As an optional embodiment, the determination module determines the specific manner of the processing parameter corresponding to each audio data according to the device parameters of the target playback device and the source device parameters, including:

[0166] For each audio data, obtain the source device parameter corresponding to the audio data;

[0167] Input the device parameters of the target playback device and the source device parameters corresponding to the audio data into the trained collaborative relationship prediction neural network to obtain the device collaborative relationship corresponding to the audio data and the target playback device; optionally, the device collaborative relationship includes background music relationship, same-camp user interaction relationship, hostile-camp user interaction relationship, same-environment mixed relationship or different-environment mixed relationship;

[0168] According to the correspondence between the preset collaborative relationship and the processing parameters, the processing parameters corresponding to the device collaborative relationship are determined to obtain the processing parameters corresponding to the audio data; the processing parameters include at least one of sampling parameters, encoding parameters, decoding parameters, compression parameters, noise reduction parameters and echo cancellation parameters.

[0169] It can be seen that through the above-mentioned optional embodiments, the device cooperation relationship can be predicted according to the device parameters based on the trained cooperation relationship prediction neural network, and then the corresponding processing parameters can be determined based on the correspondence between the preset cooperation relationship and the processing parameters, so as to facilitate the subsequent reasonable and effective audio mixing, and assist in fully combining the prediction of the source device parameters of the audio data to improve the accuracy and efficiency of the audio mixing, thereby improving the intelligence of the audio technology and the user's auditory experience.

[0170] As an optional embodiment, the mixing module determines the specific manner of the mixing strategy corresponding to at least two audio data based on the processing parameters and the source device parameters and the dynamic programming algorithm, including:

[0171] The objective function is set to maximize the number of audio data included in the mixed scheme and minimize the average value of the processing degree values ​​corresponding to all audio data in the mixed scheme; optionally, the processing degree value is obtained by predicting the processing parameters of the audio data through a trained degree prediction neural network;

[0172] Setting restrictions includes:

[0173] In the hybrid scheme, audio data with a processing degree value less than a fifth threshold value is eliminated;

[0174] The mixing weight of each audio data in the mixing scheme is the product of the first weight and the second weight; optionally, the first weight is proportional to the processing degree value of the corresponding audio data; the second weight is proportional to the number of all audio data belonging to the same device cooperation relationship corresponding to the corresponding audio data;

[0175] The difference between the mixing weights of any two audio data in the mixing scheme that belong to the same device cooperation relationship with the target playback device is less than a sixth threshold;

[0176] The difference between the mixing weights of any two audio data in different device cooperation relationships with the target playback device in the mixing scheme is greater than the seventh threshold;

[0177] Based on the dynamic programming algorithm, all audio data are iteratively mixed according to the objective function and constraints until convergence, and the optimal mixing solution is obtained and determined as the corresponding mixing strategy.

[0178] It can be seen that through the above-mentioned optional embodiments, the weights and processing parameters in the audio mixing can be restricted according to the preset objective function and constraints, so as to calculate the optimal and most suitable audio mixing strategy, and fully combine the prediction of the source device parameters of the audio data to improve the accuracy and efficiency of the audio mixing, thereby improving the intelligence of the audio technology and the user's auditory experience.

[0179] Embodiment 3

[0180] See also Figure 3 , Figure 3 It is another audio data mixing system based on sound source parameters disclosed in an embodiment of the present invention. Figure 3 The audio data mixing system based on sound source parameters is applied to a data processing system / data processing device / data processing server (wherein the server includes a local processing server or a cloud processing server). Figure 3 As shown, the audio data mixing system based on sound source parameters may include:

[0181] A memory 301 storing executable program codes;

[0182] a processor 302 coupled to the memory 301;

[0183] The processor 302 calls the executable program code stored in the memory 301 to execute the steps of the audio data mixing method based on sound source parameters described in the first embodiment.

[0184] Embodiment 4

[0185] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program for electronic data exchange, wherein the computer program enables a computer to execute the steps of the audio data mixing method based on sound source parameters described in the first embodiment.

[0186] Embodiment 5

[0187] An embodiment of the present invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute the steps of the audio data mixing method based on sound source parameters described in the first embodiment.

[0188] The above describes specific embodiments of the present specification, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0189] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0190] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0191] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification may be in the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of this specification may be in the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0192] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0193] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0194] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0195] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0196] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0197] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0198] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0199] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0200] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0201] Finally, it should be noted that the audio data mixing method and system based on sound source parameters disclosed in the embodiments of the present invention only disclose the preferred embodiments of the present invention, which are only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An audio data mixing method based on sound source parameters, characterized in that: The method comprises: Acquire multiple audio data to be mixed and corresponding transmission parameters corresponding to the target playback device; Based on the prediction algorithm, according to the transmission parameters, determining the source device parameters corresponding to each of the audio data; Determining processing parameters corresponding to each of the audio data according to the device parameters of the target playback device and the source device parameters, including: For each of the audio data, obtaining the source device parameter corresponding to the audio data; Input the device parameters of the target playback device and the source device parameters corresponding to the audio data into a trained collaborative relationship prediction neural network to obtain the device collaborative relationship corresponding to the audio data and the target playback device; the device collaborative relationship includes background music relationship, same-camp user interaction relationship, hostile-camp user interaction relationship, same-environment mixed relationship or different-environment mixed relationship; According to the correspondence between the preset cooperation relationship and the processing parameters, determining the processing parameters corresponding to the cooperation relationship of the devices to obtain the processing parameters corresponding to the audio data; the processing parameters include at least one of sampling parameters, encoding parameters, decoding parameters, compression parameters, noise reduction parameters and echo cancellation parameters; Based on the processing parameters and the source device parameters, a mixing strategy corresponding to at least two of the audio data is determined based on a dynamic programming algorithm; the mixing strategy is used to indicate a method for mixing the audio data that has been preprocessed based on the processing parameters.

2. The audio data mixing method based on sound source parameters according to claim 1, characterized in that: The transmission parameters include at least one of a source IP address, a transmission medium type, a transmission speed, a transmission verification character, and a transmission encryption parameter.

3. The audio data mixing method based on sound source parameters according to claim 1, characterized in that: The step of determining the source device parameter corresponding to each of the audio data based on the prediction algorithm and the transmission parameter includes: For each of the audio data, determining a source device prediction neural network corresponding to the audio data; the source device prediction neural network is trained by a training data set including a plurality of training audio data and corresponding transmission parameter annotations and source device parameter annotations; Inputting the transmission parameters corresponding to the audio data into the source device prediction neural network to obtain the prediction device parameters corresponding to the audio data; The source device parameters corresponding to the audio data are determined according to the predicted device parameters corresponding to the audio data.

4. The audio data mixing method based on sound source parameters according to claim 3 is characterized in that: The step of determining a source device prediction neural network corresponding to the audio data includes: For each candidate prediction neural network, obtaining a training data set and a validation data set of the candidate prediction neural network; Calculate the proportion of all training audio data in the training data set whose similarity with the audio data is greater than a first threshold, and obtain a proportion parameter; Calculate the prediction accuracy corresponding to all verification audio data in the verification data set whose similarity with the audio data is greater than a second threshold, and obtain an accuracy parameter; Calculate the weighted average of the proportion parameter and the accuracy parameter to obtain the network priority parameter of the candidate prediction neural network; The neural network with the highest network priority parameter among all the candidate prediction neural networks is determined as the source device prediction neural network corresponding to the audio data.

5. The audio data mixing method based on sound source parameters according to claim 3 is characterized in that: The step of determining the source device parameter corresponding to the audio data according to the predicted device parameter corresponding to the audio data includes: Screening out a plurality of similar audio data having a similarity with the audio data greater than a third threshold from all other audio data; Calculating parameter similarity between the predicted device parameters of each of the similar audio data and the predicted device parameters of the audio data; Filter out the predicted device parameters of all the similar audio data whose parameter similarity is greater than a fourth threshold, to obtain a similar device parameter set; Calculating the intersection of all the predicted device parameters in the similar device parameter set to obtain intersection device parameters; The union of the intersection device parameters and the predicted device parameters corresponding to the audio data is calculated to obtain the source device parameters corresponding to the audio data.

6. The audio data mixing method based on sound source parameters according to claim 1, characterized in that: The source device parameter includes at least one of a source user type, a source device type, a source biological type, a source device geographic location, and a source device current activity type.

7. The audio data mixing method based on sound source parameters according to claim 1, characterized in that: The step of determining, based on the processing parameters and the source device parameters and based on a dynamic programming algorithm, at least two mixing strategies corresponding to the audio data include: The objective function is set to maximize the number of audio data included in the mixed scheme and minimize the average value of the processing degree corresponding to all the audio data in the mixed scheme; the processing degree value is obtained by predicting the processing parameters of the audio data through a trained degree prediction neural network; Setting restrictions includes: The audio data whose processing degree value in the mixing scheme is less than the fifth threshold value is eliminated; The mixing weight of each audio data in the mixing scheme is the product of the first weight and the second weight; the first weight is proportional to the processing degree value of the corresponding audio data; the second weight is proportional to the number of all the audio data belonging to the same device cooperation relationship corresponding to the corresponding audio data; The difference between the mixing weights of any two audio data in the mixing scheme that belong to the same device cooperation relationship with the target playback device is less than a sixth threshold; The difference between the mixing weights of any two audio data in different device cooperation relationships with the target playback device in the mixing scheme is greater than a seventh threshold; Based on the dynamic programming algorithm, all the audio data are iteratively mixed according to the objective function and the constraint conditions until convergence, and an optimal mixing solution is obtained, which is determined as the corresponding mixing strategy.

8. An audio data mixing system based on sound source parameters, characterized in that: The system comprises: An acquisition module, used for acquiring a plurality of audio data to be mixed and corresponding transmission parameters corresponding to a target playback device; A prediction module, used for determining source device parameters corresponding to each of the audio data according to the transmission parameters based on a prediction algorithm; A determination module, used to determine the processing parameters corresponding to each of the audio data according to the device parameters of the target playback device and the source device parameters, including: For each of the audio data, obtaining the source device parameter corresponding to the audio data; Input the device parameters of the target playback device and the source device parameters corresponding to the audio data into a trained collaborative relationship prediction neural network to obtain the device collaborative relationship corresponding to the audio data and the target playback device; the device collaborative relationship includes background music relationship, same-camp user interaction relationship, hostile-camp user interaction relationship, same-environment mixed relationship or different-environment mixed relationship; According to the correspondence between the preset cooperation relationship and the processing parameters, determining the processing parameters corresponding to the cooperation relationship of the devices to obtain the processing parameters corresponding to the audio data; the processing parameters include at least one of sampling parameters, encoding parameters, decoding parameters, compression parameters, noise reduction parameters and echo cancellation parameters; A mixing module is used to determine a mixing strategy corresponding to at least two of the audio data based on the processing parameters and the source device parameters and based on a dynamic programming algorithm; the mixing strategy is used to indicate a way to mix the audio data that has been preprocessed based on the processing parameters.

9. An audio data mixing system based on sound source parameters, characterized in that: The system comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the audio data mixing method based on sound source parameters as described in any one of claims 1-7.

Images