Audio data mixing method and system based on prediction algorithm

Abstract

The present invention discloses a method and system for mixing audio data based on a prediction algorithm, the method comprising: obtaining multiple audio data to be mixed; extracting audio feature parameters corresponding to each of the audio data based on a feature extraction model; determining user attention corresponding to each of the audio data based on historical user feedback records; determining a mixing strategy corresponding to the multiple audio data based on the audio feature parameters and the user attention based on a prediction algorithm; the mixing strategy is used to indicate an operation for mixing the multiple audio data. It can be seen that the present invention can achieve more efficient and accurate mixing of audio data based on audio features and user attention, so as to effectively improve sound effects and provide users with a better sound experience.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an audio data mixing method and system based on a prediction algorithm. Background Art

[0002] Accurate mixing and presentation of audio data can effectively improve the accuracy of information dissemination and the user's sound environment experience. Therefore, effectively and accurately mixing different audio data is a critical technical issue. However, most existing audio data mixing technologies still rely on manually set rules to directly mix audio data from different sources, rather than leveraging predictive algorithms that combine audio data characteristics and user feedback for more accurate mixing. As a result, these audio mixing technologies are generally ineffective and inefficient, resulting in a mediocre user experience. Clearly, existing technologies have shortcomings that urgently need to be addressed. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide an audio data mixing method and system based on a prediction algorithm, which can achieve more efficient and accurate mixing of audio data based on audio features and user attention, so as to effectively improve the sound effect and provide users with a better sound experience.

[0004] In order to solve the above technical problems, the first aspect of the present invention discloses an audio data mixing method based on a prediction algorithm, the method comprising:

[0005] Obtain multiple audio data to be mixed;

[0006] Extracting audio feature parameters corresponding to each of the audio data based on a feature extraction model;

[0007] Determining user attention corresponding to each of the audio data based on historical user feedback records;

[0008] According to the audio feature parameters and the user attention, based on a prediction algorithm, a mixing strategy corresponding to the multiple audio data is determined; the mixing strategy is used to indicate an operation of mixing the multiple audio data.

[0009] As an optional implementation, in the first aspect of the present invention, the audio feature parameters include one or more of audio duration, audio volume change, audio instrument type, and audio text content.

[0010] As an optional embodiment, in the first aspect of the present invention, extracting audio feature parameters corresponding to each audio data based on a feature extraction model includes:

[0011] For each of the audio data, input the audio data into a trained classifier algorithm to determine an audio type parameter corresponding to the audio data; the audio type parameter includes one or more of a vocal type, a single instrument type, a mixed instrument type, and a musical genre type; the classifier algorithm is trained using a training data set including a plurality of training audio data and corresponding audio type parameter annotations;

[0012] Determining a target feature extraction model from a plurality of candidate feature extraction models based on the audio type parameter;

[0013] The audio data is input into the target feature extraction model to obtain audio feature parameters corresponding to the audio data.

[0014] As an optional embodiment, in the first aspect of the present invention, determining a target feature extraction model from a plurality of candidate feature extraction models based on the audio type parameter includes:

[0015] For each candidate feature extraction model, calculating the similarity between the audio type annotation in the training data set corresponding to the candidate feature extraction model and the audio type parameter;

[0016] The candidate feature extraction model with the highest similarity is determined as the target feature extraction model; wherein the candidate feature extraction model is trained by the following steps:

[0017] Training a preset fully convolutional neural network based on a training data set including a plurality of training audio data and corresponding audio feature parameter annotations until convergence;

[0018] The parameters of the feature extraction layer in the fully convolutional neural network are fixed to obtain a candidate feature extraction model.

[0019] As an optional implementation, in the first aspect of the present invention, determining the user attention corresponding to each audio data based on historical user feedback records includes:

[0020] Acquire historical user feedback records including feedback records of multiple users on different types of audio data;

[0021] For each of the audio data, based on the audio feature parameters corresponding to the audio data, a plurality of similar audio data are screened from all evaluated audio data in the historical user feedback records; wherein a feature similarity between the audio features of the similar audio data and the audio feature parameters is greater than a first similarity threshold;

[0022] Calculate the weighted sum average of the user feedback parameters corresponding to each of the similar audio data to obtain the user attention corresponding to the audio data.

[0023] As an optional embodiment, in the first aspect of the present invention, the user feedback parameter is the user's attention duration on the similar audio data, and the attention duration is calculated by the following steps:

[0024] When playing mixed audio including the similar audio data and other audio data to the user, obtaining a user face image;

[0025] Based on the face attention recognition model, continuously identifying the attention parameters corresponding to the user's face image;

[0026] First, continuously reduce the proportion of the similar audio data in the mixed audio until the attention parameter is lower than a preset first parameter threshold, and calculate the reduction duration;

[0027] Then, the proportion of the similar audio data in the mixed audio is continuously increased until the attention parameter is higher than a preset second parameter threshold, and the increase duration is calculated; the second parameter threshold is higher than the first parameter threshold;

[0028] The ratio of the reduced duration to the increased duration is calculated to obtain the attention duration corresponding to the similar audio data.

[0029] As an optional embodiment, in the first aspect of the present invention, when calculating the user attention, the weighted calculation weight corresponding to the user feedback parameter corresponding to each similar audio data is the product of a first weight and a second weight; the first weight is inversely proportional to the audio duration and / or average audio volume of the similar audio data; the second weight is directly proportional to the feature similarity corresponding to the similar audio data.

[0030] As an optional embodiment, in the first aspect of the present invention, determining the mixing strategy corresponding to the multiple audio data based on the audio feature parameters and the user attention based on a prediction algorithm includes:

[0031] The objective function is set such that the user satisfaction corresponding to the mixing strategy is greater than a satisfaction threshold; the user satisfaction is obtained by inputting mixing parameters corresponding to all the audio data in the mixing strategy and the user attention into a trained user satisfaction prediction model; the mixing parameters include a mixing volume weight, an audio mixing mode, and a mixed audio portion;

[0032] Setting restrictions includes:

[0033] The predicted average audio volume corresponding to the mixed volume weight of any audio data in the mixing strategy is greater than a volume threshold; the predicted average audio volume is obtained by inputting any audio data and the corresponding mixed volume weight into a volume prediction model;

[0034] The parameter similarity between the mixing parameters corresponding to any two audio data in the mixing strategy is greater than a second similarity threshold;

[0035] Based on a dynamic programming algorithm, the multiple audio data are iteratively calculated according to the objective function and the limiting conditions to obtain a mixing strategy corresponding to the multiple audio data.

[0036] A second aspect of an embodiment of the present invention discloses an audio data mixing system based on a prediction algorithm, the system comprising:

[0037] An acquisition module, used for acquiring multiple audio data to be mixed;

[0038] An extraction module, configured to extract audio feature parameters corresponding to each of the audio data based on a feature extraction model;

[0039] A determination module, configured to determine the user attention corresponding to each of the audio data based on historical user feedback records;

[0040] A prediction module is used to determine a mixing strategy corresponding to the multiple audio data based on the prediction algorithm according to the audio feature parameters and the user attention; the mixing strategy is used to indicate an operation of mixing the multiple audio data.

[0041] As an optional implementation, in the second aspect of the present invention, the audio feature parameters include one or more of audio duration, audio volume change, audio instrument type, and audio text content.

[0042] As an optional embodiment, in the second aspect of the present invention, the specific manner in which the extraction module extracts the audio feature parameters corresponding to each of the audio data based on the feature extraction model includes:

[0043] For each of the audio data, input the audio data into a trained classifier algorithm to determine an audio type parameter corresponding to the audio data; the audio type parameter includes one or more of a vocal type, a single instrument type, a mixed instrument type, and a musical genre type; the classifier algorithm is trained using a training data set including a plurality of training audio data and corresponding audio type parameter annotations;

[0044] Determining a target feature extraction model from a plurality of candidate feature extraction models based on the audio type parameter;

[0045] The audio data is input into the target feature extraction model to obtain audio feature parameters corresponding to the audio data.

[0046] As an optional embodiment, in the second aspect of the present invention, the specific manner in which the extraction module determines the target feature extraction model from a plurality of candidate feature extraction models based on the audio type parameter includes:

[0047] For each candidate feature extraction model, calculating the similarity between the audio type annotation in the training data set corresponding to the candidate feature extraction model and the audio type parameter;

[0048] The candidate feature extraction model with the highest similarity is determined as the target feature extraction model; wherein the candidate feature extraction model is trained by the following steps:

[0049] Training a preset fully convolutional neural network based on a training data set including a plurality of training audio data and corresponding audio feature parameter annotations until convergence;

[0050] The parameters of the feature extraction layer in the fully convolutional neural network are fixed to obtain a candidate feature extraction model.

[0051] As an optional embodiment, in the second aspect of the present invention, the determination module determines the specific manner in which the user attention corresponding to each audio data is obtained based on historical user feedback records, including:

[0052] Acquire historical user feedback records including feedback records of multiple users on different types of audio data;

[0053] For each of the audio data, based on the audio feature parameters corresponding to the audio data, a plurality of similar audio data are screened from all evaluated audio data in the historical user feedback records; wherein a feature similarity between the audio features of the similar audio data and the audio feature parameters is greater than a first similarity threshold;

[0054] Calculate the weighted sum average of the user feedback parameters corresponding to each of the similar audio data to obtain the user attention corresponding to the audio data.

[0055] As an optional embodiment, in the second aspect of the present invention, the user feedback parameter is the user's attention duration on the similar audio data, and the attention duration is calculated by the following steps:

[0056] When playing mixed audio including the similar audio data and other audio data to the user, obtaining a user face image;

[0057] Based on the face attention recognition model, continuously identifying the attention parameters corresponding to the user's face image;

[0058] First, continuously reduce the proportion of the similar audio data in the mixed audio until the attention parameter is lower than a preset first parameter threshold, and calculate the reduction duration;

[0059] Then, the proportion of the similar audio data in the mixed audio is continuously increased until the attention parameter is higher than a preset second parameter threshold, and the increase duration is calculated; the second parameter threshold is higher than the first parameter threshold;

[0060] The ratio of the reduced duration to the increased duration is calculated to obtain the attention duration corresponding to the similar audio data.

[0061] As an optional embodiment, in the second aspect of the present invention, when calculating the user attention, the weighted calculation weight corresponding to the user feedback parameter corresponding to each similar audio data is the product of a first weight and a second weight; the first weight is inversely proportional to the audio duration and / or average audio volume of the similar audio data; the second weight is directly proportional to the feature similarity corresponding to the similar audio data.

[0062] As an optional embodiment, in the second aspect of the present invention, the prediction module determines the specific manner of the mixing strategy corresponding to the multiple audio data based on the prediction algorithm according to the audio feature parameters and the user attention, including:

[0063] The objective function is set such that the user satisfaction corresponding to the mixing strategy is greater than a satisfaction threshold; the user satisfaction is obtained by inputting mixing parameters corresponding to all the audio data in the mixing strategy and the user attention into a trained user satisfaction prediction model; the mixing parameters include a mixing volume weight, an audio mixing mode, and a mixed audio portion;

[0064] Setting restrictions includes:

[0065] The predicted average audio volume corresponding to the mixed volume weight of any audio data in the mixing strategy is greater than a volume threshold; the predicted average audio volume is obtained by inputting any audio data and the corresponding mixed volume weight into a volume prediction model;

[0066] The parameter similarity between the mixing parameters corresponding to any two audio data in the mixing strategy is greater than a second similarity threshold;

[0067] Based on a dynamic programming algorithm, the multiple audio data are iteratively calculated according to the objective function and the limiting conditions to obtain a mixing strategy corresponding to the multiple audio data.

[0068] A third aspect of the present invention discloses another audio data mixing system based on a prediction algorithm, the system comprising:

[0069] a memory storing executable program code;

[0070] a processor coupled to the memory;

[0071] The processor calls the executable program code stored in the memory to execute part or all of the steps in the audio data mixing method based on the prediction algorithm disclosed in the first aspect of the present invention.

[0072] The fourth aspect of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute some or all of the steps in the audio data mixing method based on the prediction algorithm disclosed in the first aspect of the present invention.

[0073] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0074] The present invention can extract the audio feature parameters corresponding to each audio data based on the feature extraction model, and then determine the user attention corresponding to each audio data based on historical user feedback records, so as to comprehensively and accurately predict the mixing strategies corresponding to multiple audio data, thereby achieving more efficient and accurate mixing of audio data based on audio features and user attention, so as to effectively improve the sound effect and provide users with a better sound experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0076] Figure 1 This is a flow chart of an audio data mixing method based on a prediction algorithm disclosed in an embodiment of the present invention.

[0077] Figure 2 This is a structural diagram of an audio data mixing system based on a prediction algorithm disclosed in an embodiment of the present invention.

[0078] Figure 3 It is a structural diagram of another audio data mixing system based on a prediction algorithm disclosed in an embodiment of the present invention.

[0079] Figure 4 This is a code diagram of an audio data mixing dynamic programming algorithm disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0080] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0081] The terms "first," "second," and so on, in the description and claims of the present invention and the accompanying drawings are used to distinguish between different objects, not to describe a specific order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or device comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or device.

[0082] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0083] The present invention discloses a method and system for audio data mixing based on a prediction algorithm. This method can extract audio feature parameters corresponding to each audio data point based on a feature extraction model, and then determine the user attention corresponding to each audio data point based on historical user feedback records, thereby comprehensively and accurately predicting the mixing strategies corresponding to multiple audio data points. This method can achieve more efficient and accurate mixing of audio data based on audio features and user attention, effectively improving sound effects and providing users with a better sound experience. These are described in detail below.

[0084] Example 1

[0085] See also Figure 1 , Figure 1 This is a flow chart of an audio data mixing method based on a prediction algorithm disclosed in an embodiment of the present invention. Figure 1 The described audio data mixing method based on the prediction algorithm can be applied to a data processing system / data processing device / data processing server (wherein the server includes a local processing server or a cloud processing server). Figure 1As shown, the audio data mixing method based on the prediction algorithm may include the following operations:

[0086] 101. Obtain multiple audio data to be mixed.

[0087] 102. Extract audio feature parameters corresponding to each audio data based on the feature extraction model.

[0088] 103. Based on historical user feedback records, determine the user attention corresponding to each audio data.

[0089] 104. Determine a mixing strategy corresponding to the multiple audio data based on the prediction algorithm according to the audio feature parameters and the user's attention.

[0090] Optionally, the mixing strategy is used to indicate an operation of mixing multiple audio data.

[0091] It can be seen that the above-mentioned embodiments of the invention can extract the audio feature parameters corresponding to each audio data based on the feature extraction model, and then determine the user attention corresponding to each audio data based on historical user feedback records, so as to comprehensively and accurately predict the mixing strategies corresponding to multiple audio data, thereby being able to achieve more efficient and accurate mixing of audio data based on audio features and user attention, so as to effectively improve the sound effect, and thus provide users with a better sound experience.

[0092] As an optional embodiment, in the above steps, the audio feature parameters include one or more of audio duration, audio volume change, audio instrument type and audio text content.

[0093] It can be seen that through the above optional embodiments, the content of the audio feature parameters is clarified, which can more accurately and comprehensively characterize the characteristics of the audio data, so as to facilitate subsequent audio mixing predictions, and assist in achieving more efficient and accurate mixing of audio data based on audio features and user attention, so as to effectively improve the sound effect and thus provide users with a better sound experience.

[0094] As an optional embodiment, in the above step, extracting audio feature parameters corresponding to each audio data based on the feature extraction model includes:

[0095] For each audio data, input the audio data into a trained classifier algorithm to determine an audio type parameter corresponding to the audio data; optionally, the audio type parameter includes one or more of a vocal type, a single instrument type, a mixed instrument type, and a musical genre type; the classifier algorithm is trained using a training dataset including a plurality of training audio data and corresponding audio type parameter annotations;

[0096] Determining a target feature extraction model from a plurality of candidate feature extraction models based on an audio type parameter;

[0097] The audio data is input into a target feature extraction model to obtain audio feature parameters corresponding to the audio data.

[0098] It can be seen that through the above optional embodiments, the audio type parameters of the audio data can be determined based on the trained classifier algorithm, and then the target feature extraction model can be screened out to extract the audio feature parameters corresponding to the audio data, so as to facilitate subsequent audio mixing predictions, and assist in achieving more efficient and accurate mixing of audio data based on audio features and user attention, so as to effectively improve the sound effect, and thus provide users with a better sound experience.

[0099] As an optional embodiment, in the above step, determining a target feature extraction model from a plurality of candidate feature extraction models based on the audio type parameter includes:

[0100] For each candidate feature extraction model, calculate the similarity between the audio type annotation and the audio type parameters in the training data set corresponding to the candidate feature extraction model;

[0101] The candidate feature extraction model with the highest similarity is determined as the target feature extraction model; wherein the candidate feature extraction model is trained by the following steps:

[0102] Training a preset fully convolutional neural network based on a training data set including a plurality of training audio data and corresponding audio feature parameter annotations until convergence;

[0103] The parameters of the feature extraction layer in the fully convolutional neural network are fixed to obtain a candidate feature extraction model.

[0104] It can be seen that through the above optional embodiments, a suitable feature extraction model can be screened out based on the similarity between the audio type annotations and the audio type parameters in the training data set corresponding to the candidate feature extraction model, so as to facilitate subsequent audio mixing predictions, and assist in achieving more efficient and accurate mixing of audio data based on audio features and user attention, so as to effectively improve the sound effect and thus provide users with a better sound experience.

[0105] As an optional embodiment, in the above step, determining the user attention corresponding to each audio data based on historical user feedback records includes:

[0106] Acquire historical user feedback records including feedback records of multiple users on different types of audio data;

[0107] For each audio data, based on the audio feature parameters corresponding to the audio data, multiple similar audio data are screened from all evaluated audio data in the historical user feedback records; optionally, the feature similarity between the audio features of the similar audio data and the audio feature parameters is greater than a first similarity threshold;

[0108] Calculate the weighted sum average of the user feedback parameters corresponding to each similar audio data to obtain the user attention corresponding to the audio data.

[0109] It can be seen that through the above-mentioned optional embodiments, the user attention corresponding to each audio data can be determined based on the user feedback parameters corresponding to the audio data similar to each audio data in the evaluated audio data in the historical user feedback records, so as to facilitate subsequent audio mixing prediction, and assist in achieving more efficient and accurate mixing of audio data based on audio features and user attention, so as to effectively improve the sound effect and thus provide users with a better sound experience.

[0110] As an optional embodiment, in the above steps, the user feedback parameter is the user's attention duration on similar audio data, and the attention duration is calculated by the following steps:

[0111] When playing mixed audio including similar audio data and other audio data to the user, obtaining a user face image;

[0112] Based on the face attention recognition model, continuously identify the attention parameters corresponding to the user's face image;

[0113] First, continuously reduce the proportion of similar audio data in the mixed audio until the attention parameter is lower than the preset first parameter threshold, and calculate the reduction duration;

[0114] Then, the proportion of similar audio data in the mixed audio is continuously increased until the attention parameter is higher than a preset second parameter threshold, and the increase duration is calculated; optionally, the second parameter threshold is higher than the first parameter threshold;

[0115] Calculate the ratio of the reduced duration to the increased duration to obtain the attention duration corresponding to similar audio data.

[0116] It can be seen that through the above optional embodiments, the content and calculation steps of the user feedback parameters are clarified, which can accurately characterize the user's attention duration on the audio data, so as to facilitate subsequent audio mixing prediction, and assist in achieving more efficient and accurate mixing of audio data based on audio features and user attention, so as to effectively improve the sound effect and thus provide users with a better sound experience.

[0117] As an optional embodiment, in the above steps, when calculating user attention, the weighted calculation weight corresponding to the user feedback parameter corresponding to each similar audio data is the product of the first weight and the second weight; the first weight is inversely proportional to the audio duration and / or average audio volume of the similar audio data; the second weight is directly proportional to the feature similarity corresponding to the similar audio data.

[0118] It can be seen that through the above optional embodiments, the weight rules corresponding to different similar audio data in the user attention calculation are defined, so that the calculated user attention accurately represents the length of time the user pays attention to the audio data, so as to facilitate subsequent audio mixing prediction, and assist in achieving more efficient and accurate mixing of audio data based on audio features and user attention, so as to effectively improve the sound effect, and thus give users a better sound experience.

[0119] As an optional embodiment, in the above steps, determining the mixing strategy corresponding to the multiple audio data based on the prediction algorithm according to the audio feature parameters and the user's attention includes:

[0120] The objective function is set to be that the user satisfaction corresponding to the hybrid strategy is greater than a satisfaction threshold; optionally, the user satisfaction is obtained by inputting the hybrid parameters corresponding to all audio data in the hybrid strategy and the user attention into a trained user satisfaction prediction model; the hybrid parameters include the hybrid volume weight, the audio mixing method, and the hybrid audio portion;

[0121] Setting restrictions includes:

[0122] The predicted average audio volume corresponding to the mixed volume weight of any audio data in the mixing strategy is greater than the volume threshold; optionally, the predicted average audio volume is obtained by inputting any audio data and the corresponding mixed volume weight into a volume prediction model;

[0123] The parameter similarity between the mixing parameters corresponding to any two audio data in the mixing strategy is greater than a second similarity threshold;

[0124] Based on the dynamic programming algorithm, multiple audio data are iteratively calculated according to the objective function and the constraints to obtain the hybrid strategy corresponding to the multiple audio data.

[0125] In a specific embodiment, Figure 4 As shown, a solution of a dynamic programming model and a prediction model is implemented based on Python code. It can be seen that through the above optional embodiments, it is possible to perform iterative calculations through a dynamic programming algorithm based on a preset objective function and limiting conditions to obtain a mixing strategy corresponding to multiple audio data, so as to achieve more efficient and accurate mixing of audio data based on audio features and user attention, so as to effectively improve the sound effect and provide users with a better sound experience.

[0126] Example 2

[0127] See also Figure 2 , Figure 2 : is a structural diagram of an audio data mixing system based on a prediction algorithm disclosed in an embodiment of the present invention. Figure 2 The described audio data mixing system based on the prediction algorithm can be applied to a data processing system / data processing device / data processing server (wherein the server includes a local processing server or a cloud processing server). Figure 2 As shown, the audio data mixing system based on the prediction algorithm may include:

[0128] The acquisition module 201 is configured to acquire a plurality of audio data to be mixed.

[0129] The extraction module 202 is configured to extract audio feature parameters corresponding to each audio data based on a feature extraction model.

[0130] The determination module 203 is configured to determine the user attention corresponding to each audio data based on historical user feedback records.

[0131] The prediction module 204 is configured to determine a mixing strategy corresponding to the plurality of audio data based on a prediction algorithm according to audio feature parameters and user attention.

[0132] Optionally, the mixing strategy is used to indicate an operation of mixing multiple audio data.

[0133] It can be seen that the above-mentioned embodiments of the invention can extract the audio feature parameters corresponding to each audio data based on the feature extraction model, and then determine the user attention corresponding to each audio data based on historical user feedback records, so as to comprehensively and accurately predict the mixing strategies corresponding to multiple audio data, thereby being able to achieve more efficient and accurate mixing of audio data based on audio features and user attention, so as to effectively improve the sound effect, and thus provide users with a better sound experience.

[0134] As an optional embodiment, the audio feature parameters include one or more of audio duration, audio volume change, audio instrument type, and audio text content.

[0135] It can be seen that through the above optional embodiments, the content of the audio feature parameters is clarified, which can more accurately and comprehensively characterize the characteristics of the audio data, so as to facilitate subsequent audio mixing predictions, and assist in achieving more efficient and accurate mixing of audio data based on audio features and user attention, so as to effectively improve the sound effect and thus provide users with a better sound experience.

[0136] As an optional embodiment, the specific method of extracting the audio feature parameters corresponding to each audio data based on the feature extraction model by the extraction module includes:

[0137] For each audio data, input the audio data into a trained classifier algorithm to determine an audio type parameter corresponding to the audio data; optionally, the audio type parameter includes one or more of a vocal type, a single instrument type, a mixed instrument type, and a musical genre type; the classifier algorithm is trained using a training dataset including a plurality of training audio data and corresponding audio type parameter annotations;

[0138] Determining a target feature extraction model from a plurality of candidate feature extraction models based on an audio type parameter;

[0139] The audio data is input into a target feature extraction model to obtain audio feature parameters corresponding to the audio data.

[0140] It can be seen that through the above optional embodiments, the audio type parameters of the audio data can be determined based on the trained classifier algorithm, and then the target feature extraction model can be screened out to extract the audio feature parameters corresponding to the audio data, so as to facilitate subsequent audio mixing predictions, and assist in achieving more efficient and accurate mixing of audio data based on audio features and user attention, so as to effectively improve the sound effect, and thus provide users with a better sound experience.

[0141] As an optional embodiment, the extraction module determines a target feature extraction model from multiple candidate feature extraction models based on the audio type parameter in a specific manner, including:

[0142] For each candidate feature extraction model, calculate the similarity between the audio type annotation and the audio type parameters in the training data set corresponding to the candidate feature extraction model;

[0143] The candidate feature extraction model with the highest similarity is determined as the target feature extraction model; wherein the candidate feature extraction model is trained by the following steps:

[0144] Training a preset fully convolutional neural network based on a training data set including a plurality of training audio data and corresponding audio feature parameter annotations until convergence;

[0145] The parameters of the feature extraction layer in the fully convolutional neural network are fixed to obtain a candidate feature extraction model.

[0146] It can be seen that through the above optional embodiments, a suitable feature extraction model can be screened out based on the similarity between the audio type annotations and the audio type parameters in the training data set corresponding to the candidate feature extraction model, so as to facilitate subsequent audio mixing predictions, and assist in achieving more efficient and accurate mixing of audio data based on audio features and user attention, so as to effectively improve the sound effect and thus provide users with a better sound experience.

[0147] As an optional embodiment, the determination module determines the specific manner in which the user attention corresponding to each audio data is determined based on historical user feedback records, including:

[0148] Acquire historical user feedback records including feedback records of multiple users on different types of audio data;

[0149] For each audio data, based on the audio feature parameters corresponding to the audio data, multiple similar audio data are screened from all evaluated audio data in the historical user feedback records; optionally, the feature similarity between the audio features of the similar audio data and the audio feature parameters is greater than a first similarity threshold;

[0150] Calculate the weighted sum average of the user feedback parameters corresponding to each similar audio data to obtain the user attention corresponding to the audio data.

[0151] It can be seen that through the above-mentioned optional embodiments, the user attention corresponding to each audio data can be determined based on the user feedback parameters corresponding to the audio data similar to each audio data in the evaluated audio data in the historical user feedback records, so as to facilitate subsequent audio mixing prediction, and assist in achieving more efficient and accurate mixing of audio data based on audio features and user attention, so as to effectively improve the sound effect and thus provide users with a better sound experience.

[0152] As an optional embodiment, the user feedback parameter is the user's attention duration on similar audio data, and the attention duration is calculated by the following steps:

[0153] When playing mixed audio including similar audio data and other audio data to the user, obtaining a user face image;

[0154] Based on the face attention recognition model, continuously identify the attention parameters corresponding to the user's face image;

[0155] First, continuously reduce the proportion of similar audio data in the mixed audio until the attention parameter is lower than the preset first parameter threshold, and calculate the reduction duration;

[0156] Then, the proportion of similar audio data in the mixed audio is continuously increased until the attention parameter is higher than a preset second parameter threshold, and the increase duration is calculated; optionally, the second parameter threshold is higher than the first parameter threshold;

[0157] Calculate the ratio of the reduced duration to the increased duration to obtain the attention duration corresponding to similar audio data.

[0158] It can be seen that through the above optional embodiments, the content and calculation steps of the user feedback parameters are clarified, which can accurately characterize the user's attention duration on the audio data, so as to facilitate subsequent audio mixing prediction, and assist in achieving more efficient and accurate mixing of audio data based on audio features and user attention, so as to effectively improve the sound effect and thus provide users with a better sound experience.

[0159] As an optional embodiment, when calculating user attention, the weighted calculation weight corresponding to the user feedback parameter corresponding to each similar audio data is the product of the first weight and the second weight; the first weight is inversely proportional to the audio duration and / or average audio volume of the similar audio data; the second weight is directly proportional to the feature similarity corresponding to the similar audio data.

[0160] It can be seen that through the above optional embodiments, the weight rules corresponding to different similar audio data in the user attention calculation are defined, so that the calculated user attention accurately represents the length of time the user pays attention to the audio data, so as to facilitate subsequent audio mixing prediction, and assist in achieving more efficient and accurate mixing of audio data based on audio features and user attention, so as to effectively improve the sound effect, and thus give users a better sound experience.

[0161] As an optional embodiment, the prediction module determines the specific manner of the mixing strategy corresponding to the multiple audio data based on the prediction algorithm according to the audio feature parameters and the user's attention, including:

[0162] The objective function is set to be that the user satisfaction corresponding to the hybrid strategy is greater than a satisfaction threshold; optionally, the user satisfaction is obtained by inputting the hybrid parameters corresponding to all audio data in the hybrid strategy and the user attention into a trained user satisfaction prediction model; the hybrid parameters include the hybrid volume weight, the audio mixing method, and the hybrid audio portion;

[0163] Setting restrictions includes:

[0164] The predicted average audio volume corresponding to the mixed volume weight of any audio data in the mixing strategy is greater than the volume threshold; optionally, the predicted average audio volume is obtained by inputting any audio data and the corresponding mixed volume weight into a volume prediction model;

[0165] The parameter similarity between the mixing parameters corresponding to any two audio data in the mixing strategy is greater than a second similarity threshold;

[0166] Based on the dynamic programming algorithm, multiple audio data are iteratively calculated according to the objective function and the constraints to obtain the hybrid strategy corresponding to the multiple audio data.

[0167] It can be seen that through the above optional embodiments, it is possible to perform iterative calculations based on preset objective functions and limiting conditions through a dynamic programming algorithm to obtain mixing strategies corresponding to multiple audio data, so as to achieve more efficient and accurate mixing of audio data based on audio features and user attention, so as to effectively improve the sound effect and provide users with a better sound experience.

[0168] Example 3

[0169] See also Figure 3 , Figure 3 This is another audio data mixing system based on a prediction algorithm disclosed in an embodiment of the present invention. Figure 3 The described audio data mixing system based on the prediction algorithm is applied to a data processing system / data processing device / data processing server (wherein the server includes a local processing server or a cloud processing server). Figure 3 As shown, the audio data mixing system based on the prediction algorithm may include:

[0170] A memory 301 storing executable program code;

[0171] a processor 302 coupled to the memory 301;

[0172] The processor 302 calls the executable program code stored in the memory 301 to execute the steps of the audio data mixing method based on the prediction algorithm described in the first embodiment.

[0173] Example 4

[0174] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program for electronic data exchange, wherein the computer program enables a computer to execute the steps of the audio data mixing method based on a prediction algorithm described in the first embodiment.

[0175] Example 5

[0176] An embodiment of the present invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute the steps of the audio data mixing method based on the prediction algorithm described in the first embodiment.

[0177] The foregoing description of specific embodiments of the present disclosure is intended to illustrate a method for performing a multi-tasking process. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0178] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0179] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0180] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0181] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0182] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0183] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0184] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0185] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0186] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0187] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0188] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0189] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0190] Finally, it should be noted that the audio data mixing method and system based on the prediction algorithm disclosed in the embodiment of the present invention are only preferred embodiments of the present invention, which are only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An audio data mixing method based on a prediction algorithm, characterized in that: The method comprises: Obtain multiple audio data to be mixed; Extracting audio feature parameters corresponding to each of the audio data based on a feature extraction model; Determining user attention corresponding to each of the audio data based on historical user feedback records; Determining, based on the audio feature parameters and the user's attention and a prediction algorithm, a mixing strategy corresponding to the plurality of audio data, including: The objective function is set such that the user satisfaction corresponding to the mixing strategy is greater than a satisfaction threshold; the user satisfaction is obtained by inputting mixing parameters corresponding to all the audio data in the mixing strategy and the user attention into a trained user satisfaction prediction model; the mixing parameters include a mixing volume weight, an audio mixing mode, and a mixed audio portion; Setting restrictions includes: The predicted average audio volume corresponding to the mixed volume weight of any audio data in the mixing strategy is greater than a volume threshold; the predicted average audio volume is obtained by inputting any audio data and the corresponding mixed volume weight into a volume prediction model; The parameter similarity between the mixing parameters corresponding to any two audio data in the mixing strategy is greater than a second similarity threshold; Based on a dynamic programming algorithm, the multiple audio data are iteratively calculated according to the objective function and the limiting conditions to obtain a mixing strategy corresponding to the multiple audio data; the mixing strategy is used to indicate an operation of mixing the multiple audio data.

2. The audio data mixing method based on the prediction algorithm according to claim 1, characterized in that The audio feature parameters include one or more of audio duration, audio volume change, audio instrument type, and audio text content.

3. The audio data mixing method based on the prediction algorithm according to claim 1, characterized in that The extracting audio feature parameters corresponding to each of the audio data based on the feature extraction model includes: For each of the audio data, input the audio data into a trained classifier algorithm to determine an audio type parameter corresponding to the audio data; the audio type parameter includes one or more of a vocal type, a single instrument type, a mixed instrument type, and a musical genre type; the classifier algorithm is trained using a training data set including a plurality of training audio data and corresponding audio type parameter annotations; Determining a target feature extraction model from a plurality of candidate feature extraction models based on the audio type parameter; The audio data is input into the target feature extraction model to obtain audio feature parameters corresponding to the audio data.

4. The audio data mixing method based on the prediction algorithm according to claim 3, characterized in that: The step of determining a target feature extraction model from a plurality of candidate feature extraction models based on the audio type parameter includes: For each candidate feature extraction model, calculating the similarity between the audio type annotation in the training data set corresponding to the candidate feature extraction model and the audio type parameter; The candidate feature extraction model with the highest similarity is determined as the target feature extraction model; wherein the candidate feature extraction model is trained by the following steps: Training a preset fully convolutional neural network based on a training data set including a plurality of training audio data and corresponding audio feature parameter annotations until convergence; The parameters of the feature extraction layer in the fully convolutional neural network are fixed to obtain a candidate feature extraction model.

5. The audio data mixing method based on the prediction algorithm according to claim 1, characterized in that: The determining of the user attention corresponding to each audio data based on the historical user feedback record includes: Acquire historical user feedback records including feedback records of multiple users on different types of audio data; For each of the audio data, based on the audio feature parameters corresponding to the audio data, a plurality of similar audio data are screened from all evaluated audio data in the historical user feedback records; wherein a feature similarity between the audio features of the similar audio data and the audio feature parameters is greater than a first similarity threshold; Calculate the weighted average of the user feedback parameters corresponding to each of the similar audio data to obtain the user attention corresponding to the audio data.

6. The audio data mixing method based on the prediction algorithm according to claim 5, characterized in that: The user feedback parameter is the user's attention duration on the similar audio data, and the attention duration is calculated by the following steps: When playing mixed audio including the similar audio data and other audio data to the user, obtaining a user face image; Based on the face attention recognition model, continuously identifying the attention parameters corresponding to the user's face image; First, continuously reduce the proportion of the similar audio data in the mixed audio until the attention parameter is lower than a preset first parameter threshold, and calculate the reduction duration; Then, continuously increasing the proportion of the similar audio data in the mixed audio until the attention parameter is higher than a preset second parameter threshold, and calculating the increase duration; The second parameter threshold is higher than the first parameter threshold; The ratio of the reduced duration to the increased duration is calculated to obtain the attention duration corresponding to the similar audio data.

7. The audio data mixing method based on the prediction algorithm according to claim 5, characterized in that: When calculating the user attention, the weighted calculation weight corresponding to the user feedback parameter corresponding to each similar audio data is the product of the first weight and the second weight; the first weight is inversely proportional to the audio duration and / or average audio volume of the similar audio data; the second weight is directly proportional to the feature similarity corresponding to the similar audio data.

8. An audio data mixing system based on a prediction algorithm, characterized in that: The system comprises: An acquisition module, used for acquiring multiple audio data to be mixed; An extraction module, configured to extract audio feature parameters corresponding to each of the audio data based on a feature extraction model; A determination module, configured to determine the user attention corresponding to each of the audio data based on historical user feedback records; A prediction module is configured to determine, based on a prediction algorithm and the audio feature parameters and the user's attention, a mixing strategy corresponding to the plurality of audio data, including: The objective function is set such that the user satisfaction corresponding to the mixing strategy is greater than a satisfaction threshold; the user satisfaction is obtained by inputting mixing parameters corresponding to all the audio data in the mixing strategy and the user attention into a trained user satisfaction prediction model; the mixing parameters include a mixing volume weight, an audio mixing mode, and a mixed audio portion; Setting restrictions includes: The predicted average audio volume corresponding to the mixed volume weight of any audio data in the mixing strategy is greater than a volume threshold; the predicted average audio volume is obtained by inputting any audio data and the corresponding mixed volume weight into a volume prediction model; The parameter similarity between the mixing parameters corresponding to any two audio data in the mixing strategy is greater than a second similarity threshold; Based on a dynamic programming algorithm, the multiple audio data are iteratively calculated according to the objective function and the limiting conditions to obtain a mixing strategy corresponding to the multiple audio data; the mixing strategy is used to indicate an operation of mixing the multiple audio data.

9. An audio data mixing system based on a prediction algorithm, characterized in that: The system comprises: a memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the audio data mixing method based on the prediction algorithm according to any one of claims 1 to 7.

Images