Audio processing method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to an audio processing method, device, electronic equipment and storage medium. The method comprises: obtaining a target audio segment to be encoded and a target audio quality indicator; the target audio quality indicator is an audio quality indicator indicated for the target audio segment; performing audio quality indicator prediction processing on the target audio segment based on an audio quality prediction model to obtain audio quality indicators corresponding to the target audio segment under a plurality of preset code rates respectively; constructing a mapping relationship between code rate and audio quality indicator according to the plurality of preset code rates and the audio quality indicators corresponding to the plurality of preset code rates respectively; determining a target code rate corresponding to the target audio quality indicator based on the mapping relationship, and the target code rate is used to encode the target audio segment. According to the technical scheme provided by the present disclosure, the encoding efficiency and flexibility of the audio can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an audio processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Audio encoding refers to the process of compressing audio signals using encoding techniques during audio transmission and storage. Related technologies often involve manually setting the bitrate for audio encoding, but this can only be done based on a few features and limited strategies, making it difficult to cover a wide range of audio content scenarios. Furthermore, different application scenarios have different sound quality goals, and setting different processing strategies for different sound quality goals is cumbersome and time-consuming. Summary of the Invention

[0003] This disclosure provides an audio processing method, apparatus, electronic device, and storage medium. The technical solution of this disclosure is as follows:

[0004] According to a first aspect of the present disclosure, an audio processing method is provided, comprising:

[0005] Obtain the target audio segment to be encoded and the target audio quality index; the target audio quality index is the audio quality index indicated for the target audio segment;

[0006] Based on the audio quality prediction model, the target audio segment is subjected to audio quality index prediction processing to obtain the audio quality index corresponding to the target audio segment under various preset bitrates.

[0007] Based on the various preset bitrates and the corresponding audio quality indicators under the various preset bitrates, a mapping relationship between bitrate and audio quality indicators is constructed;

[0008] Based on the mapping relationship, the target bitrate corresponding to the target audio quality index is determined, and the target bitrate is used to encode the target audio segment.

[0009] In one possible implementation, the audio quality prediction process based on the audio quality prediction model is used to predict the audio quality index of the target audio segment to obtain the audio quality index of the target audio segment at various preset bitrates, including:

[0010] The target audio segment is subjected to feature extraction processing to obtain audio features;

[0011] The audio features are input into the audio quality prediction model for audio quality index prediction processing to obtain the audio quality index corresponding to the target audio segment under the various preset bitrates.

[0012] In one possible implementation, the feature extraction process for the target audio segment to obtain audio features includes:

[0013] The target audio segment is subjected to frequency domain transformation processing to obtain the spectral data of multiple audio frames;

[0014] The audio features are obtained based on the spectrum data.

[0015] In one possible implementation, obtaining the audio features based on the spectral data includes:

[0016] Extract frame features from the spectral data of each audio frame;

[0017] The frame features are statistically processed to obtain the feature statistics of the multiple audio frames;

[0018] The audio features are obtained based on the aforementioned feature statistics.

[0019] In one possible implementation, the mapping relationship is a bitrate-to-audio quality curve; the step of constructing the mapping relationship between bitrate and audio quality indicators based on the multiple preset bitrates and the corresponding audio quality indicators under the multiple preset bitrates includes:

[0020] Construct a two-dimensional coordinate system, wherein the coordinate axes of the two-dimensional coordinate system correspond to the sound quality index and the bit rate, respectively;

[0021] Mark the first set of coordinate points consisting of each preset bitrate and its corresponding sound quality index in the two-dimensional coordinate system;

[0022] Between adjacent preset bitrates, a linear interpolation method is used to fit the bitrate and sound quality indicators, resulting in a second set of coordinate points between adjacent preset bitrates.

[0023] The bitrate audio quality curve is obtained based on the first set of coordinate points and the second set of coordinate points.

[0024] In one possible implementation, the method further includes:

[0025] Acquire sample audio segments and corresponding annotation data, wherein the annotation data characterizes the reference sound quality index of the sample audio segments under the various preset bitrates;

[0026] The sample audio segments are subjected to feature extraction processing to obtain sample audio features;

[0027] The sample audio features are input into a preset neural network for audio quality index prediction processing to obtain the predicted audio quality index corresponding to the sample audio segment under the various preset bitrates.

[0028] Based on the labeled data and the predicted sound quality indicators, the loss information is determined;

[0029] The preset neural network is trained based on the loss information until the training iteration conditions are met, thereby obtaining the sound quality prediction model.

[0030] In one possible implementation, obtaining the sample audio segment and its corresponding annotation data includes:

[0031] Obtain historical audio segments as sample audio segments;

[0032] The sample audio segment is encoded using the multiple preset bitrates to obtain multiple encoded audio segments corresponding to the sample audio segment, and each encoded audio segment corresponds to a preset bitrate.

[0033] The sample audio segment is compared and analyzed with the corresponding multiple coded audio segments to obtain the reference audio quality index corresponding to the sample audio segment, which is used as the annotation data corresponding to the sample audio segment.

[0034] In one possible implementation, obtaining the target audio segment to be encoded includes:

[0035] Obtain the target audio to be encoded;

[0036] The target audio is segmented to obtain multiple audio segments;

[0037] Choose any one of the multiple audio segments as the target audio segment.

[0038] In one possible implementation, obtaining the target sound quality index includes:

[0039] Obtain the content category information and target bandwidth information of the target audio segment;

[0040] The target audio quality index is determined based on the content category information and / or target bandwidth information.

[0041] According to a second aspect of the present disclosure, an audio processing apparatus is provided, comprising:

[0042] The first acquisition module is configured to acquire the target audio segment to be encoded and the target audio quality index; the target audio quality index is the audio quality index indicated for the target audio segment;

[0043] The audio quality prediction module is configured to perform audio quality index prediction processing on the target audio segment based on the audio quality prediction model, and obtain the audio quality index corresponding to the target audio segment under various preset bitrates.

[0044] The mapping relationship construction module is configured to construct a mapping relationship between bitrate and sound quality index based on the multiple preset bitrates and the corresponding sound quality indexes under the multiple preset bitrates.

[0045] The bitrate determination module is configured to determine the target bitrate corresponding to the target audio quality index based on the mapping relationship, and the target bitrate is used to encode the target audio segment.

[0046] In one possible implementation, the sound quality prediction module includes:

[0047] The feature extraction unit is configured to perform feature extraction processing on the target audio segment to obtain audio features;

[0048] The audio quality prediction unit is configured to perform audio quality index prediction processing by inputting the audio features into the audio quality prediction model, and obtain the audio quality index corresponding to the target audio segment under the various preset bitrates.

[0049] In one possible implementation, the feature extraction unit includes:

[0050] The frequency domain transformation subunit is configured to perform frequency domain transformation processing on the target audio segment to obtain spectral data of multiple audio frames;

[0051] An audio feature acquisition subunit is configured to acquire the audio features based on the spectrum data.

[0052] In one possible implementation, the audio feature acquisition subunit includes:

[0053] The frame feature extraction subunit is configured to extract frame features from the spectral data of each audio frame;

[0054] The feature statistics subunit is configured to perform statistical processing on the frame features to obtain feature statistics information of the multiple audio frames;

[0055] An audio feature determination subunit is configured to perform an action based on the feature statistics to obtain the audio features.

[0056] In one possible implementation, the mapping relationship is a bitrate-to-audio quality curve; the mapping relationship construction module includes:

[0057] The coordinate system construction unit is configured to construct a two-dimensional coordinate system, wherein the coordinate axes of the two-dimensional coordinate system correspond to the sound quality index and the bit rate, respectively.

[0058] The marking unit is configured to mark a first set of coordinate points consisting of each preset bitrate and its corresponding audio quality index in the two-dimensional coordinate system;

[0059] The fitting unit is configured to perform a linear interpolation process between adjacent preset bitrates to fit the bitrate and sound quality indicators, thereby obtaining a second set of coordinate points between adjacent preset bitrates.

[0060] The curve construction unit is configured to perform operations based on the first set of coordinate points and the second set of coordinate points to obtain the bitrate-quality curve corresponding to the target audio segment.

[0061] In one possible implementation, the device further includes:

[0062] The second acquisition module is configured to acquire sample audio segments and corresponding annotation data, wherein the annotation data characterizes the reference sound quality indexes corresponding to the sample audio segments under the various preset bitrates.

[0063] The feature extraction module is configured to perform feature extraction processing on the sample audio segment to obtain sample audio features;

[0064] The audio quality prediction module is configured to input the sample audio features into a preset neural network to perform audio quality index prediction processing, and obtain the predicted audio quality indexes corresponding to the sample audio segment under the various preset bitrates.

[0065] The loss determination module is configured to determine loss information based on the labeled data and the predicted sound quality index;

[0066] The training module is configured to train the preset neural network based on the loss information until the training iteration conditions are met, thereby obtaining the sound quality prediction model.

[0067] In one possible implementation, the second acquisition module includes:

[0068] The sample audio segment acquisition unit is configured to acquire historical audio segments as sample audio segments;

[0069] The encoding unit is configured to perform encoding processing on the sample audio segment using the multiple preset bitrates to obtain multiple encoded audio segments corresponding to the sample audio segment, each encoded audio segment corresponding to a preset bitrate;

[0070] The audio quality analysis unit is configured to perform audio quality comparison analysis between the sample audio segment and the corresponding multiple coded audio segments to obtain a reference audio quality index corresponding to the sample audio segment, which is used as the annotation data corresponding to the sample audio segment.

[0071] In one possible implementation, the first acquisition module includes:

[0072] The target audio acquisition unit is configured to acquire the target audio to be encoded.

[0073] The segmentation processing unit is configured to perform segmentation processing on the target audio to obtain multiple audio segments;

[0074] The target audio segment acquisition unit is configured to acquire any one of the plurality of audio segments as the target audio segment.

[0075] In one possible implementation, the first acquisition module further includes:

[0076] The content category and bandwidth acquisition unit is configured to acquire the content category information and target bandwidth information of the target audio segment.

[0077] The target audio quality determination unit is configured to determine the target audio quality index based on the content category information and / or target bandwidth information.

[0078] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the method as described in any one of the first aspects above.

[0079] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided such that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform any of the methods described in the first aspect of the present disclosure.

[0080] According to a fifth aspect of the present disclosure, a computer program product is provided, including computer instructions that, when executed by a processor, cause a computer to perform the method described in any one of the first aspects of the present disclosure.

[0081] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0082] By establishing a mapping relationship between bitrate and audio quality indicators, the target bitrate corresponding to any specified target audio quality indicator can be quickly determined based on this mapping relationship. This improves the efficiency and flexibility of determining the target bitrate for audio segment encoding, adaptively obtaining the target bitrate required for encoding the target audio segment. In other words, the bitrate corresponding to the audio quality indicator requirement can be obtained quickly and accurately. Furthermore, for the target audio segment, with the mapping relationship between bitrate and audio quality indicators, the target audio quality indicator can be quickly and dynamically adjusted to achieve bitrate adjustment for different audio quality indicators. This allows for flexible application to bandwidth requirements, high encoding efficiency, and a wide range of applicable scenarios.

[0083] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0084] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0085] Figure 1 This is a schematic diagram illustrating an application environment according to an exemplary embodiment.

[0086] Figure 2 This is a flowchart illustrating an audio processing method according to an exemplary embodiment.

[0087] Figure 3 This is a flowchart illustrating an exemplary embodiment of a method for predicting the audio quality index of a target audio segment based on an audio quality prediction model, thereby obtaining the audio quality index corresponding to the target audio segment under various preset bitrates.

[0088] Figure 4 This is a schematic diagram illustrating a bitrate audio quality curve according to an exemplary embodiment.

[0089] Figure 5 This is a schematic diagram illustrating a process for obtaining a reference sound quality index according to an exemplary embodiment.

[0090] Figure 6 This is a schematic diagram illustrating sound quality prediction according to an exemplary embodiment.

[0091] Figure 7 This is a block diagram of an audio processing apparatus according to an exemplary embodiment.

[0092] Figure 8 This is a block diagram illustrating an electronic device for audio processing according to an exemplary embodiment.

[0093] Figure 9 This is a block diagram illustrating another electronic device for audio processing according to an exemplary embodiment. Detailed Implementation

[0094] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0095] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0096] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. AI software technology mainly includes computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0097] In recent years, with the research and progress of artificial intelligence technology, it has been widely applied in many fields. The solutions provided in this application involve technologies such as machine learning / deep learning, which are specifically illustrated through the following embodiments:

[0098] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application environment according to an exemplary embodiment, such as... Figure 1 As shown, the application environment may include server 01 and terminal 02.

[0099] In an optional embodiment, server 01 can be used for audio processing. Specifically, server 01 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0100] In an optional embodiment, terminal 02 can be used to provide target audio to be encoded, acquire audio after encoding the target audio segment using a target bitrate for transmission, etc. Specifically, terminal 02 can be, but is not limited to, electronic devices such as smartphones, desktop computers, tablets, laptops, smart speakers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, and smart wearable devices. Optionally, the operating system running on the electronic device can be, but is not limited to, Android, iOS, Linux, Windows, etc.

[0101] In addition, it should be noted that, Figure 1 The illustration shows only one application environment of the audio processing method provided in this disclosure. Alternatively, the audio processing can be performed by the terminal, or the server can train a sound quality prediction model and send the sound quality prediction model to the terminal for the terminal to perform the audio processing.

[0102] In the embodiments described in this specification, the server 01 and the terminal 02 can be directly or indirectly connected through wired or wireless communication, and this application does not impose any restrictions on this.

[0103] It should be noted that the following diagram illustrates one possible sequence of steps, and it is not strictly required to follow this order. Some steps can be performed in parallel without interdependence. The user information (including but not limited to user device information, user personal information, user behavior information, etc.) and data (including but not limited to data used for display, training data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0104] Figure 2 This is a flowchart illustrating an audio processing method according to an exemplary embodiment. Figure 2 As shown, the steps may include the following.

[0105] In step S201, the target audio segment to be encoded and the target sound quality index are obtained.

[0106] In the embodiments of this specification, the target audio segment can refer to any segment of an audio file, which may include audio content such as speech, music, and noise. The target audio quality index can be the audio quality index indicated for the target audio segment, that is, the audio quality index specified for the target audio segment. The audio quality index can refer to the speech quality of the audio. As an example, the target audio quality index can be specified by the user on the terminal side, or the target audio quality index of the target audio segment can be automatically specified; this disclosure does not limit this.

[0107] In practical applications, transmission bandwidth and storage resources are dynamically changing. Encoding the entire audio file using the same bitrate cannot effectively utilize real-time bandwidth or storage resources. Therefore, the target audio to be encoded can be segmented, allowing for differentiated encoding of different audio segments within an audio file. This adapts to real-time bandwidth or storage resources, fully utilizing their capabilities and balancing sound quality and encoding efficiency. Furthermore, it has been found that different content categories in audio require different bitrates to achieve the same sound quality. For example, to achieve a sound quality target that is virtually lossless to the human ear, music requires a higher bitrate than speech, and female voices require a higher bitrate than male voices. Therefore, to improve encoding efficiency, different content categories in the audio can be encoded using different bitrates. Here, content category (or content category information) can refer to the category of content in the audio, such as speech, music, and noise, or it can be a more subdivided category, such as speech being subdivided into male and female voices, and music being subdivided into classical and pop music. This disclosure does not limit this specific category. Optionally, the content category information can be determined based on a neural network model. For example, the target audio segment can be input into the neural network model to obtain the predicted content category information. The neural network model can be trained on a pre-defined machine learning model using a large number of sample video segments labeled with content category information. Alternatively, it can be determined by analyzing the spectral characteristics of the target audio segment; this disclosure does not limit this approach.

[0108] Based on the relationship between transmission bandwidth, storage resources, content category, and audio quality and bitrate, in order to balance encoding efficiency and audio quality indicators, the target audio can be segmented for encoding of each segment. Specifically, the target audio to be encoded can be obtained; thus, the target audio can be segmented to obtain multiple audio segments; any audio segment from these multiple audio segments can be selected as the target audio segment. The segmentation can be based on a preset duration, which can be fixed or variable, and this disclosure does not limit this. For example, the target audio can be segmented based on a preset duration to obtain multiple audio segments, each of which can have a preset duration. Optionally, when the preset duration is variable, the start and end times of the preset duration can be determined based on the amplitude of the audio. For example, if the change in audio amplitude is greater than a threshold, the end of the previous preset duration (end time of the previous preset duration) and the start of the next preset duration (start time of the next preset duration) can be determined. This allows the target audio to be segmented, and the duration of the segmented audio segments is not fixed. Optionally, the target audio can be segmented based on content category, ensuring that the content category information within a single target audio segment is the same, such as a speech segment or a music segment. By segmenting the target audio into segments, different bitrates can be used to encode different segments of the same audio, improving encoding flexibility while maintaining sound quality. Furthermore, segmentation based on content category allows audio segments of the same content category to be encoded using the same bitrate. Although the bitrates of different segments differ, high sound quality can still be achieved, avoiding the use of a high bitrate for the entire target audio in pursuit of sound quality, thus optimizing bitrate and saving resources.

[0109] In specific implementations, the target audio to be encoded can be uploaded by the terminal or historical audio from an audio platform; this disclosure does not limit this.

[0110] For automatically specifying the target audio quality index for a target audio segment, in one optional approach, the content category information and target bandwidth information of the target audio segment can be obtained; thus, the target audio quality index can be determined based on the content category information and / or the target bandwidth information. The target bandwidth information can refer to the current bandwidth information or it can be preset bandwidth information used to transmit the target audio. Specifically, based on different content categories in the audio, the required encoding bitrate to achieve the same audio quality index varies. The target audio quality index can be automatically determined based on the content category information. This disclosure does not limit this approach, as long as it can reduce the bitrate to save bandwidth. For example, music requires a higher bitrate than speech, so the target audio quality index for music can be determined to be lower than that for speech. Alternatively, the appropriate target audio quality index can be dynamically determined based on the target bandwidth information to avoid transmission stuttering. Or, the target audio quality index for the target audio segment can be determined by combining the content category information and the target bandwidth information. This disclosure does not limit the determination method, as long as it can save resources or ensure the audio quality effect after encoding recovery. By determining the target audio quality index through content category information and / or target bandwidth information, it is possible to effectively match the transmission resources and audio content categories, balancing coding efficiency and audio quality requirements.

[0111] In step S203, the target audio segment is subjected to audio quality index prediction processing based on the audio quality prediction model to obtain the audio quality index corresponding to the target audio segment under various preset bitrates.

[0112] In the embodiments of this specification, the sound quality prediction model can be obtained by pre-training a preset neural network based on sample audio segments and corresponding labeled data. The labeled data can characterize reference sound quality indicators of the sample audio segments, i.e., the training labels of the sample audio segments.

[0113] In one possible implementation, the frequency domain features of the target audio segment can be obtained based on the audio signal. These features can then be input into a sound quality prediction model to predict the sound quality index of the target audio segment, yielding the sound quality index corresponding to the target audio segment under various preset bitrates. This disclosure does not limit the number of preset bitrates; for example, it can be 10. As an example, the various preset bitrates R can be represented as R = [r1, r2, ..., rN]. When N = 10, R = [24, 32, 40, 48, 56, 64, 72, 80, 88, 96]. Thus, after processing by the sound quality prediction model, the sound quality index corresponding to the target audio segment under various preset bitrates can refer to the respective sound quality indexes under r1, r2, ..., r10.

[0114] In another possible implementation, such as Figure 3 As shown, step S203 may include the following steps:

[0115] In step S301, feature extraction processing is performed on the target audio segment to obtain audio features.

[0116] In the embodiments of this specification, the prediction of the sound quality index may include two parts: feature extraction and prediction of the sound quality index based on a sound quality prediction model. Based on this, feature extraction processing can be performed on the target audio segment to obtain audio features. In a possible implementation manner, features such as the audio effective bandwidth, audio amplitude, and audio richness can be extracted from the target audio segment, so that multiple features can be spliced together to obtain audio features. The audio features may be a D-dimensional feature vector, where D may be a positive integer. For example, for a 3-second audio segment, a 40-dimensional audio feature vector can be extracted. The present disclosure does not limit this.

[0117] In another possible implementation manner, frequency domain transformation processing is performed on the target audio segment to obtain spectral data of multiple audio frames; for example, this frequency domain transformation processing can be performed based on the Fourier transform. Thus, audio features can be obtained according to the spectral data. In an optional embodiment, the spectral data of multiple audios can be spliced to obtain the audio features of the target audio segment.

[0118] In another optional embodiment, frame features can be extracted from the spectral data of each audio frame; statistical processing is performed on the frame features to obtain feature statistical information of multiple audio frames; and audio features can be obtained based on the feature statistical information. For example, the Fourier transform can be performed on the target audio segment to obtain spectral data corresponding to multiple audio frames. Further, multiple frame features can be extracted from each audio frame, and then the statistical mean and variance of the frame features are used as the audio features of the entire target audio segment. As an example, the frame features may include MFCC (Mel Frequency Cepstral Coefficients), sub-band energy ratio (dividing the spectrum into sub-bands, the ratio of the energy of the sub-band to the total energy of the spectrum), etc.

[0119] In an example, based on the Short-Time Fourier Transform (STFT), frequency domain transformation processing is performed on the target audio segment to obtain spectral data of multiple audio frames. Assume that the target audio segment x with a preset duration T is represented as x(t), where t represents time, 0 < t ≤ T. Then, after STFT, the spectral data of x can be represented by the following formula:

[0120] X(n,k) = STFT(x(t))

[0121] Where n is the frame index, 0 < n ≤ N, and N is the total number of frames; k is the frequency band point, 0 < k ≤ K, and K is the total number of frequency bands; STFT(·) represents performing short-time Fourier transform processing.

[0122] By performing a frequency-domain transformation on the target audio segment, the corresponding spectral data is obtained, thereby obtaining the audio features of the target audio segment, enabling the audio features to effectively represent the audio content of the target audio segment.

[0123] In step S303, the audio features are input into the sound quality prediction model for sound quality index prediction processing, and the sound quality indexes corresponding to the target audio segment under multiple preset bitrates are obtained.

[0124] In the embodiments of this specification, the audio features can be input into the sound quality prediction model for sound quality index prediction processing to obtain the sound quality indexes corresponding to the target audio segment under multiple preset bitrates. The sound quality index can be an N-dimensional feature vector, and N can be the same as or different from the above D. This disclosure does not make any limitation in this regard. By combining feature extraction and the sound quality prediction model to obtain the sound quality indexes corresponding to the target audio segment under multiple preset bitrates, the accuracy and efficiency can be improved.

[0125] In step S205, according to the multiple preset bitrates and the sound quality indexes corresponding to them respectively, a mapping relationship between the bitrate and the sound quality index is constructed.

[0126] In the embodiments of this specification, after obtaining the sound quality indexes corresponding to the target audio segment under multiple preset bitrates, a mapping relationship between the bitrate and the sound quality index can be constructed according to the multiple preset bitrates and the sound quality indexes corresponding to them. The bitrates in this mapping relationship can include the above multiple preset bitrates. As an example, the bitrates in this mapping relationship can be continuous. This disclosure does not make any limitation in this regard.

[0127] In a possible implementation manner, the mapping relationship between the bitrate and the sound quality index can be a bitrate-sound quality curve, that is, the bitrate-sound quality curve corresponding to the target audio segment. Based on this, in an example, a two-dimensional coordinate system can be constructed. The coordinate axes of this two-dimensional coordinate system correspond to the sound quality index and the bitrate respectively. For example Figure 4 as shown, the abscissa can be the bitrate (the unit can be kpbs, kilobits per second), and the ordinate can be the sound quality index. And the set of first coordinate points composed of each preset bitrate and the corresponding sound quality index can be marked in the two-dimensional coordinate system; and between adjacent preset bitrates, a linear interpolation method can be used for fitting processing of the bitrate and the sound quality index to obtain the set of second coordinate points between adjacent preset bitrates. Further, based on the set of first coordinate points and the set of second coordinate points, the bitrate-sound quality curve corresponding to the target audio segment can be obtained. Among them, Figure 4The bitrate-to-audio quality curves shown are for N=10 and R=[24,32,40,48,56,64,72,80,88,96]. The asterisks (*) correspond to the first set of coordinate points, i.e., the coordinate points corresponding to multiple preset bitrates. By constructing the bitrate-to-audio quality curve through fitting, the correspondence between continuous bitrate values ​​and continuous audio quality indicators can be obtained. Therefore, the bitrate corresponding to any audio quality indicator can be determined based on this bitrate-to-audio quality curve.

[0128] In step S207, based on the above mapping relationship, the target bitrate corresponding to the target audio quality index is determined, and the target bitrate is used to encode the target audio segment.

[0129] In practical applications, after obtaining the above mapping relationship, for any target audio quality index specified for the target audio segment, the corresponding target bitrate can be obtained based on the mapping relationship.

[0130] When the mapping relationship is a bitrate audio quality curve, such as Figure 4 As shown, it can be based on the target sound quality index on the vertical axis (such as...). Figure 4 The target audio quality is directly obtained by mapping the target bitrate (e.g., on the horizontal axis) to the corresponding target bitrate. Figure 4 The predicted bitrate in the data makes the determination of the target bitrate fast and accurate.

[0131] Furthermore, the target bitrate can be used to encode the target audio segment to obtain the encoded audio segment corresponding to the target audio segment. Thus, when all audio segments in the target audio are assigned a corresponding target audio quality index, the target bitrate corresponding to each audio segment in the target audio can be obtained. Encoding all audio segments in the target audio yields all encoded audio segments of the target audio, completing the encoding of a complete target audio segment. Optionally, these encoded audio segments can be transmitted to achieve interaction with the target audio; this disclosure does not limit this. Based on this, as an optional implementation, the target audio quality index can be adjusted based on the transmission performance obtained through interaction and / or the audio quality index information fed back by the other party. For example, if the audio quality index information is lower than the audio quality index threshold set by the other party, the target audio quality index can be increased; or if the transmission performance indicates transmission stuttering, the target audio quality index can be decreased, thereby reducing the required transmission bandwidth and alleviating the transmission stuttering phenomenon. Alternatively, the target audio quality index can be adjusted comprehensively by combining transmission performance and audio quality index information.

[0132] By establishing a mapping relationship between bitrate and audio quality indicators, the target bitrate corresponding to any specified target audio quality indicator can be quickly determined based on this mapping relationship. This improves the efficiency and flexibility of determining the target bitrate for audio segment encoding, adaptively obtaining the target bitrate required for encoding the target audio segment. In other words, the bitrate corresponding to the audio quality indicator requirement can be obtained quickly and accurately. Furthermore, for the target audio segment, with the mapping relationship between bitrate and audio quality indicators, the target audio quality indicator can be quickly and dynamically adjusted to achieve bitrate adjustment for different audio quality indicators. This allows for flexible application to bandwidth requirements, high encoding efficiency, and a wide range of applicable scenarios.

[0133] The training process for the aforementioned sound quality prediction model can be achieved using the following steps, which may include:

[0134] Obtain sample audio segments and their corresponding annotation data. The annotation data can characterize the reference sound quality indicators of the sample audio segments under the above-mentioned multiple preset bitrates.

[0135] The sample audio segments are processed to extract features, and the sample audio features are obtained; for details, please refer to step S301 above, which will not be repeated here.

[0136] The sample audio features are input into a preset neural network for audio quality index prediction, resulting in predicted audio quality indices for the sample audio segments at various preset bitrates; for example... Figure 6 The S' shown can be found in step S303 above, and will not be repeated here.

[0137] Based on the labeled data and predicted sound quality indicators, loss information is determined; and a preset neural network can be trained based on the loss information until the training iteration conditions are met to obtain a sound quality prediction model. The training iteration conditions can be that the loss information is less than a loss threshold, or an iteration number threshold, or the difference in loss information between two iterations is less than a loss difference threshold, etc., which are not limited in this disclosure. Optionally, the structure of the preset neural network can be determined based on the scale of processing resources. When the scale of processing resources is less than the preset scale, a lightweight neural network can be selected as the preset neural network, such as a simple multi-layer fully connected network with a total of L layers and K nodes per layer. As an example, L=2 and K=100, which are not limited in this disclosure. When the scale of processing resources is greater than or equal to the preset scale, a more complex model structure can be selected, such as multi-layer CNN (Convolutional Neural Networks), LSTM (Long Short-Term Memory), and DNN (Deep Neural Networks) structures, which can obtain more accurate predictions of sound quality indicators.

[0138] In one example, to obtain sample audio segments and corresponding annotation data, historical audio segments can be obtained as sample audio segments; and multiple preset bitrates can be used to encode the sample audio segments to obtain multiple encoded audio segments corresponding to the sample audio segments, each encoded audio segment corresponding to a preset bitrate. For example, an audio coding algorithm (such as HE-AAC, High Efficiency Advanced Audio Coding) can be used to encode at N preset bitrates. These N preset bitrates can be R = [r1, r2, ..., rN], and the encoded audio segments output after encoding can be Y = [y1, y2, ..., yN], where y1 to yN can correspond to r1 to rN respectively. For example, N = 10, R = [24, 32, 40, 48, 56, 64, 72, 80, 88, 96], which is not limited in this disclosure.

[0139] Furthermore, the sample audio segment can be compared and analyzed with multiple corresponding coded audio segments to obtain reference audio quality indicators for the sample audio segment, which can then be used as annotation data for the sample audio segment. For example... Figure 5 As shown, based on the original input audio segment x and the encoded audio segment Y = [y1, y2, ..., yN], an objective audio quality analysis algorithm can be used to obtain the objective audio quality S = [s1, s2, ..., sN] corresponding to various preset bitrates. The objective audio quality analysis algorithm here can be PEAQ (Perceptual Evaluation of Audio Quality) recommended by the International Telecommunication Union, or PESQ (Perceptual Evaluation of Speech Quality), or a linear fusion of multiple objective audio quality analysis algorithms.

[0140] Optionally, a subjective audio quality analysis algorithm can be used to determine the reference audio quality index corresponding to the sample audio segment, so that subsequent audio quality index predictions can more closely approximate human subjective perception. Here, the subjective audio quality analysis algorithm can be based on MOS (Mean Opinion Score), and this disclosure does not limit it.

[0141] like Figure 5 As shown, after comparing and analyzing sound quality, data synthesis was achieved, which yields a large number of original audio segments x and various preset bitrates (e.g., ...). Figure 5The data pair shown is xS, representing the N types of encoded reference audio quality indicators. This data pair can be used to train the audio quality prediction model, serving as both sample audio segments and corresponding labeled data. Here, x represents a sample audio segment, and the corresponding S represents the labeled data. S can include s1 to sN, which can correspond to r1 to rN respectively. Data pairs of sample audio segments and corresponding labeled data are obtained through data synthesis. The labeled data is obtained through comparative analysis of the original audio segments and audio segments encoded with various preset bitrates, resulting in more accurate labeled data. Based on this data, the prediction accuracy of the trained audio quality prediction model can be further improved.

[0142] Figure 7 This is a block diagram illustrating an audio processing apparatus according to an exemplary embodiment. (Refer to...) Figure 7 The device may include:

[0143] The first acquisition module 701 is configured to acquire the target audio segment to be encoded and the target sound quality index; the target sound quality index is the sound quality index indicated by the target audio segment;

[0144] The audio quality prediction module 703 is configured to perform audio quality index prediction processing on the target audio segment based on the audio quality prediction model, and obtain the audio quality index corresponding to the target audio segment under various preset bitrates.

[0145] The mapping relationship construction module 705 is configured to construct a bitrate-to-sound-quality curve corresponding to the target audio segment based on the multiple preset bitrates and the corresponding sound quality indicators under the multiple preset bitrates; the bitrate-to-sound-quality curve represents the correspondence between bitrate and sound quality indicators;

[0146] The bitrate determination module 707 is configured to determine the target bitrate corresponding to the target audio quality index in the bitrate audio quality curve, wherein the target bitrate is used to encode the target audio segment.

[0147] By establishing a mapping relationship between bitrate and audio quality indicators, the target bitrate corresponding to any specified target audio quality indicator can be quickly determined based on this mapping relationship. This improves the efficiency and flexibility of determining the target bitrate for audio segment encoding, adaptively obtaining the target bitrate required for encoding the target audio segment. In other words, the bitrate corresponding to the audio quality indicator requirement can be obtained quickly and accurately. Furthermore, for the target audio segment, with the mapping relationship between bitrate and audio quality indicators, the target audio quality indicator can be quickly and dynamically adjusted to achieve bitrate adjustment for different audio quality indicators. This allows for flexible application to bandwidth requirements, high encoding efficiency, and a wide range of applicable scenarios.

[0148] In one possible implementation, the sound quality prediction module 703 described above may include:

[0149] The feature extraction unit is configured to perform feature extraction processing on the target audio segment to obtain audio features;

[0150] The audio quality prediction unit is configured to perform audio quality index prediction processing by inputting the audio features into the audio quality prediction model, and obtain the audio quality index corresponding to the target audio segment under the various preset bitrates.

[0151] In one possible implementation, the feature extraction unit described above may include:

[0152] The frequency domain transformation subunit is configured to perform frequency domain transformation processing on the target audio segment to obtain spectral data of multiple audio frames;

[0153] An audio feature acquisition subunit is configured to acquire the audio features based on the spectrum data.

[0154] In one possible implementation, the aforementioned audio feature acquisition subunit may include:

[0155] The frame feature extraction subunit is configured to extract frame features from the spectral data of each audio frame;

[0156] The feature statistics subunit is configured to perform statistical processing on the frame features to obtain feature statistics information of the multiple audio frames;

[0157] An audio feature determination subunit is configured to perform an action based on the feature statistics to obtain the audio features.

[0158] In one possible implementation, the mapping relationship is a bitrate audio quality curve; the mapping relationship construction module 705 may include:

[0159] The coordinate system construction unit is configured to construct a two-dimensional coordinate system, wherein the coordinate axes of the two-dimensional coordinate system correspond to the sound quality index and the bit rate, respectively.

[0160] The marking unit is configured to mark a first set of coordinate points consisting of each preset bitrate and its corresponding audio quality index in the two-dimensional coordinate system;

[0161] The fitting unit is configured to perform a linear interpolation process between adjacent preset bitrates to fit the bitrate and sound quality indicators, thereby obtaining a second set of coordinate points between adjacent preset bitrates.

[0162] The curve construction unit is configured to perform operations based on the first set of coordinate points and the second set of coordinate points to obtain the bitrate-quality curve corresponding to the target audio segment.

[0163] In one possible implementation, the above-mentioned apparatus may further include:

[0164] The second acquisition module is configured to acquire sample audio segments and corresponding annotation data, wherein the annotation data characterizes the reference sound quality indicators corresponding to the sample audio segments under the above-mentioned multiple preset bitrates.

[0165] The feature extraction module is configured to perform feature extraction processing on the sample audio segment to obtain sample audio features;

[0166] The audio quality prediction module is configured to input the sample audio features into a preset neural network to perform audio quality index prediction processing, and obtain the predicted audio quality indexes corresponding to the sample audio segment under the above-mentioned multiple preset bitrates respectively.

[0167] The loss determination module is configured to determine loss information based on the labeled data and the predicted sound quality index;

[0168] The training module is configured to train the preset neural network based on the loss information until the training iteration conditions are met, thereby obtaining the sound quality prediction model.

[0169] In one possible implementation, the second acquisition module described above may include:

[0170] The sample audio segment acquisition unit is configured to acquire historical audio segments as sample audio segments;

[0171] The encoding unit is configured to perform encoding processing on the sample audio segment using the multiple preset bitrates to obtain multiple encoded audio segments corresponding to the sample audio segment, each encoded audio segment corresponding to a preset bitrate;

[0172] The audio quality analysis unit is configured to perform audio quality comparison analysis between the sample audio segment and the corresponding multiple coded audio segments to obtain a reference audio quality index corresponding to the sample audio segment, which is used as the annotation data corresponding to the sample audio segment.

[0173] In one possible implementation, the first acquisition module 701 described above may include:

[0174] The target audio acquisition unit is configured to acquire the target audio to be encoded.

[0175] The segmentation processing unit is configured to perform segmentation processing on the target audio to obtain multiple audio segments;

[0176] The target audio segment acquisition unit is configured to acquire any one of the plurality of audio segments as the target audio segment.

[0177] In one possible implementation, the first acquisition module may further include:

[0178] The content category and bandwidth acquisition unit is configured to acquire the content category information and target bandwidth information of the target audio segment.

[0179] The target audio quality determination unit is configured to determine the target audio quality index based on the content category information and / or target bandwidth information.

[0180] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0181] Figure 8 This is a block diagram illustrating an electronic device for audio processing according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown, the electronic device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an audio processing method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0182] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the electronic device to which the present disclosure is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0183] Figure 9 This is a block diagram illustrating another electronic device for audio processing according to an exemplary embodiment. The electronic device may be a server, and its internal structure diagram may be as follows: Figure 9As shown, the electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an audio processing method.

[0184] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the electronic device to which the present disclosure is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0185] In an exemplary embodiment, an electronic device is also provided, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the audio processing method as described in the embodiments of this disclosure.

[0186] In an exemplary embodiment, a computer-readable storage medium is also provided, which, when executed by a processor of an electronic device, enables the electronic device to perform the audio processing method of the present disclosure embodiments. The computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0187] In an exemplary embodiment, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the audio processing method of the embodiments of this disclosure.

[0188] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0189] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0190] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An audio processing method, characterized in that, include: Obtain the target audio to be encoded and the target audio quality index; the target audio quality index is the audio quality index indicated for the target audio segment; the audio quality index indicated for the target audio segment refers to the audio quality index specified for the target audio segment; The target audio is segmented based on a preset duration to obtain multiple audio segments; the preset duration can be fixed or variable. Select any one of the multiple audio segments as the target audio segment to be encoded; Based on the audio quality prediction model, the target audio segment is subjected to audio quality index prediction processing to obtain the audio quality index corresponding to the target audio segment under various preset bitrates. Based on the various preset bitrates and the corresponding audio quality indicators under the various preset bitrates, a mapping relationship between bitrate and audio quality indicators is constructed; Based on the mapping relationship, the target bitrate corresponding to the target audio quality index is determined, and the target bitrate is used to encode the target audio segment.

2. The method according to claim 1, characterized in that, The audio quality prediction process based on the audio quality prediction model is used to predict the audio quality index of the target audio segment, obtaining the audio quality index of the target audio segment at various preset bitrates, including: The target audio segment is subjected to feature extraction processing to obtain audio features; The audio features are input into the audio quality prediction model for audio quality index prediction processing to obtain the audio quality index corresponding to the target audio segment under the various preset bitrates.

3. The method according to claim 2, characterized in that, The step of performing feature extraction processing on the target audio segment to obtain audio features includes: The target audio segment is subjected to frequency domain transformation processing to obtain the spectral data of multiple audio frames; The audio features are obtained based on the spectrum data.

4. The method according to claim 3, characterized in that, The step of obtaining the audio features based on the spectrum data includes: Extract frame features from the spectral data of each audio frame; The frame features are statistically processed to obtain the feature statistics of the multiple audio frames; The audio features are obtained based on the aforementioned feature statistics.

5. The method according to any one of claims 1-4, characterized in that, The mapping relationship is a bitrate-to-sound-quality curve; the step of constructing a mapping relationship between bitrate and sound-quality indicators based on the various preset bitrates and the corresponding sound-quality indicators under the various preset bitrates includes: Construct a two-dimensional coordinate system, wherein the coordinate axes of the two-dimensional coordinate system correspond to the sound quality index and the bit rate, respectively; Mark the first set of coordinate points consisting of each preset bitrate and its corresponding sound quality index in the two-dimensional coordinate system; Between adjacent preset bitrates, a linear interpolation method is used to fit the bitrate and sound quality indicators, resulting in a second set of coordinate points between adjacent preset bitrates. The bitrate audio quality curve is obtained based on the first set of coordinate points and the second set of coordinate points.

6. The method according to claim 1, characterized in that, The method further includes: Acquire sample audio segments and corresponding annotation data, wherein the annotation data characterizes the reference sound quality index of the sample audio segments under the various preset bitrates; The sample audio segments are subjected to feature extraction processing to obtain sample audio features; The sample audio features are input into a preset neural network for audio quality index prediction processing to obtain the predicted audio quality index of the sample audio segment under the various preset bitrates. Based on the labeled data and the predicted sound quality indicators, the loss information is determined; The preset neural network is trained based on the loss information until the training iteration conditions are met, thereby obtaining the sound quality prediction model.

7. The method according to claim 6, characterized in that, The acquisition of sample audio segments and corresponding annotation data includes: Obtain historical audio segments as sample audio segments; The sample audio segment is encoded using the multiple preset bitrates to obtain multiple encoded audio segments corresponding to the sample audio segment, and each encoded audio segment corresponds to a preset bitrate. The sample audio segment is compared and analyzed with the corresponding multiple coded audio segments to obtain the reference audio quality index corresponding to the sample audio segment, which is used as the annotation data corresponding to the sample audio segment.

8. The method according to any one of claims 1-4, characterized in that, The acquisition of the target sound quality index includes: Obtain the content category information and target bandwidth information of the target audio segment; The target audio quality index is determined based on the content category information and / or target bandwidth information.

9. An audio processing device, characterized in that, include: The first acquisition module is configured to acquire the target audio segment to be encoded and the target audio quality index; the target audio quality index is the audio quality index indicated by the target audio segment; the audio quality index indicated by the target audio segment refers to the audio quality index specified by the target audio segment. The audio quality prediction module is configured to perform audio quality index prediction processing on the target audio segment based on the audio quality prediction model, and obtain the audio quality index corresponding to the target audio segment under various preset bitrates. The mapping relationship construction module is configured to construct a mapping relationship between bitrate and sound quality index based on the multiple preset bitrates and the corresponding sound quality indexes under the multiple preset bitrates. The bitrate determination module is configured to determine the target bitrate corresponding to the target audio quality index based on the mapping relationship, wherein the target bitrate is used to encode the target audio segment; The first acquisition module includes: The target audio acquisition unit is configured to acquire the target audio to be encoded. The segmentation processing unit is configured to perform segmentation processing on the target audio based on a preset duration to obtain multiple audio segments; the preset duration can be fixed or variable. The target audio segment acquisition unit is configured to acquire any one of the plurality of audio segments as the target audio segment.

10. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the audio processing method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the audio processing method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, Includes computer instructions, which, when executed by a processor, cause the computer to perform the audio processing method as described in any one of claims 1 to 8.

Images