Audio optimization method and device for multimedia communication, equipment and storage medium

Abstract

The invention discloses an audio optimization method and device for multimedia communication, equipment and a storage medium, and relates to the technical field of audio processing, and the method comprises the steps: carrying out the noise reduction of to-be-transmitted original audio data of current multimedia equipment, and obtaining the noise-reduced audio data; performing variable-speed adjustment on the noise-reduced audio data placed in the audio buffer queue according to the delay time to obtain an adjustment parameter; enhancing and optimizing the high-frequency data through a Lagrange interpolation method to obtain target audio data; and transmitting the adjustment parameter and the target audio data to an audio receiving end according to the available audio frequency bandwidth. According to the method, after noise reduction and delay processing are carried out, the high-frequency data of the noise-reduced audio data are enhanced and optimized, and the transmission code rate is dynamically adjusted through the available audio frequency band width, so that the conditions of high-frequency signal distortion, detail loss and the like can be avoided, the definition and the accuracy of the audio are improved, and the user experience is improved. And the overall tone quality and the auditory experience of the user are improved.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to an audio optimization method, device, equipment and storage medium for multimedia communication. Background Art

[0002] With the development of IP voice technology and multimedia applications, users' requirements for audio output quality are constantly increasing. For multimedia audio playback devices, especially sound quality, we need to pay attention not only to the clarity and dynamic range of audio signals, but also to the optimization of the overall audio listening experience.

[0003] However, in the related art, during the audio transmission between multimedia devices, due to various factors such as recording equipment, environmental noise and interference during network transmission, there are generally situations such as distortion of high-frequency signals, loss of details and unnatural enhancement of audio signals. This reduces the clarity, accuracy and enjoyment of the audio, resulting in low audio transmission quality, making it difficult to provide high-quality audio effects and clear sound, which greatly affects the user experience.

[0004] Therefore, how to improve the voice audio quality in multimedia communications is an urgent problem to be solved. Summary of the invention

[0005] The main purpose of this application is to provide an audio optimization method, device, equipment and storage medium for multimedia communications, aiming to solve technical problems such as high-frequency distortion and detail loss in the audio transmission process between multimedia devices in IP voice technology and multimedia applications.

[0006] To achieve the above object, the present application proposes an audio optimization method for multimedia communication, the method comprising:

[0007] Obtain the original audio data to be transmitted from the current multimedia device;

[0008] Performing noise reduction processing on the original audio data to be transmitted by fast Fourier transform to obtain noise-reduced audio data;

[0009] The noise reduction audio data is placed into a preset audio buffer queue, and the audio buffer queue is adjusted at a variable speed according to the delay time of the original audio data to be transmitted, so as to obtain adjustment parameters matching the image data;

[0010] The high-frequency data of the noise reduction audio data is enhanced and optimized by Lagrange interpolation method to obtain target audio data;

[0011] The adjustment parameter and the target audio data are transmitted to an audio receiving end of the current multimedia device according to an available audio frequency bandwidth of the current multimedia device.

[0012] In one embodiment, the step of performing noise reduction processing on the original audio data to be transmitted by fast Fourier transform to obtain noise-reduced audio data includes:

[0013] Performing a fast Fourier transform on the original audio data to be transmitted to obtain corresponding FFT coefficients;

[0014] Determining a noise frequency band of the original audio data to be transmitted according to the density characteristics of the FFT coefficients;

[0015] Extracting the attenuation parameter of the silent segment in the original audio data to be transmitted according to the spectrum energy distribution of the FFT coefficients;

[0016] The noise frequency band of the original audio data to be transmitted is subjected to noise reduction processing according to the attenuation parameter to obtain noise-reduced audio data.

[0017] In one embodiment, the step of placing the noise reduction audio data into a preset audio buffer queue, and adjusting the audio buffer queue at a variable speed according to the delay time of the original audio data to be transmitted to obtain adjustment parameters matching the image data includes:

[0018] Put the noise reduction audio data into a preset audio buffer queue;

[0019] Determining the delay time of each audio frame in the original audio data to be transmitted according to the sending time and receiving time of the original audio data to be transmitted and the sending time and receiving time of the corresponding image data;

[0020] Dividing the audio buffer queue into a plurality of audio delay intervals according to the delay time;

[0021] The multiple audio speed change intervals are adjusted to obtain adjustment parameters that match the image data.

[0022] In one embodiment, the step of enhancing and optimizing the high-frequency data of the noise-reduced audio data by Lagrange interpolation to obtain the target audio data includes:

[0023] Convert the noise reduction audio data from the time domain to the frequency domain to obtain frequency domain audio data;

[0024] Extracting features from the frequency domain audio data using an independent component analysis algorithm to obtain high frequency data;

[0025] The high-frequency data is enhanced and optimized by Lagrange interpolation method to obtain target audio data.

[0026] In one embodiment, the step of extracting features from the frequency domain audio data using an independent component analysis algorithm to obtain high-frequency data includes:

[0027] Extracting features from the frequency domain audio data using an independent component analysis algorithm to obtain key features;

[0028] Extracting high-frequency features of the key features through a high-pass filter;

[0029] Based on the high frequency feature, high frequency data of the noise reduction audio data is determined.

[0030] In one embodiment, the step of transmitting the adjustment parameter and the target audio data to the audio receiving end of the current multimedia device according to the available audio bandwidth of the current multimedia device includes:

[0031] Determine the available audio frequency bandwidth of the current multimedia device and the original sampling rate of the original audio data to be transmitted;

[0032] Determining the audio transmission bit rate of the current multimedia device according to the available audio frequency bandwidth;

[0033] Performing audio encoding on the adjustment parameter and the target audio data according to the original sampling rate and the audio transmission bit rate to obtain an audio data packet;

[0034] The audio data packet is transmitted to an audio receiving end of the current multimedia device.

[0035] In addition, to achieve the above purpose, the present application also proposes an audio optimization device for multimedia communication, the device comprising:

[0036] A data acquisition module, used to acquire the original audio data to be transmitted from the current multimedia device;

[0037] A noise reduction processing module, used for performing noise reduction processing on the original audio data to be transmitted by fast Fourier transform to obtain noise-reduced audio data;

[0038] A speed adjustment module, used for placing the noise reduction audio data into a preset audio buffer queue, and performing speed adjustment on the audio buffer queue according to the delay time of the original audio data to be transmitted, so as to obtain adjustment parameters matching the image data;

[0039] An enhancement and optimization module, used to enhance and optimize the high-frequency data of the noise reduction audio data by Lagrange interpolation method to obtain target audio data;

[0040] The audio transmission module is used to transmit the adjustment parameter and the target audio data to the audio receiving end of the current multimedia device according to the available audio frequency bandwidth of the current multimedia device.

[0041] In addition, to achieve the above-mentioned purpose, the present application also proposes an audio optimization device for multimedia communication, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the audio optimization method for multimedia communication as described above.

[0042] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the audio optimization method for multimedia communication as described above are implemented.

[0043] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the audio optimization method for multimedia communication as described above are implemented.

[0044] One or more technical solutions proposed in this application have at least the following technical effects: the application first obtains the original audio data to be transmitted of the current multimedia device; then performs noise reduction processing on the original audio data to be transmitted through fast Fourier transform to obtain noise-reduced audio data; then puts the noise-reduced audio data into a preset audio buffer queue, and adjusts the audio buffer queue according to the delay time of the original audio data to be transmitted, and obtains adjustment parameters matching the image data; then enhances and optimizes the high-frequency data of the noise-reduced audio data through Lagrange interpolation to obtain target audio data; finally, according to the available audio bandwidth of the current multimedia device, the adjustment parameters and the target audio data are transmitted to the audio receiving end of the current multimedia device. Since the application enhances and optimizes the high-frequency data of the noise-reduced audio data after noise reduction and delay processing of the original audio data to be transmitted, and based on this, the transmission bit rate is dynamically adjusted through the available audio bandwidth, so that distortion of high-frequency signals, loss of details, etc. can be avoided, thereby improving the clarity and accuracy of the audio, solving the problem of low audio quality of network transmission in multimedia communication, and improving the overall sound quality and the user's auditory experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0047] Figure 1 A flowchart of the first embodiment of the audio optimization method for multimedia communication of the present application is provided;

[0048] Figure 2 A flowchart of the second embodiment of the audio optimization method for multimedia communication of the present application is provided;

[0049] Figure 3 This is a schematic diagram of the module structure of the audio optimization device for multimedia communication according to an embodiment of the present application;

[0050] Figure 4 This is a schematic diagram of the device structure of the hardware operating environment involved in the audio optimization method for multimedia communication in the embodiment of the present application.

[0051] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0053] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0054] It should be noted that the execution subject of this embodiment can be a computing service device with noise reduction processing, delay speed change and enhanced optimization functions, such as a personal computer, a projector, etc., or an electronic device that can realize the above functions, an audio optimization device for multimedia communication that executes the audio optimization method for multimedia communication of this application (hereinafter referred to as an audio optimization device), etc., and this embodiment does not limit this. The following takes the audio optimization device as an example to illustrate this embodiment and the following embodiments.

[0055] Based on this, the embodiment of the present application provides an audio optimization method for multimedia communication, referring to Figure 1 , Figure 1 A flowchart diagram is provided for Embodiment 1 of the audio optimization method for multimedia communication of the present application.

[0056] In this embodiment, the audio optimization method for multimedia communication includes steps S10 to S50:

[0057] Step S10: Acquire the original audio data to be transmitted of the current multimedia device.

[0058] It should be noted that current multimedia devices are electronic devices that can process, store, and display a variety of media information (such as text, images, audio, video, etc.), such as computers, mobile phones, tablets, speakers, projectors, etc., which are not limited in this embodiment.

[0059] It should be noted that the raw audio data to be transmitted is audio information that has not yet been transmitted and has not been processed.

[0060] The original audio data to be transmitted can be recorded through the current multimedia device, and then can be sent from the current multimedia device to another device through IP voice technology (Voice over Internet Protocol, referred to as VoIP, a technology for transmitting voice communication through the Internet or other IP networks).

[0061] However, during the transmission process, due to various factors, such as recording equipment, environmental noise, and interference during network transmission, there are usually distortions of high-frequency signals, loss of details, and unnatural enhancement of audio signals. Therefore, subsequent processing is required.

[0062] Step S20: performing noise reduction processing on the original audio data to be transmitted by fast Fourier transform to obtain noise-reduced audio data.

[0063] It should be noted that Fast Fourier Transform (FFT) is a method that can efficiently decompose the time domain signal in the audio into components of different frequencies, so as to analyze the characteristics and energy distribution of the audio at different frequencies.

[0064] Through fast Fourier transform, the original audio data to be transmitted can be converted from the time domain to the frequency domain to obtain data reflecting the characteristics and energy distribution of the original audio data to be transmitted on different frequency components, so as to facilitate noise reduction processing.

[0065] It should be noted that the noise reduction audio data is audio data obtained after processing the original audio data to be transmitted to reduce or eliminate the noise components therein.

[0066] Specifically, the noise reduction process may include: after obtaining the energy distribution of the original audio data to be transmitted through fast Fourier transform, the part with abnormal energy distribution may be retrieved, and the audio data with abnormal energy distribution may be subjected to noise reduction processing by adaptive filtering.

[0067] In this implementation, various noises may be mixed in during the audio recording or transmission process, such as environmental noise, device noise, etc. Through noise reduction processing, the original audio data to be transmitted can be analyzed and processed, and these noise parts can be identified and weakened, so as to make the audio clearer and purer, and improve the quality and audibility of the audio.

[0068] In a feasible implementation, step S20 of this embodiment may include the steps of: performing fast Fourier transform on the original audio data to be transmitted to obtain corresponding FFT coefficients; determining the noise frequency band of the original audio data to be transmitted based on the density characteristics of the FFT coefficients; extracting the attenuation parameters of the silent segment in the original audio data to be transmitted based on the spectral energy distribution of the FFT coefficients; and performing noise reduction processing on the noise frequency band of the original audio data to be transmitted based on the attenuation parameters to obtain noise-reduced audio data.

[0069] It should be noted that the FFT coefficients are a series of values ​​obtained by performing a fast Fourier transform on the original audio data to be transmitted.

[0070] The frequency domain of the original audio data to be transmitted after the fast Fourier transform contains information about different frequency components, which are represented by corresponding coefficients. Each coefficient corresponds to the amplitude and phase characteristics of a specific frequency, reflecting the proportion and characteristics of the frequency component in the original audio data to be transmitted.

[0071] It should be noted that the density characteristic refers to the degree of compactness or sparseness of the FFT coefficients.

[0072] For example, the FFT coefficients obtained after the fast Fourier transform have the following characteristics: the coefficient values ​​are relatively small or close to zero, and a small number of coefficients have large and significant values. The density characteristics can help understand the spectrum energy distribution.

[0073] It should be noted that the noise frequency band is the frequency interval occupied by the noise. Different types of noise have different frequency characteristics. For example, some mechanical noise may be more prominent in certain low-frequency bands, while the noise generated by electronic equipment may be more obvious in higher frequency bands.

[0074] Therefore, by analyzing the density characteristics of the FFT coefficients, the corresponding noise frequency band can be determined.

[0075] It should be noted that the spectrum energy distribution refers to data representing the characteristics and energy distribution of the original audio data to be transmitted on different frequency components.

[0076] It should be noted that the attenuation parameter of the silent segment refers to a relevant parameter describing the energy attenuation characteristics of the original audio data to be transmitted during the time period when the original audio data to be transmitted is in a silent state.

[0077] By borrowing the attenuation parameters of the silent segment, the noise reduction process of the noise frequency band can be better controlled to make it a smooth transition to reduce the impact of noise.

[0078] In this implementation, noise elimination can be performed accurately based on the spectrum energy distribution to reduce the impact of noise in the original audio data to be transmitted.

[0079] Step S30: putting the noise reduction audio data into a preset audio buffer queue, and performing variable speed adjustment on the audio buffer queue according to the delay time of the original audio data to be transmitted, so as to obtain adjustment parameters matching the image data.

[0080] It should be noted that the audio buffer queue is a structure queue used for audio processing of the delay of noise reduction audio data.

[0081] The audio buffer queue can be used to temporarily store the noise reduction audio data to be processed, and its delay time can be adjusted according to a certain order or rule to ensure the continuity and stability of the speed adjustment.

[0082] The delay time is the time interval from the generation of the original audio data to be transmitted to the actual reception or playback. In this process, there is a certain time difference due to various factors such as signal transmission, processing, and device performance. If the delay time is too long, the user may feel that the sound and image data (i.e., the original image data corresponding to the original audio data to be transmitted) are not synchronized.

[0083] Therefore, the delay time of the noise reduction audio data needs to be adjusted with the help of the audio buffer queue to obtain parameters (ie, adjustment parameters) that can synchronize the original audio data to be transmitted with the corresponding original image data when they are actually received or played.

[0084] The speed adjustment process may be achieved by adjusting the size and processing speed of the audio buffer queue, or by correcting and unifying the timestamps of the original audio data to be transmitted and the corresponding original image data, which is not limited in this embodiment.

[0085] In this implementation, by adjusting the parameters, the time at which the original audio data to be transmitted and the corresponding image data are actually received can be unified, so as to reduce the delay and achieve synchronization between image and audio.

[0086] In a feasible implementation, step S30 of this embodiment may include the steps of: placing the noise reduction audio data into a preset audio buffer queue; determining the delay time of each audio frame in the original audio data to be transmitted according to the sending time and receiving time of the original audio data to be transmitted and the sending time and receiving time of the corresponding image data; dividing the audio buffer queue into multiple audio delay intervals according to the delay time; and performing speed adjustment on the multiple audio speed change intervals to obtain adjustment parameters matching the image data.

[0087] It should be noted that the sending time refers to the specific time when the original audio data (or image data) to be transmitted starts to be sent from the transmitting end (i.e. the current multimedia device), and the receiving time refers to the specific time when the original audio data (or image data) to be transmitted is received at the receiving end.

[0088] By comparing the sending time and receiving time of the original audio data to be transmitted and the image data respectively, the delay situation existing in the transmission process of the original audio data to be transmitted can be obtained.

[0089] It should be noted that the audio delay interval is a queue interval of different delay time periods obtained by dividing the audio buffer queue according to different delay times.

[0090] In this implementation, the audio optimization device can adjust the delay threshold of the audio delay interval in real time; based on the adjusted delay threshold, the adjustment parameters after synchronization of the original audio data to be transmitted and the corresponding image data can be determined.

[0091] Step S40: enhancing and optimizing the high-frequency data of the noise reduction audio data by Lagrange interpolation method to obtain target audio data.

[0092] It should be noted that the Lagrange interpolation method is a numerical analysis method that constructs a polynomial function through some known discrete data points so that the function value of the polynomial at these known data points is equal to the given value.

[0093] Through the Lagrange interpolation method, the high-frequency data of the noise-reduced frequency data can be approximately interpolated to achieve enhanced optimization of the high-frequency data.

[0094] The high-frequency data is the higher-frequency part of the original audio data to be transmitted.

[0095] Considering that the distortion and detail loss of high-frequency signals are more obvious during the transmission of the original audio data to be transmitted, in order to improve the overall sound quality, the high-frequency data can be enhanced and optimized to obtain optimized high-frequency band data (i.e., target audio data) to reduce the distortion and detail loss of the high-frequency data.

[0096] Specifically, the enhanced optimization process may be implemented by a Lagrange interpolation method or a quadratic spline interpolation method, which is not limited in this embodiment.

[0097] In this implementation, by enhancing and optimizing the high-frequency data of the noise reduction audio data, the distortion and detail loss of the high-frequency data can be reduced, thereby improving the quality of the audio.

[0098] Step S50: transmitting the adjustment parameter and the target audio data to an audio receiving end of the current multimedia device according to the available audio frequency bandwidth of the current multimedia device.

[0099] It should be noted that the available audio bandwidth is the available network bandwidth that can be effectively transmitted and used in the audio channel of the current multimedia device.

[0100] The audio receiving end is a device used to receive the original audio data to be transmitted from the multimedia device, such as a microphone, other audio devices, etc.

[0101] In this embodiment, it is considered that packet loss is likely to occur during network audio transmission when the network bandwidth is limited. Therefore, in order to avoid retransmission or audio distortion, the transmission of adjustment parameters and target audio data can be achieved through the audio bandwidth to improve the quality of network audio transmission.

[0102] In a feasible implementation, step S50 of this embodiment may include the steps of: determining the available audio bandwidth of the current multimedia device and the original sampling rate of the original audio data to be transmitted; determining the audio transmission bit rate of the current multimedia device based on the available audio bandwidth; performing audio encoding on the adjustment parameters and the target audio data according to the original sampling rate and the audio transmission bit rate to obtain an audio data packet; and transmitting the audio data packet to the audio receiving end of the current multimedia device.

[0103] It should be noted that the original sampling rate is the number of times the original audio data to be transmitted is sampled per second when it is digitally sampled. The higher the sampling rate, the closer the original audio data to be transmitted is to the original analog audio signal, which can provide better sound quality during playback, but it will also take up more data storage space.

[0104] The audio transmission bit rate is the amount of audio data transmitted to the audio receiving end per unit time, which reflects the flow rate of data during the transmission process.

[0105] Therefore, in order to avoid packet loss during network audio transmission when the network bandwidth is limited, the appropriate audio transmission bit rate can be determined according to the available audio bandwidth, and the adjustment parameters and target audio data are encoded according to the audio transmission bit rate to obtain audio data packets that can avoid packet loss during network audio transmission.

[0106] In this implementation, by using a suitable audio transmission bit rate, packet loss during network audio transmission can be avoided, thereby improving the audio quality of audio data packets in network audio transmission.

[0107] In the technical solution provided in this embodiment, the original audio data to be transmitted can be recorded through the current multimedia device, and then it can be sent from the current multimedia device to another device through IP voice technology. However, considering that in this transmission process, due to various factors, such as recording equipment, environmental noise and interference during network transmission, there are generally situations such as distortion of high-frequency signals, loss of details and unnatural enhancement of audio signals. Therefore, it is also necessary to perform noise reduction processing on the original audio data to be transmitted through fast Fourier transform to obtain noise-reduced audio data. Then, the noise-reduced audio data is placed in a preset audio buffer queue, and the audio buffer queue is adjusted according to the delay time of the original audio data to be transmitted. The original audio data to be transmitted and the corresponding image data can be unified at the time of actual reception to reduce the delay and obtain the adjustment parameters synchronized with the image data picture and sound. Then, the high-frequency data of the noise-reduced audio data is approximately interpolated by the Lagrange interpolation method to achieve enhanced optimization of the high-frequency data and obtain the target audio data. Finally, considering that in the process of network audio transmission, packet loss is prone to occur when the network bandwidth is limited. Therefore, in order to avoid retransmission or audio distortion, the adjustment parameters and target audio data can be transmitted to the audio receiving end of the current multimedia device according to the available audio bandwidth of the current multimedia device to improve the quality of network audio transmission. Since this embodiment enhances and optimizes the high-frequency data of the noise-reduced audio data after performing noise reduction and delay processing on the original audio data to be transmitted, and then dynamically adjusts the transmission bit rate based on the available audio bandwidth, it can avoid distortion of high-frequency signals, loss of details, etc., thereby improving the clarity and accuracy of the audio, solving the problem of low audio quality of network transmission in multimedia communication, and improving the overall sound quality and the user's auditory experience.

[0108] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 1 and Figure 2 , Figure 2 A flowchart diagram is provided for Embodiment 2 of the audio optimization method for multimedia communication of the present application.

[0109] In this example, step S40 includes steps S41 to S43:

[0110] Step S41: converting the noise reduction audio data from the time domain to the frequency domain to obtain frequency domain audio data.

[0111] It should be noted that the noise reduction audio data in the time domain represents the change of the signal over time with time as the axis. Therefore, the noise reduction audio data can be converted from the time dimension to the frequency dimension (i.e., frequency domain) through a specific mathematical transformation (such as Fourier transform, etc.). At this time, the distribution of energy or amplitude of the noise reduction audio data in different frequency components can be seen, so as to analyze the characteristics of the noise reduction audio data.

[0112] Step S42: extracting features from the frequency domain audio data using an independent component analysis algorithm to obtain high frequency data.

[0113] Step S43: enhancing and optimizing the high-frequency data by Lagrange interpolation method to obtain target audio data.

[0114] It should be noted that the Independent Component Analysis (ICA) algorithm is a signal processing and data analysis method that can separate independent potential components from observed mixed signals.

[0115] Therefore, the frequency domain audio data can be separated into components through the independent component analysis algorithm, so as to obtain the high-frequency data in the noise reduction audio data.

[0116] In this embodiment, the independent component analysis algorithm can more intuitively analyze the characteristics of frequency domain audio data at different frequencies, such as which frequencies have stronger energy and which main frequency components exist, which helps to quickly extract high-frequency data from the noise reduction audio data.

[0117] In a feasible implementation, step S40 of this example includes the steps of: extracting features of the frequency domain audio data through an independent component analysis algorithm to obtain key features; extracting high-frequency features of the key features through a high-pass filter; and determining high-frequency data of the noise reduction audio data based on the high-frequency features.

[0118] It should be noted that the key features are the features of the frequency domain audio data at different frequencies extracted by the independent component analysis algorithm.

[0119] Among them, the high-pass filter is a filter that allows the high-frequency signal to pass relatively smoothly, while suppressing or blocking the low-frequency signal to a greater extent.

[0120] Therefore, the low-frequency features of the key features can be filtered out by a high-pass filter to obtain high-frequency features.

[0121] In this implementation, high-frequency data in the noise reduction audio data can be quickly extracted during the audio processing of the frequency domain audio data by using a high-pass filter.

[0122] In the technical solution provided in this embodiment, the independent component analysis algorithm can more intuitively analyze the characteristics of frequency domain audio data at different frequencies, such as which frequencies have stronger energy, which main frequency components exist, etc., which helps to quickly extract high-frequency data from the noise reduction audio data.

[0123] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the audio optimization method for multimedia communication of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0124] This application also provides an audio optimization device for multimedia communication, please refer to Figure 3 , Figure 3 This is a schematic diagram of the module structure of the audio optimization device for multimedia communication according to an embodiment of the present application; the audio optimization device for multimedia communication includes:

[0125] The data acquisition module 301 is used to acquire the original audio data to be transmitted from the current multimedia device;

[0126] The noise reduction processing module 302 is used to perform noise reduction processing on the original audio data to be transmitted by fast Fourier transform to obtain noise-reduced audio data;

[0127] The speed adjustment module 303 is used to put the noise reduction audio data into a preset audio buffer queue, and perform speed adjustment on the audio buffer queue according to the delay time of the original audio data to be transmitted, so as to obtain adjustment parameters matching the image data;

[0128] An enhancement and optimization module 304 is used to enhance and optimize the high-frequency data of the noise reduction audio data by Lagrange interpolation method to obtain target audio data;

[0129] The audio transmission module 305 is used to transmit the adjustment parameter and the target audio data to the audio receiving end of the current multimedia device according to the available audio frequency bandwidth of the current multimedia device.

[0130] In this embodiment, the noise reduction processing module 302 is also used to perform fast Fourier transform on the original audio data to be transmitted to obtain corresponding FFT coefficients; determine the noise frequency band of the original audio data to be transmitted according to the density characteristics of the FFT coefficients; extract the attenuation parameters of the silent segment in the original audio data to be transmitted according to the spectral energy distribution of the FFT coefficients; perform noise reduction processing on the noise frequency band of the original audio data to be transmitted according to the attenuation parameters to obtain noise-reduced audio data.

[0131] Furthermore, the speed adjustment module 303 is also used to put the noise reduction audio data into a preset audio buffer queue; determine the delay time of each audio frame in the original audio data to be transmitted according to the sending time and receiving time of the original audio data to be transmitted and the sending time and receiving time of the corresponding image data; divide the audio buffer queue into multiple audio delay intervals according to the delay time; and perform speed adjustment on the multiple audio speed change intervals to obtain adjustment parameters matching the image data.

[0132] Furthermore, the enhancement and optimization module 304 is also used to convert the noise reduction audio data from the time domain to the frequency domain to obtain frequency domain audio data; perform feature extraction on the frequency domain audio data through an independent component analysis algorithm to obtain high-frequency data; and enhance and optimize the high-frequency data through a Lagrange interpolation method to obtain target audio data.

[0133] Furthermore, the enhancement optimization module 304 is also used to extract features of the frequency domain audio data through an independent component analysis algorithm to obtain key features; extract high-frequency features of the key features through a high-pass filter; and determine high-frequency data of the noise reduction audio data based on the high-frequency features.

[0134] Furthermore, the audio transmission module 305 is also used to determine the available audio frequency bandwidth of the current multimedia device and the original sampling rate of the original audio data to be transmitted; determine the audio transmission bit rate of the current multimedia device according to the available audio frequency bandwidth; perform audio encoding on the adjustment parameters and the target audio data according to the original sampling rate and the audio transmission bit rate to obtain an audio data packet; and transmit the audio data packet to the audio receiving end of the current multimedia device.

[0135] Other embodiments or specific implementation methods of the audio optimization device for multimedia communication of the present application can refer to the above-mentioned method embodiments and will not be repeated here.

[0136] The audio optimization device for multimedia communication provided by the present application adopts the audio optimization method for multimedia communication in the above-mentioned embodiment, and can solve the technical problems of high-frequency distortion, detail loss, etc. in the audio transmission process between multimedia devices in IP voice technology and multimedia applications. Compared with the prior art, the beneficial effects of the audio optimization device for multimedia communication provided by the present application are the same as the beneficial effects of the audio optimization method for multimedia communication provided by the above-mentioned embodiment, and the other technical features of the audio optimization device for multimedia communication are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0137] The present application provides an audio optimization device for multimedia communication, and the audio optimization device for multimedia communication includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the audio optimization method for multimedia communication in the above-mentioned embodiment one.

[0138] Reference below Figure 4 , Figure 4 The schematic diagram of the device structure of the hardware operating environment involved in the audio optimization method for multimedia communication in the embodiment of the present application shows a schematic diagram of the structure of the audio optimization device for multimedia communication suitable for implementing the embodiment of the present application. The audio optimization device for multimedia communication in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The audio optimization device for multimedia communication shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0139] like Figure 4As shown, the audio optimization device for multimedia communication may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the audio optimization device for multimedia communication are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the audio optimization device for multimedia communication to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an audio optimization device for multimedia communication with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0140] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0141] The audio optimization device for multimedia communication provided by the present application adopts the audio optimization method for multimedia communication in the above-mentioned embodiment, which can solve the technical problems of high-frequency distortion, detail loss, etc. in the audio transmission process between multimedia devices in IP voice technology and multimedia applications. Compared with the prior art, the beneficial effects of the audio optimization device for multimedia communication provided by the present application are the same as the beneficial effects of the audio optimization method for multimedia communication provided by the above-mentioned embodiment, and the other technical features of the audio optimization device for multimedia communication are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0142] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0143] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0144] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the audio optimization method for multimedia communication in the above-mentioned embodiment.

[0145] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0146] The computer-readable storage medium may be included in the audio optimization device for multimedia communication; or may exist independently without being assembled into the audio optimization device for multimedia communication.

[0147] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the audio optimization device for multimedia communication, the audio optimization device for multimedia communication: obtains the original audio data to be transmitted of the current multimedia device; performs noise reduction processing on the original audio data to be transmitted by fast Fourier transform to obtain noise-reduced audio data; puts the noise-reduced audio data into a preset audio buffer queue, and performs variable speed adjustment on the audio buffer queue according to the delay time of the original audio data to be transmitted to obtain adjustment parameters matching the image data; enhances and optimizes the high-frequency data of the noise-reduced audio data by Lagrange interpolation method to obtain target audio data; and transmits the adjustment parameters and the target audio data to the audio receiving end of the current multimedia device according to the available audio bandwidth of the current multimedia device.

[0148] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0149] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0150] The modules involved in the embodiments described in the present application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0151] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned audio optimization method for multimedia communication, and can solve the technical problems of high-frequency distortion, detail loss, etc. in the audio transmission process between multimedia devices in IP voice technology and multimedia applications. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the audio optimization method for multimedia communication provided by the above-mentioned embodiment, and will not be repeated here.

[0152] The present application also provides a computer program product, including a computer program, which implements the steps of the audio optimization method for multimedia communication as described above when the computer program is executed by a processor.

[0153] The computer program product provided by the present application can solve the technical problems of high-frequency distortion, detail loss, etc. in the audio transmission process between multimedia devices in IP voice technology and multimedia applications. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the audio optimization method for multimedia communication provided by the above-mentioned embodiment, which will not be described in detail here.

[0154] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An audio optimization method for multimedia communication, characterized in that: The method includes: Obtain the original audio data to be transmitted from the current multimedia device; Performing noise reduction processing on the original audio data to be transmitted by fast Fourier transform to obtain noise-reduced audio data; The noise reduction audio data is placed into a preset audio buffer queue, and the audio buffer queue is adjusted at a variable speed according to the delay time of the original audio data to be transmitted, so as to obtain adjustment parameters matching the image data; The high-frequency data of the noise reduction audio data is enhanced and optimized by Lagrange interpolation method to obtain target audio data; The adjustment parameter and the target audio data are transmitted to an audio receiving end of the current multimedia device according to an available audio frequency bandwidth of the current multimedia device.

2. The method according to claim 1, characterized in that The step of performing noise reduction processing on the original audio data to be transmitted by fast Fourier transform to obtain noise-reduced audio data comprises: Performing a fast Fourier transform on the original audio data to be transmitted to obtain corresponding FFT coefficients; Determining a noise frequency band of the original audio data to be transmitted according to the density characteristics of the FFT coefficients; Extracting the attenuation parameter of the silent segment in the original audio data to be transmitted according to the spectrum energy distribution of the FFT coefficients; The noise frequency band of the original audio data to be transmitted is subjected to noise reduction processing according to the attenuation parameter to obtain noise-reduced audio data.

3. The method according to claim 1, characterized in that The step of placing the noise reduction audio data into a preset audio buffer queue, and adjusting the audio buffer queue at a variable speed according to the delay time of the original audio data to be transmitted to obtain adjustment parameters matching the image data includes: Put the noise reduction audio data into a preset audio buffer queue; Determining the delay time of each audio frame in the original audio data to be transmitted according to the sending time and receiving time of the original audio data to be transmitted and the sending time and receiving time of the corresponding image data; Dividing the audio buffer queue into a plurality of audio delay intervals according to the delay time; The multiple audio speed change intervals are adjusted to obtain adjustment parameters that match the image data.

4. The method according to claim 1, characterized in that The step of enhancing and optimizing the high-frequency data of the noise-reduced audio data by Lagrange interpolation method to obtain target audio data comprises: Convert the noise reduction audio data from the time domain to the frequency domain to obtain frequency domain audio data; Extracting features from the frequency domain audio data using an independent component analysis algorithm to obtain high frequency data; The high-frequency data is enhanced and optimized by Lagrange interpolation method to obtain target audio data.

5. The method according to claim 4, characterized in that The step of extracting features from the frequency domain audio data by an independent component analysis algorithm to obtain high-frequency data comprises: Extracting features from the frequency domain audio data using an independent component analysis algorithm to obtain key features; Extracting high-frequency features of the key features through a high-pass filter; Based on the high frequency feature, high frequency data of the noise reduction audio data is determined.

6. The method according to any one of claims 1 to 5, characterized in that The step of transmitting the adjustment parameter and the target audio data to the audio receiving end of the current multimedia device according to the available audio frequency bandwidth of the current multimedia device comprises: Determine the available audio frequency bandwidth of the current multimedia device and the original sampling rate of the original audio data to be transmitted; Determining the audio transmission bit rate of the current multimedia device according to the available audio frequency bandwidth; Performing audio encoding on the adjustment parameter and the target audio data according to the original sampling rate and the audio transmission bit rate to obtain an audio data packet; The audio data packet is transmitted to an audio receiving end of the current multimedia device.

7. An audio optimization device for multimedia communication, characterized in that: The device comprises: A data acquisition module, used to acquire the original audio data to be transmitted from the current multimedia device; A noise reduction processing module, used for performing noise reduction processing on the original audio data to be transmitted by fast Fourier transform to obtain noise-reduced audio data; A speed adjustment module, used for placing the noise reduction audio data into a preset audio buffer queue, and performing speed adjustment on the audio buffer queue according to the delay time of the original audio data to be transmitted, so as to obtain adjustment parameters matching the image data; An enhancement and optimization module, used to enhance and optimize the high-frequency data of the noise reduction audio data by Lagrange interpolation method to obtain target audio data; The audio transmission module is used to transmit the adjustment parameter and the target audio data to the audio receiving end of the current multimedia device according to the available audio frequency bandwidth of the current multimedia device.

8. An audio optimization device for multimedia communication, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the audio optimization method for multimedia communication according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the audio optimization method for multimedia communication according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the audio optimization method for multimedia communication according to any one of claims 1 to 6 are implemented.

Images