Speech enhancement method, device, equipment, storage medium and program product

Abstract

The present application discloses a speech enhancement method, device, equipment, storage medium and program product, which relates to the field of speech technology processing. The method includes: dividing the target audio to be speech enhanced into frequency bands along the frequency domain dimension to obtain at least two sub-bands; obtaining sub-band energy data corresponding to at least two sub-bands respectively; analyzing the sub-band energy data corresponding to at least two sub-bands respectively along the time domain dimension to obtain sub-band energy distribution data corresponding to at least two sub-bands respectively; when the sub-band energy distribution data corresponding to the specified sub-band meets the adjustment conditions, adjusting the sub-band energy data of the specified sub-band to obtain the target enhanced audio. In the above manner, the sub-band energy data that does not meet the adjustment conditions in the target audio can be selectively adjusted, and the quality of speech enhancement is improved while fully considering the characteristics of the target audio. The present application can be applied to various scenarios such as cloud technology, artificial intelligence, and smart transportation.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of speech technology processing, and in particular to a speech enhancement method, apparatus, device, storage medium and program product. Background Art

[0002] During the transmission process, speech will inevitably be interfered by the surrounding environment and the internal noise of the communication equipment. Therefore, it is necessary to use speech enhancement technology to extract the purest original speech from the noisy speech signal. Speech enhancement technology plays an important role in speech processing, speech recognition, speech detection and other fields.

[0003] In related technologies, noise suppression, echo cancellation, volume adjustment and other methods are usually used to process speech signals. For example, the noise component in the speech signal is suppressed through deep learning methods, and a speech signal with enhanced signal-to-noise ratio is output.

[0004] Although the speech signal obtained by the above method reduces the interference of noise to a certain extent, when the receiver receives the speech signal for a long time, it may cause auditory fatigue to the speech signal, which affects the subsequent processing process of the speech signal. Summary of the invention

[0005] The embodiments of the present application provide a speech enhancement method, apparatus, device, storage medium and program product, which can selectively adjust sub-band energy data in target audio that does not meet the adjustment conditions, while fully considering the characteristics of the target audio and improving the quality of speech enhancement. The technical solution is as follows.

[0006] In one aspect, a method for speech enhancement is provided, the method comprising:

[0007] Acquire target audio, where the target audio is audio data to be speech enhanced;

[0008] Dividing the target audio frequency band along the frequency domain dimension to obtain at least two sub-bands;

[0009] Acquire sub-band energy data corresponding to the at least two sub-bands respectively, where the sub-band energy data is used to indicate frequency changes of audio frames in the target audio along the frequency domain dimension within the sub-bands;

[0010] Analyzing the sub-band energy data corresponding to the at least two sub-frequency bands respectively along the time domain dimension to obtain sub-band energy distribution data corresponding to the at least two sub-frequency bands respectively, wherein the sub-band energy distribution data is used to indicate the frequency distribution of the target audio in the at least two sub-frequency bands;

[0011] If, among the at least two sub-frequency bands, sub-band energy distribution data corresponding to a designated sub-frequency band meets the adjustment condition, an adjustment parameter is determined based on the sub-band energy distribution data corresponding to the designated sub-frequency band, and the sub-band energy data of the designated sub-frequency band is adjusted to obtain the target enhanced audio.

[0012] In another aspect, a speech enhancement device is provided, the device comprising:

[0013] An audio acquisition module, used to acquire target audio, where the target audio is audio data to be speech enhanced;

[0014] A frequency band segmentation module, used for performing frequency band segmentation on the target audio along a frequency domain dimension to obtain at least two sub-frequency bands;

[0015] A data acquisition module, used to acquire sub-band energy data corresponding to the at least two sub-bands respectively, wherein the sub-band energy data is used to indicate a frequency change of an audio frame in the target audio along a frequency domain dimension within the sub-band;

[0016] a data analysis module, configured to analyze the sub-band energy data corresponding to the at least two sub-frequency bands respectively along the time domain dimension to obtain sub-band energy distribution data corresponding to the at least two sub-frequency bands respectively, wherein the sub-band energy distribution data is used to indicate the frequency distribution of the target audio in the at least two sub-frequency bands;

[0017] An energy adjustment module is used to determine an adjustment parameter based on the sub-band energy distribution data corresponding to the specified sub-band in the at least two sub-bands, if the sub-band energy distribution data corresponding to the specified sub-band meets the adjustment conditions, and adjust the sub-band energy data of the specified sub-band to obtain the target enhanced audio.

[0018] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a speech enhancement method as described in any of the above-mentioned embodiments of the present application.

[0019] On the other hand, a computer-readable storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a speech enhancement method as described in any of the above-mentioned embodiments of the present application.

[0020] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the speech enhancement method described in any of the above embodiments.

[0021] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0022] The target audio for speech enhancement is divided into frequency bands along the frequency domain dimension to obtain at least two sub-bands, and sub-band energy data and sub-band energy distribution data corresponding to different sub-bands are obtained. When there is sub-band energy distribution data corresponding to a specified sub-band that meets the adjustment condition, an adjustment parameter is determined based on the sub-band energy distribution data corresponding to the specified sub-band, and the sub-band energy data of the specified sub-band is adjusted using the adjustment parameter, thereby effectively avoiding aimless adjustment of the entire speech signal of the target audio. By judging whether the sub-band energy distribution data of the sub-band meets the adjustment condition, the sub-band energy data of the specified sub-band that does not meet the adjustment condition is selectively adjusted, and then the sub-band energy data of the specified sub-band is adjusted using the adjustment parameter determined by the sub-band energy distribution data corresponding to the specified sub-band, and based on the adjusted sub-band energy data, a selectively enhanced target enhanced audio is obtained, while fully considering the characteristics of the target audio, the quality of speech enhancement is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 It is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application;

[0025] Figure 2 is a flow chart of a speech enhancement method provided by an exemplary embodiment of the present application;

[0026] Figure 3 is a schematic diagram of frequency band segmentation provided by an exemplary embodiment of the present application;

[0027] Figure 4 is a flow chart of a speech enhancement method provided by another exemplary embodiment of the present application;

[0028] Figure 5is a schematic diagram of target audio provided by an exemplary embodiment of the present application;

[0029] Figure 6 is a flow chart of a speech enhancement method provided by another exemplary embodiment of the present application;

[0030] Figure 7 is a schematic diagram of sub-band energy data adjustment provided by an exemplary embodiment of the present application;

[0031] Figure 8 is a processing flow chart of a speech enhancement method provided by an exemplary embodiment of the present application;

[0032] Fig. 9 is a flow chart of a speech enhancement method provided by another exemplary embodiment of the present application;

[0033] Fig.10 is a structural block diagram of a speech enhancement device provided by an exemplary embodiment of the present application;

[0034] Fig.11 It is a structural block diagram of a server provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0036] In the related technology, methods such as noise suppression, echo cancellation, and volume adjustment are usually used to process voice signals. For example, the noise component in the voice signal is suppressed by a deep learning method, and a voice signal with enhanced signal-to-noise ratio is output. Although the voice signal obtained by the above method reduces the interference of noise to a certain extent, when the receiver receives the voice signal for a long time, it may cause auditory fatigue to the voice signal, which affects the subsequent processing of the voice signal.

[0037] In an embodiment of the present application, a speech enhancement method is provided, which can selectively adjust sub-band energy data in the target audio that does not meet the adjustment conditions, while fully considering the characteristics of the target audio, and improving the quality of speech enhancement. The speech enhancement method trained in the present application can be applied in various scenarios including voice call enhancement scenarios and audio enhancement scenarios.

[0038] It is worth noting that the above application scenarios are only illustrative examples. The speech enhancement method provided in this embodiment can also be applied to other speech scenarios, and the embodiments of the present application are not limited to this.

[0039] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the voice data, target audio, etc. involved in this application are all obtained with full authorization.

[0040] Next, the implementation environment involved in the embodiments of the present application is described. For illustration, please refer to Figure 1 The implementation environment involves a terminal 110 and a server 120 , and the terminal 110 and the server 120 are connected via a communication network 130 .

[0041] In some embodiments, an application with an audio acquisition function is installed in the terminal 110. In some embodiments, the terminal 110 is used to send the target audio to the server 120. The server 120 can perform speech enhancement on the target audio through the speech enhancement model 121, and feed the enhanced target enhanced audio back to the terminal 110 for playback.

[0042] The application process of the speech enhancement model 121 is as follows: the acquired target audio is divided into frequency bands along the frequency domain dimension to obtain at least two sub-bands (sub-band 1, sub-band 2, ... sub-band n), and then the sub-band energy data corresponding to the at least two sub-bands are obtained (sub-band energy data 1, sub-band energy data 2, ... sub-band energy data n), wherein the sub-band energy data is used to indicate the frequency change of the audio frame in the target audio in the sub-band along the frequency domain dimension, and then the sub-band energy data corresponding to the at least two sub-bands are analyzed along the time domain dimension to obtain the sub-band energy distribution data corresponding to the at least two sub-bands (sub-band energy distribution data 1, sub-band energy distribution data 2, ... sub-band energy distribution data n), and the frequency distribution of the target audio on the at least two sub-bands can be determined through the sub-band energy distribution data. When there is a sub-band energy distribution data corresponding to a specified sub-band in at least two sub-bands that meets the adjustment condition, the adjustment parameter is determined based on the sub-band energy distribution data corresponding to the specified sub-band. For example: if the subband energy distribution data 1 meets the adjustment condition, then the subband 1 is the designated subband, and the adjustment parameter corresponding to the subband 1 is determined based on the subband energy analysis data 1, and the subband energy data (subband energy data 1) of the designated subband (subband 1) is adjusted by the adjustment parameter, so as to obtain the adjusted subband energy data (subband energy data 1) corresponding to the designated subband (subband 1), and obtain the target enhanced audio according to the adjusted subband energy data. The above process is an example of a non-unique case of the application process of the speech enhancement model 121.

[0043] It is worth noting that the above-mentioned terminals include but are not limited to mobile terminals such as mobile phones, tablet computers, portable laptops, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, and can also be implemented as desktop computers, etc.; the above-mentioned servers can be independent physical servers, or they can be server clusters or distributed systems composed of multiple physical servers, or they can be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0044] Among them, cloud technology refers to a hosting technology that unifies hardware, application programs, networks and other resources within a wide area network or local area network to achieve data computing, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient.

[0045] In some embodiments, the above server can also be implemented as a node in a blockchain system.

[0046] In combination with the above-mentioned noun introduction and application scenarios, the speech enhancement method provided by the present application is described, and the method is applied to a server as an example. Figure 2 As shown, the method includes the following steps 210 to 250.

[0047] Step 210, obtaining target audio.

[0048] The target audio is the audio data to be speech enhanced.

[0049] Schematically, audio is used to indicate data with audio information, such as a piece of music, a voice message, etc. Optionally, the audio is acquired by using a device with a built-in or external voice acquisition component, such as a terminal or a recorder. For example, the audio is acquired by using a terminal equipped with a microphone, a microphone array, or a pickup; or, the audio is synthesized by using an audio synthesis application to acquire the audio, etc. Optionally, the target audio is the audio data acquired by using the above acquisition method or synthesis method.

[0050] In an optional embodiment, the target audio is audio data acquired in real time. Schematically, in a scenario of a two-person call or a multi-person call, such as a telephone, a Voice over Internet Protocol (VoIP), a telephone conference, etc., the real-time generated voice is used as the target audio, that is, the target audio is the audio data during the call.

[0051] Optionally, the target audio to be voice enhanced is a song, which includes instrument audio data corresponding to multiple instrument sound sources and vocal audio data corresponding to vocal sound sources; or, the target audio to be voice enhanced is multi-person real-time call audio, which includes vocal audio data corresponding to multiple vocal sound sources and background sound audio data corresponding to background sound sources (such as ambient sound, noise, etc. audio data), etc.

[0052] Step 220 , dividing the target audio into frequency bands along the frequency domain dimension to obtain at least two sub-bands.

[0053] The frequency domain dimension is used to describe the dimensionality of the frequency characteristics of the target audio. By analyzing the target audio in the frequency domain dimension, the oscillation information of the target audio in the frequency domain dimension can be provided.

[0054] Alternatively, if Figure 3 As shown, after the target audio is obtained, the frequency band corresponding to the target audio 310 is divided along the frequency domain dimension, thereby obtaining at least two sub-bands 320.

[0055] Schematically, when the input target audio 310 is divided into frequency bands along the frequency domain dimension, the frequency band corresponding to the target audio 310 is divided into K sub-bands, and the dimension of each sub-band is F. k , k = 1, ... K, and satisfies

[0056] Optionally, K is a randomly generated number; or, K is a pre-set number, etc. Schematically, if the target audio 310 is divided into frequency bands in the same frequency band width (dimension), the frequency band widths of the K sub-bands are the same; or, if the target audio 310 is divided into frequency bands in different frequency band widths, the frequency band widths of the K sub-bands are different, for example, the frequency band widths of the K sub-bands are increased in sequence, the frequency band widths of the K sub-bands are randomly selected, etc.

[0057] Step 230: Obtain sub-band energy data corresponding to at least two sub-frequency bands.

[0058] Illustratively, after the target audio is frequency-band segmented to obtain at least two sub-bands, sub-band energy data corresponding to the at least two sub-bands are determined respectively, wherein the sub-band energy data is used to indicate the frequency variation of the audio frame in the target audio along the frequency domain dimension within the sub-band.

[0059] The audio frame is a unit of measurement corresponding to the audio. Optionally, the target audio is divided into frames according to the distribution of the audio signal corresponding to the target audio, the oscillation of the audio data, etc. For example, a section of audio data with a relatively stable frequency change is taken as an audio frame. Optionally, a fixed time interval is taken as the time interval corresponding to an audio frame, such as: 32 milliseconds is taken as an audio frame. Among them, an audio frame generally takes 20 to 50 milliseconds.

[0060] Based on the above audio frame division standard, the target audio is divided into frames, and a small segment of audio data obtained by interception is used as an audio frame, so as to determine the multiple audio frames corresponding to the target audio.

[0061] Optionally, when acquiring sub-band energy data corresponding to at least two sub-frequency bands, the sub-band energy data corresponding to at least two sub-frequency bands in different audio frames are acquired in units of audio frames, that is, in different audio frames, there are differences in the sub-band energy data corresponding to at least two sub-frequency bands.

[0062] In an optional embodiment, at least two sub-frequency bands in any audio frame are analyzed.

[0063] Illustratively, after sampling the target audio, a plurality of sampling points corresponding to the target audio are obtained. When determining the sub-band energy data of a certain sub-band, the sub-band energy data corresponding to the sub-band is determined according to the energy data corresponding to the plurality of sampling points. For example, the energy data corresponding to the plurality of sampling points in the sub-band are integrated to obtain the sub-band energy data corresponding to the sub-band; or, when determining the sub-band energy data of a certain sub-band, the plurality of sampling points are first band-pass filtered, and then the energy data corresponding to the plurality of filtered sampling points in the sub-band are integrated to obtain the sub-band energy data corresponding to the sub-band, etc.

[0064] Optionally, based on the above-mentioned method for determining sub-band energy data, sub-band energy data corresponding to multiple sub-bands under the same audio frame are determined, and sub-band energy data corresponding to multiple sub-bands under other audio frames are determined, so as to obtain sub-band energy data corresponding to different sub-bands under different audio frames.

[0065] Step 240: Analyze the sub-band energy data corresponding to the at least two sub-frequency bands respectively along the time domain dimension to obtain sub-band energy distribution data corresponding to the at least two sub-frequency bands respectively.

[0066] Schematically, the time domain dimension is a dimensional situation in which a time ruler is used to record the temporal changes of the target audio. Optionally, an audio frame is used to describe the changes in the time domain dimension, that is, a process of analyzing sub-band energy data corresponding to at least two sub-frequency bands along the time domain dimension, that is, a process of analyzing sub-band energy data corresponding to at least two sub-frequency bands under different audio frames.

[0067] Optionally, after obtaining the subband energy data corresponding to at least two subbands, the frequency change of the same subband in different audio frames is analyzed to determine the subband energy distribution data corresponding to at least two subbands. For example, there are subband A and subband B in the target audio, and in audio frame 1, the subband energy data a corresponding to subband A and the subband energy data b corresponding to subband B are obtained, and in audio frame 2, the subband energy data a' corresponding to subband A and the subband energy data b' corresponding to subband B are obtained, wherein audio frame 1 and audio frame 2 are different audio frames distributed along the time domain dimension.

[0068] Schematically, sub-band A is analyzed as an example, and the sub-band energy distribution data corresponding to sub-band A is determined according to the frequency change between sub-band energy data a of sub-band A in audio frame 1 and sub-band energy data a' in audio frame 2; or, sub-band B is analyzed as an example, and the sub-band energy distribution data corresponding to sub-band B is determined according to the frequency change between sub-band energy data b of sub-band B in audio frame 1 and sub-band energy data b' in audio frame 2. That is, the sub-band energy distribution data is used to indicate the frequency distribution of the target audio in at least two sub-bands.

[0069] Optionally, taking a certain sub-frequency band as an example, since the sub-band energy distribution data is based on data obtained under different audio frames in the sub-frequency band, for a sub-frequency band, it includes sub-band energy distribution data corresponding to different audio frames. One sub-band energy distribution data represents the relationship between the sub-band energy data under the audio frame and the sub-band energy data under other audio frames, etc.

[0070] Step 250: If there is sub-band energy distribution data corresponding to a specified sub-band that meets the adjustment condition in at least two sub-bands, an adjustment parameter is determined based on the sub-band energy distribution data corresponding to the specified sub-band, and the sub-band energy data of the specified sub-band is adjusted to obtain the target enhanced audio.

[0071] Optionally, the adjustment condition is a pre-set condition, for example: the adjustment condition is a preset energy threshold; or, the adjustment condition is a condition determined in real time based on multiple sub-band energy distribution data, for example: the adjustment condition is: the sub-band energy distribution data of the specified sub-band is greater than the average value of multiple sub-band energy distribution data, etc.

[0072] In an illustrative manner, after the target audio is subjected to frequency domain dimension analysis and time domain dimension analysis by the above method, the subband energy distribution data corresponding to at least two subbands are determined. Optionally, the subband energy distribution data corresponding to at least two subbands are compared with preset adjustment conditions to determine whether to adjust the subband energy data corresponding to at least two subbands. For example, the subband energy distribution data corresponding to a certain subband in different audio frames are compared with the adjustment conditions.

[0073] In an optional embodiment, when there is sub-band energy distribution data corresponding to a certain sub-band that meets the adjustment conditions, the sub-band energy data of the sub-band is adjusted; or, when the sub-band energy distribution data corresponding to a certain sub-band does not meet the adjustment conditions, the sub-band energy data of the sub-band is not adjusted.

[0074] In an illustrative manner, the sub-band that meets the adjustment condition is used as the designated sub-band, that is, the designated sub-band is used to indicate the sub-band that meets the adjustment condition in at least two sub-bands. For example, the sub-band energy distribution data corresponding to the designated sub-band is greater than a preset energy threshold; or the sub-band energy distribution data corresponding to the designated sub-band is less than 3 times the preset energy threshold, etc.

[0075] Optionally, based on the frequency band characteristics of the designated sub-frequency band (the sub-frequency band energy distribution data of the designated sub-frequency band meets the adjustment conditions), the designated sub-frequency band may be one of at least two sub-frequency bands, or may be multiple sub-frequency bands of at least two sub-frequency bands (e.g., at least two sub-frequency bands are designated sub-frequency bands).

[0076] In an optional embodiment, when adjusting the sub-band energy data of the designated sub-frequency band, an adjustment parameter is determined by sub-band energy distribution data corresponding to the designated sub-frequency band, and the sub-band energy data of the designated sub-frequency band is adjusted by the adjustment parameter.

[0077] Schematically, when obtaining the subband energy distribution data corresponding to a specified subband, based on the time domain relationship corresponding to the subband energy distribution data, the subband energy distribution data corresponding to the specified subband in different audio frames are obtained. For example: taking the specified subband as subband A as an example for explanation (the energy distribution data corresponding to subband A meets the adjustment conditions), when obtaining the subband energy distribution data corresponding to subband A, subband energy distribution data 1 corresponding to subband A in audio frame 1 and subband energy distribution data 2 corresponding to subband A in audio frame 2 are obtained, and subband energy distribution data 1 and subband energy distribution data 2 are used as the subband energy distribution data corresponding to subband A.

[0078] Optionally, according to changes in sub-band energy distribution data 1 and sub-band energy distribution data 2 corresponding to sub-band A, an adjustment parameter corresponding to sub-band A is determined, and the sub-band energy data of sub-band A is adjusted by adjusting the adjustment parameter.

[0079] Illustratively, according to the above-mentioned method for determining the adjustment parameters, the adjustment parameters corresponding to different designated sub-bands are determined, and the sub-band energy data of the designated sub-bands corresponding to the different designated sub-bands are adjusted with the adjustment parameters corresponding to the different designated sub-bands, thereby obtaining the adjusted sub-band energy data corresponding to the different designated sub-bands.

[0080] In an optional embodiment, the target enhanced audio is obtained based on the adjusted sub-band energy data corresponding to the designated sub-band.

[0081] Schematically, taking any audio frame as an example for analysis, according to the relationship between the specified sub-band and at least two sub-bands in the audio frame, when obtaining the target enhanced audio according to the adjusted sub-band energy data, at least one of the following situations is included.

[0082] (1) Some of the at least two sub-bands are designated sub-bands.

[0083] In an illustrative manner, when the designated sub-band is a part of the at least two sub-bands, it means that the sub-band energy distribution data corresponding to the part of the sub-band meets the adjustment condition, then the corresponding adjustment parameter is determined according to the sub-band energy distribution data corresponding to the part of the sub-band, and when the sub-band energy data of the designated sub-band is adjusted by the adjustment parameter, the sub-band energy data of the designated sub-band corresponding to the different adjustment parameters are adjusted, so that the sub-band energy data of the part of the sub-band that becomes the designated sub-band is adjusted. However, the sub-band energy distribution data of the remaining sub-bands except the designated sub-band in the at least two sub-bands do not meet the adjustment condition, so the remaining sub-bands do not need to determine the adjustment parameters and perform the sub-band energy data adjustment process.

[0084] That is: for at least two sub-bands, the sub-band energy data corresponding to some sub-bands (designated sub-bands) are adjusted, and the adjusted sub-band energy data corresponding to different designated sub-bands are obtained; and the sub-band energy data corresponding to the remaining sub-bands (at least two sub-bands except the designated sub-bands) are not adjusted.

[0085] Optionally, when obtaining the target enhanced audio based on the adjusted sub-band energy data, the frequency band area of ​​the remaining sub-bands that have not been adjusted in the target audio is taken into consideration, and the adjusted sub-band energy data corresponding to the adjusted designated sub-band and the sub-band energy data corresponding to the remaining sub-bands that have not been adjusted are comprehensively considered to obtain the enhanced target enhanced audio.

[0086] In an optional embodiment, the subband energy data of the designated subband is adjusted to obtain the adjusted subband energy data corresponding to the designated subband, wherein the at least two subbands also include candidate discarded subbands for which energy adjustment has not been performed, that is, the candidate discarded subbands are used to indicate subbands other than the designated subband in the at least two subbands.

[0087] Optionally, for audio frames in the target audio, in response to the fact that the frequency band share of the energy-adjusted designated sub-band in at least two sub-bands exceeds a preset share threshold, and the candidate discarded sub-band is outside the frequency domain range corresponding to the human voice, the sub-band energy data of the energy-adjusted designated sub-band is retained, and the sub-band energy data of the candidate discarded sub-band that has not been energy adjusted is discarded to obtain the target enhanced audio.

[0088] Schematically, after determining that the frequency band proportion of a designated sub-frequency band in at least two sub-frequency bands reaches a preset proportion threshold, it is determined whether candidate discarded sub-frequency bands other than the designated sub-frequency band in at least two sub-frequency bands are outside the frequency domain range corresponding to human voice, wherein the frequency domain range corresponding to human voice is 20HZ-20KHZ.

[0089] When the candidate discarded sub-band is outside the frequency domain range corresponding to the human voice, the candidate discarded sub-band is used as the discarded sub-band, that is, the candidate discarded sub-band outside the frequency domain range corresponding to the human voice is discarded; when the candidate discarded sub-band is within the frequency domain range corresponding to the human voice, the candidate discarded sub-band is not discarded, that is, the candidate discarded sub-band within the frequency domain range corresponding to the human voice is retained. For example: the preset proportion threshold is 95%, when 95% of the frequency bands in at least two sub-bands are designated sub-bands, the designated sub-bands are used as candidate discarded sub-bands, and it is determined whether the candidate discarded sub-band is outside the frequency domain range corresponding to the human voice, so as to determine whether to discard or retain the candidate discarded sub-band.

[0090] Illustratively, when a candidate discarded sub-band is a sub-band outside the frequency domain range corresponding to human voice, the candidate discarded sub-band is discarded, thereby determining the target enhanced audio based on the adjusted sub-band energy data corresponding to the designated sub-band; or, when a candidate discarded sub-band is a sub-band within the frequency domain range corresponding to human voice, the candidate discarded sub-band is retained, thereby determining the target enhanced audio based on the adjusted sub-band energy data corresponding to the designated sub-band and the sub-band energy data corresponding to the retained sub-band, etc.

[0091] (2) All of the at least two sub-bands are designated sub-bands.

[0092] Illustratively, when all sub-bands of at least two sub-bands are designated sub-bands, adjustment parameters corresponding to the at least two sub-bands are determined according to the sub-band energy distribution data corresponding to the at least two sub-bands, and the sub-band energy data of the sub-bands corresponding to the at least two sub-bands are adjusted by the adjustment parameters, so as to obtain adjusted sub-band energy data corresponding to the at least two sub-bands. Optionally, based on the adjusted sub-band energy data corresponding to the at least two sub-bands, the target enhanced audio is obtained.

[0093] Illustratively, an energy distribution curve is determined based on the adjusted sub-band energy data corresponding to at least two sub-frequency bands, a sound signal is restored based on the energy distribution curve, and a target enhanced audio is determined based on the restored sound signal.

[0094] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0095] In summary, the target audio for speech enhancement is divided into frequency bands along the frequency domain dimension to obtain at least two sub-bands, and the sub-band energy data and sub-band energy distribution data corresponding to different sub-bands are obtained. When there is sub-band energy distribution data corresponding to a specified sub-band that meets the adjustment conditions, the adjustment parameters are determined based on the sub-band energy distribution data corresponding to the specified sub-band, and the sub-band energy data of the specified sub-band is adjusted using the adjustment parameters, thereby effectively avoiding the purposeless adjustment of the entire speech signal of the target audio. By judging whether the sub-band energy distribution data of the sub-band meets the adjustment conditions, the sub-band energy data corresponding to the specified sub-band that does not meet the adjustment conditions is selectively adjusted, and then the sub-band energy data of the specified sub-band is adjusted using the adjustment parameters determined by the sub-band energy distribution data corresponding to the specified sub-band, and the enhanced target enhanced audio is obtained based on the adjusted sub-band energy data, while fully considering the characteristics of the target audio, the quality of speech enhancement is improved.

[0096] In an optional embodiment, whether to adjust the sub-band energy data is determined based on the sub-band energy long-term distribution data included in the sub-band energy distribution data. That is, the sub-band energy distribution data includes the sub-band energy long-term distribution data, and the sub-band energy long-term distribution data is used to indicate the change of the sub-band energy data of two adjacent audio frames. Schematically, as Figure 4 As shown above Figure 2 The illustrated embodiment may also be implemented as the following steps 410 to 470 .

[0097] Step 410, obtaining target audio.

[0098] The target audio is the audio data to be speech enhanced.

[0099] Illustratively, audio is used to indicate data with audio information, such as a piece of music, a voice message, etc. Optionally, a device with a built-in or external voice collection component such as a terminal or a recorder is used to acquire the target audio; or, the target audio is audio data during a call, etc.

[0100] Step 420 , dividing the target audio into frequency bands along the frequency domain dimension to obtain at least two sub-bands.

[0101] Indicatively, according to the frequency distribution range corresponding to the target audio, the target audio is divided into frequency bands along the frequency domain dimension.

[0102] In an optional embodiment, a fixed frequency bandwidth is used to divide the target audio into frequency bands.

[0103] For example, the frequency distribution range of the target audio is 100 Hz-399 Hz. When the target audio is divided into frequency bands, each frequency band width is 100 Hz, and three sub-bands are obtained, including a sub-band with a frequency band range of 100 Hz-199 Hz, a sub-band with a frequency band range of 200 Hz-299 Hz, and a sub-band with a frequency band range of 300 Hz-399 Hz.

[0104] In an optional embodiment, the target audio is divided into frequency bands using a critical frequency band division standard.

[0105] Among them, the critical band is used to indicate the frequency bandwidth of the auditory filter generated by the cochlear structure, that is, the critical band is the sound frequency band, in which the perception of the first single tone will be disturbed by the auditory masking of the second single tone, thereby dividing 24 critical bands, corresponding to the 24 frequency points that the human ear structure can resonate, and the 24 critical bands are usually represented in the sound domain (Bark domain). That is, the critical band division standard is the preset frequency band division standard.

[0106] For example: the frequency distribution range corresponding to the target audio is 20HZ-400HZ. When the target audio is divided into frequency bands, according to the division standard of the critical band, 4 sub-bands are obtained, including a sub-band with a frequency band range of 20HZ-100HZ (frequency band width is 80, frequency band center is 50) (critical frequency band 1), a sub-band with a frequency band range of 100HZ-200HZ (frequency band width is 100, frequency band center is 150) (critical frequency band 2), a sub-band with a frequency band range of 200HZ-300HZ (frequency band width is 100, frequency band center is 250) (critical frequency band 3) and a sub-band with a frequency band range of 300HZ-400HZ (frequency band width is 100, frequency band center is 350) (critical frequency band 4).

[0107] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0108] Step 430: Obtain sub-band energy data corresponding to at least two sub-frequency bands.

[0109] The sub-band energy data is used to indicate the frequency variation of the audio frame in the target audio along the frequency domain dimension within the sub-band.

[0110] Schematically, after sampling the target audio, multiple sampling points corresponding to the target audio are obtained. When determining the sub-band energy data corresponding to at least two sub-bands, the energy data corresponding to the multiple sampling points are integrated to determine the sub-band energy data corresponding to the sub-band; or, the multiple sampling points are band-pass filtered, and then the energy data corresponding to the multiple filtered sampling points in the sub-band are integrated to obtain the sub-band energy data corresponding to the sub-band, etc.

[0111] Optionally, based on the above-mentioned method for determining sub-band energy data, sub-band energy data corresponding to multiple sub-bands under the same audio frame are determined, and sub-band energy data corresponding to multiple sub-bands under other audio frames are determined, so as to obtain sub-band energy data corresponding to different sub-bands under different audio frames.

[0112] In an optional embodiment, the above method is used to determine sub-band energy values ​​corresponding to different sub-frequency bands, a logarithmic operation is performed on the sub-band energy values, and the result after the logarithmic operation is used as the sub-band energy data.

[0113] In an illustrative manner, the subband energy value corresponding to the current audio frame is obtained as x, and then the logarithm of the subband energy value is taken, for example: log10(x), and the value after taking the logarithm is used as the subband energy data, that is, the subband energy data is log10(x). Optionally, Eb(i,k) indicates the subband energy data of the kth subband corresponding to the i-th audio frame.

[0114] Step 440: Obtain sub-band energy data corresponding to at least two sub-frequency bands in the i-th audio frame and sub-band energy distribution data corresponding to at least two sub-frequency bands in the (i-1)-th audio frame.

[0115] The i-th audio frame and the (i-1)-th audio frame are two adjacent audio frames distributed along the time domain dimension, and i is a positive integer greater than 1.

[0116] Illustratively, the above-mentioned sub-band energy acquisition method is used to obtain the sub-band energy data corresponding to at least two sub-bands in the i-th audio frame. Optionally, the sub-band energy distribution data corresponding to the i-1-th audio frame is obtained, wherein the sub-band energy distribution data is used to indicate the frequency distribution of the target audio in at least two sub-bands.

[0117] That is, when determining the subband energy distribution data corresponding to at least two subbands in the current audio frame, an iterative determination method is adopted to jointly determine the subband energy distribution data corresponding to at least two subbands in the current audio frame through the subband energy distribution data corresponding to at least two subbands in the previous audio frame and the subband energy data corresponding to at least two subbands in the current audio frame.

[0118] Illustratively, the above method is used to determine sub-band energy distribution data corresponding to different audio frames.

[0119] The first audio frame is the first audio frame of the target audio in the time domain dimension. Optionally, when determining the subband energy distribution data corresponding to different subbands in the first audio frame, the subband energy data corresponding to different subbands in the first audio frame are used as the corresponding subband energy distribution data. For example: in the first audio frame, the subband energy data corresponding to subband A is x, then the subband energy data x corresponding to subband A in the first audio frame is used as the subband energy distribution data corresponding to subband A in the first audio frame.

[0120] Alternatively, when determining the subband energy distribution data corresponding to different subbands in the first audio frame, the subband energy data corresponding to different subbands in the first audio frame are processed respectively, and the processed data are used as the subband energy distribution data corresponding to different subbands. For example, in the first audio frame, the subband energy data corresponding to subband A is x, the preset parameter is multiplied by the subband energy data x, and the product is used as the subband energy distribution data corresponding to subband A in the first audio frame.

[0121] Optionally, when determining the sub-band energy distribution data corresponding to different sub-frequency bands in audio frames other than the first audio frame, a preset energy data calculation method is used to determine the sub-band energy distribution data.

[0122] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0123] Step 450, weightedly fuse the subband energy data corresponding to at least two subbands in the i-th audio frame and the subband energy long-time distribution data corresponding to at least two subbands in the i-1-th audio frame with a first preset weight to determine the subband energy long-time distribution data corresponding to at least two subbands in the i-th audio frame.

[0124] Optionally, after obtaining the subband energy data corresponding to the i-th audio frame and the subband energy long-term distribution data corresponding to the i-1-th audio frame, the subband energy long-term distribution data corresponding to the i-th audio frame is determined according to a first preset weight. The first preset weight is a predetermined weight value, which is used to balance the weights of the subband energy data corresponding to the i-th audio frame and the subband energy long-term distribution data corresponding to the i-1-th audio frame when determining the subband energy long-term distribution data corresponding to the i-th audio frame.

[0125] Illustratively, the following calculation formula for sub-band energy distribution data is used to determine the sub-band energy long-term distribution data corresponding to the i-th audio frame.

[0126] Eb_lt(i,k)=a*Eb_lt(i-1,k)+(1-a)*Eb(i,k)a=0.993

[0127] Among them, i is used to indicate an audio frame; k is used to indicate a sub-band; a is used to indicate a first preset weight; Eb_lt(i,k) is used to indicate the long-term distribution data of the sub-band energy corresponding to the k-th frequency band in the i-th audio frame, lt is the abbreviation of long-term, which is used to describe the iterative relationship between the long-term distribution data of the sub-band energy corresponding to different audio frames in the time domain dimension; Eb_lt(i-1,k) is used to indicate the long-term distribution data of the sub-band energy corresponding to the k-th frequency band in the i-1-th audio frame; Eb(i,k) is used to indicate the sub-band energy data corresponding to the k-th frequency band in the i-th audio frame.

[0128] Optionally, the calculation formula for the sub-band energy long-term distribution data is used to determine the sub-band energy long-term distribution data corresponding to different sub-frequency bands in different audio frames.

[0129] Step 460: In response to the fact that the sub-band energy long-term distribution data corresponding to the designated sub-frequency band in at least two sub-frequency bands reaches a preset hearing threshold, determining an adjustment parameter.

[0130] Optionally, taking any audio frame as an example, after determining the long-term distribution data of sub-band energy corresponding to at least two sub-bands in the audio frame, the sub-band energy distribution data corresponding to the at least two sub-bands are compared with a preset auditory threshold, and based on the comparison result, it is determined whether to adjust the sub-band energy data corresponding to the at least two sub-bands.

[0131] The preset hearing threshold is used to indicate the adjustment condition, which is a pre-set numerical condition used to avoid the occurrence of auditory fatigue.

[0132] The main reason for auditory fatigue is that the sound signal being listened to has continuous energy concentration and overload in one or some sub-bands in the frequency domain dimension, such as Figure 5 As shown in FIG. 5 , it is a schematic frequency distribution diagram of the target audio, wherein the horizontal axis 510 is used to indicate the time dimension, and the vertical axis 520 is used to indicate the frequency distribution. Figure 5 As shown, there are obvious bright lines in the low-frequency part 530 (indicating that the energy of the target audio in the low frequency is too large), and the listener's ears are subjected to the signal stimulation from this frequency band (low-frequency band) for a long time, causing the hearing threshold (audio threshold) to move up. At this time, the audio information of other frequency bands is insensitive due to the adjustment of the hearing threshold, so it is not clear to hear, and it will also make people tired and unable to concentrate after listening for a long time, which leads to the occurrence of auditory fatigue.

[0133] In an optional embodiment, different preset hearing thresholds are determined for different sub-frequency bands according to the principle of auditory equal loudness curves.

[0134] Among them, the auditory equal loudness curve is a curve that is equal to the subjective perception of sound loudness (loudness level) obtained by measuring the actual sound perception of the human ear. Since different sub-bands correspond to different frequency ranges, when determining the preset auditory thresholds corresponding to different sub-bands, the auditory thresholds corresponding to the frequency range are determined, that is, the preset auditory thresholds corresponding to different sub-bands are determined.

[0135] Optionally, reaching a preset hearing threshold is used to indicate that the long-term distribution data of the sub-band energy is greater than a preset hearing threshold; or, reaching a preset hearing threshold is used to indicate that the long-term distribution data of the sub-band energy is greater than or equal to a preset hearing threshold.

[0136] In an illustrative manner, the sub-band corresponding to the sub-band energy long-term distribution data reaching the preset hearing threshold is used as the designated sub-band. For example, the sub-band energy long-term distribution data corresponding to the designated sub-band is greater than the preset hearing threshold.

[0137] In an optional embodiment, when adjusting the sub-band energy data of the designated sub-frequency band, an adjustment parameter is determined by sub-band energy distribution data corresponding to the designated sub-frequency band, and the sub-band energy data of the designated sub-frequency band is adjusted by the adjustment parameter.

[0138] Step 470: Adjust the sub-band energy data of the designated sub-frequency band based on the adjustment parameter to obtain the target enhanced audio.

[0139] Optionally, the same sub-band corresponds to different adjustment parameters in different audio frames, and any audio frame is used as an example for explanation. In an illustrative example, after determining the adjustment parameters corresponding to different sub-bands, the sub-band energy data of the corresponding sub-bands are adjusted according to the adjustment parameters corresponding to the different sub-bands.

[0140] In an optional embodiment, the sub-band energy data corresponding to the specified sub-frequency band is adjusted by adjusting the parameters to determine the energy adjustment gain corresponding to the specified sub-frequency band.

[0141] The energy adjustment gain is used to indicate the data ratio adjustment status of the sub-band energy data.

[0142] Illustratively, after obtaining the adjustment parameter, the adjustment parameter is converted to obtain an energy adjustment gain corresponding to a specified sub-band, and speech enhancement is performed on the target audio based on the energy adjustment gain to obtain the target enhanced audio.

[0143] To summarize, at least two sub-bands corresponding to the target audio are obtained through frequency band segmentation, and sub-band energy data and sub-band energy distribution data corresponding to different sub-bands are obtained. The designated sub-band that does not meet the adjustment conditions is determined through the sub-band energy distribution data of the sub-band, and then the sub-band energy distribution data corresponding to the designated sub-band is used to determine the adjustment parameters corresponding to different designated sub-bands, so that the sub-band energy data of the designated sub-band is selectively adjusted using the adjustment parameters, and based on the adjusted sub-band energy data, the enhanced target enhanced audio is obtained, which improves the quality of speech enhancement while fully considering the characteristics of the target audio.

[0144] In an embodiment of the present application, subband energy long-time distribution data included in the subband energy distribution data is introduced. Through the subband energy long-time distribution data, the change of subband energy data of two adjacent audio frames can be determined, and then the subband energy long-time distribution data with an iterative relationship in the time domain dimension is used to compare the subband energy long-time distribution data with a preset auditory threshold. When the subband energy long-time distribution data reaches the preset auditory threshold, the subband energy data corresponding to the specified subband is adjusted by adjusting the parameters, so that the subband energy data corresponding to the specified subband is limited to a certain value, thereby avoiding the occurrence of auditory fatigue and the like due to excessive subband energy data corresponding to the specified subband, reducing the fatigue of the object when listening to the target audio, and selectively adjusting the subband energy data corresponding to the target audio.

[0145] In an optional embodiment, the adjustment parameter is determined based on the sub-band energy high-order distribution data and the sub-band energy low-order distribution data included in the sub-band energy distribution data. Figure 6 As shown above Figure 2 The illustrated embodiment may also be implemented as the following steps 610 to 690 .

[0146] Step 610, obtaining target audio.

[0147] The target audio is the audio data to be speech enhanced.

[0148] Step 610 has been described in the above step 210 and will not be repeated here.

[0149] Step 620 , dividing the target audio into frequency bands along the frequency domain dimension to obtain at least two sub-bands.

[0150] In an optional embodiment, based on a preset frequency band division standard, the target audio is divided into frequency bands along the frequency domain dimension to obtain at least two sub-bands. In an exemplary embodiment, the target audio is divided into frequency bands using a predetermined critical frequency band division standard to obtain at least two sub-bands.

[0151] Step 620 has been described in the above steps 220 and 420 and will not be repeated here.

[0152] Step 630: Obtain sub-band energy data corresponding to at least two sub-frequency bands.

[0153] The sub-band energy data is used to indicate the frequency variation of the audio frame in the target audio along the frequency domain dimension within the sub-band.

[0154] Indicatively, the subband energy values ​​corresponding to different audio frames are obtained as x, and then the logarithms of the different subband energy values ​​are taken respectively, and the values ​​after taking the logarithms are used as subband energy data, so as to obtain the subband energy data corresponding to different sub-bands in different audio frames.

[0155] Step 630 has been described in the above-mentioned steps 230 and 430 and will not be repeated here.

[0156] Step 640: Obtain sub-band energy data corresponding to at least two sub-frequency bands in the i-th audio frame and sub-band energy distribution data corresponding to at least two sub-frequency bands in the (i-1)-th audio frame.

[0157] Wherein, i is a positive integer greater than 1, and the sub-band energy distribution data is used to indicate the frequency distribution of the target audio in at least two sub-bands.

[0158] The i-th audio frame and the (i-1)-th audio frame are two adjacent audio frames distributed along the time domain dimension, and i is a positive integer greater than 1.

[0159] Schematically, when determining the sub-band energy distribution data corresponding to the current frame (the i-th frame) audio frame, an iterative determination method is adopted to jointly determine the sub-band energy distribution data corresponding to the current frame audio frame through the sub-band energy distribution data corresponding to the previous frame (the i-1-th frame) audio frame and the sub-band energy data corresponding to the current frame audio frame (the i-th frame).

[0160] Step 650, based on the subband energy data corresponding to at least two subbands in the i-th frame audio frame, the subband energy long-time distribution data corresponding to at least two subbands in the i-1-th frame audio frame, and the first preset weight, determine the subband energy long-time distribution data corresponding to at least two subbands in the i-th frame audio frame.

[0161] Optionally, subband energy data corresponding to at least two subbands in the i-th audio frame and subband energy long-time distribution data corresponding to at least two subbands in the i-1-th audio frame are weightedly fused using a predetermined first preset weight, thereby determining the subband energy long-time distribution data corresponding to at least two subbands in the i-th audio frame.

[0162] Illustratively, the sub-band energy distribution data corresponding to the k-th sub-band in the i-th audio frame is determined using the calculation formula for the sub-band energy distribution data shown in step 450. Similarly, the sub-band energy distribution data corresponding to different sub-bands in different audio frames are determined using the calculation formula for the sub-band energy distribution data.

[0163] Step 660, based on the subband energy data corresponding to at least two subbands in the i-th audio frame, the subband energy high-order distribution data corresponding to at least two subbands in the i-1-th audio frame, and the second preset weight, obtain the subband energy high-order distribution data corresponding to at least two subbands in the i-th audio frame.

[0164] Among them, the sub-band energy distribution data includes sub-band energy high-order distribution data, and the sub-band energy high-order distribution data is used to indicate the data comparison between the sub-band energy data of two adjacent audio frames of the i-th audio frame and the sub-band energy high-order distribution data of the i-1-th audio frame, wherein the i-th audio frame and the i-th audio frame are adjacent audio frames.

[0165] Optionally, after obtaining the subband energy data corresponding to at least two subbands in the i-th audio frame, the subband energy data corresponding to at least two subbands in the i-th audio frame are compared with the subband energy high-order distribution data corresponding to at least two subbands in the i-1-th audio frame, and the preset weights for the weighted fusion process are determined according to the comparison results.

[0166] Schematically, the sub-band energy data of the kth frequency band in the i-th audio frame is expressed as Eb(i, k), and the sub-band energy high-order distribution data corresponding to the kth frequency band in the i-1-th audio frame is expressed as Eb_up(i-1, k), which is used to describe the iterative relationship between the sub-band energy high-order distribution data corresponding to different audio frames in the time domain dimension, where up is used to indicate the high order.

[0167] Optionally, taking a specified sub-band as an example for analysis, after obtaining Eb(i, k) and Eb up(i-1，k) Then, Eb(i, k) and Eb up(i-1，k) A comparison is performed, and according to the data comparison result, the sub-band energy data corresponding to the i-th audio frame and the sub-band energy high-order distribution data corresponding to the i-1-th audio frame are weightedly fused using the second preset weight to obtain the sub-band energy high-order distribution data corresponding to the i-th audio frame.

[0168] Optionally, after obtaining Eb(i, k) and Eb up(i-1，k) Then, Eb(i, k) and Eb up(i-1，k) According to the data comparison result, different second preset weights are used to perform weighted fusion on the sub-band energy data corresponding to the i-th audio frame and the sub-band energy high-order distribution data corresponding to the i-th audio frame, so as to obtain the sub-band energy high-order distribution data corresponding to the i-th audio frame. That is, according to Eb(i, k) and Eb up(i-1，k) There is a numerical relationship between the values ​​of the second preset weights.

[0169] Illustratively, the sub-band energy high-bit distribution data corresponding to the i-th audio frame is determined using the following calculation formula for the sub-band energy high-bit distribution data.

[0170] E up(i，k) =b*Eb up(i-1，k) +(1-b)*Eb(i,k)

[0171] Wherein, i is used to indicate an audio frame; k is used to indicate a sub-band; b is used to indicate a second preset weight; Eb up(i，k) Used to indicate the high-order distribution data of the sub-band energy corresponding to the k-th frequency band in the i-th audio frame; Eb up(i-1，k)It is used to indicate the high-order distribution data of the sub-band energy corresponding to the k-th frequency band in the i-1-th audio frame; Eb(i, k) is used to indicate the sub-band energy data corresponding to the k-th frequency band in the i-th audio frame.

[0172] Optionally, according to Eb(i, k) and Eb up(i-1，k) There is a numerical comparison relationship between the values ​​of the second preset weight b. Schematically, the second preset weight b is expressed as follows.

[0173]

[0174] Among them, when Eb(i, k) is less than Eb up(i-1，k) When Eb(i, k) is not less than (greater than or equal to) Eb up(i-1，k) When the second preset weight b is taken as 0.95, that is, when determining the high-order distribution data of sub-band energy corresponding to the k-th frequency band in the i-th audio frame, the weight of the high-order distribution data of sub-band energy corresponding to the k-th frequency band in the i-1-th audio frame has little influence.

[0175] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0176] Step 670, based on the subband energy data corresponding to at least two subbands in the i-th frame audio frame, the subband energy low-bit distribution data corresponding to at least two subbands in the i-1-th frame audio frame, and the third preset weight, obtain the subband energy low-bit distribution data corresponding to at least two subbands in the i-th frame audio frame.

[0177] The subband energy distribution data includes subband energy low-bit distribution data, and the subband energy low-bit distribution data is used to indicate a data comparison between the subband energy data of the i-th audio frame and the subband energy low-bit distribution data of the i-1-th audio frame.

[0178] Optionally, after obtaining the subband energy data corresponding to at least two subbands in the i-th audio frame, the subband energy data corresponding to at least two subbands in the i-th audio frame are compared with the subband energy low-bit distribution data corresponding to at least two subbands in the i-1-th audio frame, and the preset weights for the weighted fusion process are determined according to the comparison results.

[0179] Schematically, the sub-band energy data of the k-th frequency band in the i-th audio frame is represented as Eb(i, k), and the sub-band energy low-order distribution data corresponding to the k-th frequency band in the i-1-th audio frame is represented as Ebdw(i，k) , where dw is used to indicate the low bit, which is used to describe the iterative relationship of the low bit distribution data of sub-band energy corresponding to different audio frames in the time domain dimension.

[0180] Optionally, taking a specified sub-band as an example for analysis, after obtaining Eb(i, k) and Eb dw(i-1，k) Then, Eb(i, k) and Eb dw(i-1，k) A comparison is performed, and according to the data comparison result, the sub-band energy data corresponding to the i-th audio frame and the sub-band energy low-bit distribution data corresponding to the i-1-th audio frame are weightedly fused using a third preset weight to obtain the sub-band energy low-bit distribution data corresponding to the i-th audio frame.

[0181] Optionally, after obtaining Eb(i, k) and Eb dw(i-1，k) Then, Eb(i, k) and Eb dw(i-1，k) According to the data comparison result, different third preset weights are used to perform weighted fusion on the sub-band energy data corresponding to the i-th audio frame and the sub-band energy low-bit distribution data corresponding to the i-th audio frame, so as to obtain the sub-band energy low-bit distribution data corresponding to the i-th audio frame. That is, according to Eb(i, k) and Eb dw(i-1，k) There is a numerical relationship between the values ​​of the third preset weights.

[0182] Illustratively, the following calculation formula for sub-band energy low-bit distribution data is used to determine the sub-band energy low-bit distribution data corresponding to the i-th audio frame.

[0183] E dw(i，k) =c*Eb dw(i-1，k) +(1-c)*Eb(i,k)

[0184] Wherein, i is used to indicate an audio frame; k is used to indicate a sub-band; c is used to indicate a third preset weight; Eb dw(i，k) Used to indicate the low-bit distribution data of the sub-band energy corresponding to the k-th frequency band in the i-th audio frame; Eb dw(i-1，k) It is used to indicate the low-bit distribution data of the sub-band energy corresponding to the k-th frequency band in the i-1-th audio frame; Eb(i, k) is used to indicate the sub-band energy data corresponding to the k-th frequency band in the i-th audio frame.

[0185] Optionally, according to Eb(i, k) and Eb dw(i-1，k) There are differences in the numerical comparison relationship of the third preset weight c. Schematically, the third preset weight c is expressed as follows.

[0186]

[0187] Among them, when Eb(i, k) is greater than Eb dw(i-1，k)When Eb(i, k) is not greater than (less than or equal to) Eb dw(i-1，k) When the third preset weight c is taken as 0.95, that is, when determining the low-order distribution data of sub-band energy corresponding to the k-th frequency band in the i-th audio frame, the weight of the low-order distribution data of sub-band energy corresponding to the k-th frequency band in the i-1-th audio frame has little influence.

[0188] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0189] Step 680 : determining an adjustment parameter corresponding to the designated sub-frequency band based on the high-order distribution data of the sub-frequency band energy corresponding to the designated sub-frequency band, the low-order distribution data of the sub-frequency band energy corresponding to the designated sub-frequency band, and a preset auditory threshold.

[0190] The preset auditory threshold is used to assist in limiting the data range of the sub-band energy data corresponding to the specified sub-frequency band.

[0191] Optionally, when determining the sub-band energy high-order distribution data Eb corresponding to the specified sub-band up(i，k) And the sub-band energy low-order distribution data Eb corresponding to the specified sub-band dw(i，k) Then, according to the preset hearing threshold, Eb up(i，k) and Eb dw(i，k) , determine the adjustment parameters corresponding to the specified sub-band.

[0192] Schematically, the calculation formula for the adjustment parameters is shown below.

[0193]

[0194] Wherein, Thrd is used to indicate a preset hearing threshold. Optionally, the adjustment parameter corresponding to the specified sub-band is determined by the preset hearing threshold, and when the sub-band energy data corresponding to the specified sub-band is adjusted by the adjustment parameter, the sub-band energy data corresponding to the adjusted specified sub-band is limited to a value within the preset hearing threshold.

[0195] Step 690: Adjust the sub-band energy data of the designated sub-frequency band based on the adjustment parameter to obtain the target enhanced audio.

[0196] Indicative, such as Figure 7As shown, it is a schematic diagram of adjusting the sub-band energy data Eb(i, k) corresponding to the k-th frequency band in the i-th audio frame, wherein the horizontal axis represents the sub-band energy data Eb(i, k) corresponding to the k-th frequency band in the i-th audio frame; the vertical axis represents the adjusted sub-band energy data Eb′(i, k) corresponding to the k-th frequency band in the i-th audio frame, wherein point A 710 is used to indicate the sub-band energy low-order distribution data Eb corresponding to the sub-band dw(i，k) Point B 720 is used to indicate the sub-band energy high-order distribution data Eb corresponding to the sub-band up(i，k) ; Point C 730 is used to indicate the adjusted maximum sub-band energy data Eb′(i, k) corresponding to the k-th frequency band in the i-th audio frame, which is used to limit the long-term data of the sub-band energy based on the preset auditory threshold, so the adjusted sub-band energy data Eb′(i, k) is less than the preset auditory threshold Thrd; Point D 740 is used to indicate the adjusted minimum sub-band energy data Eb′(i, k) corresponding to the k-th frequency band in the i-th audio frame, wherein Point A 710, Point B 720, Point C 730 and Point D 740 are on a slant line, the slope of the slant line is the above-mentioned adjustment parameter, and the slant line expression of the slant line is as follows.

[0197]

[0198] In schematic form, the horizontal axis Eb(i, k) is taken as input and the vertical axis Eb′(i, k) is taken as output. The input Eb(i, k) is adjusted by the above-mentioned adjustment rule of the oblique line to obtain the output Eb′(i, k). That is, at point A 710, the input Eb(i, k) is Eb dw(i，k) According to the above slash expression, the output Eb′(i, k) is determined to be 0.2*Thrd; at point B 720, the input Eb(i, k) is Eb up(i，k) According to the above slash expression, the output Eb′(i, k) is determined to be 0.8*Thrd; at point C 730, the input Eb(i, k) is (4*Eb up -Eb dw ) / 3, according to the above slash expression, the output Eb′(i, k) is determined to be Thrd; at point D 740, the input Eb(i, k) is (4*Eb dw -Eb up ) / 3, according to the above slash expression, the output Eb′(i, k) is determined to be 0, etc.

[0199] In an illustrative manner, the above-mentioned slash expression is used to determine the corresponding output Eb′(i, k) according to different inputs Eb(i, k), thereby realizing the adjustment process of adjusting the sub-band energy data. Optionally, the adjustment parameter corresponding to the specified sub-band is determined by a preset hearing threshold, and when the sub-band energy data corresponding to the specified sub-band is adjusted by the adjustment parameter, the sub-band energy data corresponding to the adjusted specified sub-band is limited to a value within the preset hearing threshold.

[0200] In an optional embodiment, after obtaining the adjusted sub-band energy data Eb′(i, k), the energy adjustment gain corresponding to different sub-bands is determined according to the adjusted sub-band energy data Eb′(i, k) and the sub-band energy data Eb(i, k) before adjustment, wherein the expression of the sub-band energy gain and loss is as follows.

[0201]

[0202] Wherein, gain(i,k) is used to indicate the sub-band energy gain or loss corresponding to the k-th sub-band in the i-th audio frame; and sqrt is used to indicate the calculation function of the positive square root.

[0203] In an optional embodiment, a gain transformation is performed on the energy adjustment gain corresponding to the specified sub-frequency band to determine the frequency gain corresponding to the mid-frequency point of the specified sub-frequency band; and the target enhanced audio is obtained based on the product of the frequency gain corresponding to the mid-frequency point of the specified sub-frequency band and the frequency amplitude corresponding to the mid-frequency point of the specified sub-frequency band.

[0204] Indicatively, after obtaining the energy adjustment gain corresponding to the specified sub-band, the energy adjustment gain corresponding to the specified sub-band is inversely transformed through the Bark domain to determine the frequency gain corresponding to the frequency point in the specified sub-band. The frequency point is a point distributed in the frequency band. Optionally, the sampling point corresponding to the specified sub-band in the above-mentioned calculation of the sub-band energy data is used as the frequency point; or, any point or multiple points in the specified sub-band are used as the frequency point, etc. The frequency gain is used to indicate the proportional adjustment of the sub-band energy data corresponding to the frequency point.

[0205] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0206] In an optional embodiment, a time domain transform is performed on the product of a frequency gain corresponding to a frequency point in a specified sub-frequency band and a frequency amplitude corresponding to a frequency point in the specified sub-frequency band to determine an adjusted specified sub-frequency band; and a frequency band splicing operation is performed on at least one adjusted specified sub-frequency band to determine a target enhanced audio.

[0207] Indicatively, after obtaining the frequency gain corresponding to the frequency point in the specified sub-frequency band, the product of the frequency gain corresponding to the frequency point in the specified sub-frequency band and the frequency amplitude corresponding to the frequency point in the specified sub-frequency band is transformed in the time domain.

[0208] The frequency amplitude corresponding to the frequency point is the frequency amplitude corresponding to the specified sub-band in the target audio, that is, the frequency amplitude is the value before energy adjustment.

[0209] Optionally, the time domain transform is used to indicate that the specified sub-band is converted from the frequency domain dimension to the time domain dimension. In an illustrative embodiment, an inverse Fourier transform is used, and after obtaining the product of the frequency point gain corresponding to the mid-frequency point of the specified sub-band and the frequency point amplitude corresponding to the mid-frequency point of the specified sub-band, the product result is subjected to an inverse Fourier transform, thereby obtaining the target enhanced audio represented in the time domain dimension.

[0210] To summarize, at least two sub-bands corresponding to the target audio are obtained through frequency band segmentation, and sub-band energy data and sub-band energy distribution data corresponding to different sub-bands are obtained. The designated sub-band that does not meet the adjustment conditions is determined through the sub-band energy distribution data of the sub-band, and then the sub-band energy distribution data corresponding to the designated sub-band is used to determine the adjustment parameters corresponding to different designated sub-bands, so that the sub-band energy data of the designated sub-band is selectively adjusted using the adjustment parameters, and based on the adjusted sub-band energy data, the enhanced target enhanced audio is obtained, which improves the quality of speech enhancement while fully considering the characteristics of the target audio.

[0211] In an embodiment of the present application, subband energy high-order distribution data and subband energy low-order distribution data included in the subband energy distribution data are introduced. The subband energy high-order distribution data and the subband energy low-order distribution data are used to indicate the data comparison of subband energy data of two adjacent audio frames. The subband energy data corresponding to different sub-bands in different audio frames are used to differentially determine the subband energy high-order distribution data and the subband energy low-order distribution data, and the adjustment parameters are determined with the help of the subband energy high-order distribution data and the subband energy low-order distribution data. Then, the process of adjusting the subband energy data of different specified sub-bands is realized by adjusting the parameters, so that the subband energy data corresponding to at least two sub-bands can be limited within a preset auditory threshold, and speech enhancement is performed through the frequency band energy dynamic range control method, thereby better improving the auditory fatigue problem.

[0212] In an optional embodiment, the technical principle of the above-mentioned speech enhancement method is described as follows: Figure 8 As shown, the speech enhancement method includes the following three processing parts, namely: (i) a pre-processing part 810; (ii) a processing part 820; and (iii) a post-processing part 830.

[0213] (I) Preprocessing part 810

[0214] Step 811, microphone recording or sound signal decoding.

[0215] In an illustrative manner, based on the auditory fatigue phenomenon, a method based on auditory fatigue frequency domain feature extraction and sub-band dynamic range control is proposed. Optionally, a microphone recording device is used to obtain target audio; or audio data obtained by network transmission is used as the obtained target audio, etc.

[0216] Step 812, input a sound signal.

[0217] Illustratively, after the target audio is acquired, the compressed data of the target audio is decoded to obtain a sound signal corresponding to the target audio.

[0218] Step 813, Fourier transform.

[0219] Illustratively, after obtaining the sound signal, the sound signal is subjected to frequency domain conversion (fast Fourier transform) to realize the process of converting the sound signal from the time domain dimension representation to the frequency domain dimension representation, thereby obtaining the sound signal represented in the frequency domain.

[0220] Step 814, Bark domain transformation.

[0221] Illustratively, the sound signal represented in the frequency domain is divided into sub-bands according to the Bark domain division standard.

[0222] (II) Processing part 820

[0223] Step 821, determining the power spectra and related feature detection values ​​corresponding to different sub-frequency bands.

[0224] Illustratively, after the sound signal is divided into sub-bands, multiple sub-bands corresponding to the target audio are obtained, such as: a first sub-band, a second sub-band, an Nth sub-band, etc.

[0225] Optionally, the power spectra, related feature detection values, etc. corresponding to different sub-bands are determined respectively. Schematically, the sub-band energy data of the current frame of each sub-band is obtained (the logarithm is taken, such as log10(x), where x is the sub-band energy value), and then the sub-band energy long-term distribution data, the sub-band energy high-order distribution data, and the sub-band energy low-order distribution data are calculated. That is, the sub-band energy long-term distribution data, the sub-band energy high-order distribution data, and the sub-band energy low-order distribution data corresponding to the first sub-band are determined; the sub-band energy long-term distribution data, the sub-band energy high-order distribution data, and the sub-band energy low-order distribution data corresponding to the second sub-band are determined; the sub-band energy long-term distribution data, the sub-band energy high-order distribution data, and the sub-band energy low-order distribution data corresponding to the Nth sub-band are determined.

[0226] Among them, the sub-band energy long-term distribution data is determined according to the power spectrum corresponding to the sub-band; the sub-band energy high-order distribution data and the sub-band energy low-order distribution data are determined by the sub-band energy long-term distribution data, and the sub-band energy long-term distribution data, the sub-band energy high-order distribution data and the sub-band energy low-order distribution data are used to indicate the above-mentioned related feature detection values.

[0227] Step 822: Auditory fatigue judgment.

[0228] In schematic form, after obtaining the long-term distribution data of the sub-band energy, the long-term distribution data of the sub-band energy is compared with a preset auditory fatigue threshold value (different sub-bands are set with different values ​​according to the principle of auditory equal loudness curves). If the threshold value is exceeded, it is judged that the sub-band will cause auditory fatigue and needs to enter the dynamic range control (DRC) processing flow to obtain the sub-band gain value; on the contrary, if the auditory fatigue threshold value is not reached, no DRC processing is required (ie, the sub-band gain value is 1).

[0229] Schematically, the preset auditory fatigue threshold value is expressed as Thrd, and the long-time distribution data of the sub-band energy is expressed as Eb_lt(i, k), that is, the sub-band decision condition for auditory fatigue is: Eb_lt(i, k)>Thrd. When the long-time distribution data of the sub-band energy of the k-th sub-band at the i-th audio frame meets the above sub-band decision condition, step 823 is entered; when the long-time distribution data of the sub-band energy of the k-th sub-band at the i-th audio frame does not meet the above sub-band decision condition, step 831 is entered.

[0230] Step 823, DRC parameter adjustment.

[0231] Indicative, such as Figure 7 As shown, the horizontal axis is the input value of the DRC process, that is, the unprocessed sub-band energy data Eb(i, k), and the vertical axis is the output value of the DRC process, that is: the sub-band energy data Eb′(i, k) after DRC processing. Among them, the four points ABCD are on the same oblique line, and the slope of the oblique line is as follows.

[0232]

[0233] Point C represents the maximum output value (this value is expressed as Thrd), and its corresponding input is (4*Eb_up(i, k)-Eb_dw(i, k)) / 3. When the input value is greater than point C, the maximum value Thrd is output; point D is the minimum output value, which is optionally set to 0 here, and its corresponding input is (4*Eb_dw(i, k)-Eb_up(i, k)) / 3. When the input value is less than point D, the minimum value 0 is output. After DRC processing, the sub-band output energy can be controlled within the range of auditory fatigue.

[0234] Step 824, DRC control gain.

[0235] Optionally, after the above-mentioned DRC processing, the output processed subband energy data Eb′(i, k) is obtained, and then based on the processed subband energy data Eb′(i, k) and the unprocessed subband energy data Eb(i, k), the subband gain after DRC processing is obtained, and the subband gain is shown as follows.

[0236]

[0237] (III) Post-processing part 830

[0238] Step 831, Bark domain gain inverse transform.

[0239] Indicatively, after obtaining the subband gain gain(i, k) corresponding to the kth subband in the i-th audio frame, the subband gain is converted into a linear domain gain through an inverse transformation in the Bark domain, that is, the frequency point gains corresponding to multiple frequency points in the kth subband are obtained.

[0240] Optionally, taking the kth sub-band as an example, the frequency gains corresponding to different frequency points in the kth sub-band are multiplied by the power spectrum of the original sound signal, that is, the frequency gains corresponding to different frequency points in the kth sub-band are multiplied by the frequency amplitudes of the frequency points corresponding to the original sound signal.

[0241] Step 832, inverse Fourier transform.

[0242] Optionally, the numerical values ​​corresponding to the different frequency points after multiplication are subjected to inverse Fourier transformation to realize the process of converting the sound signal from the frequency domain dimension representation to the time domain dimension representation.

[0243] Step 833, output the sound signal.

[0244] Schematically, after inverse Fourier transform, a processed sound signal represented in the time domain is obtained.

[0245] Step 834, speaker playback or server transcoding.

[0246] Optionally, the sound signal obtained after the inverse Fourier transform is a signal represented in the time domain, so the output sound signal can be played directly through a speaker, or transmitted to a terminal through network coding compression for decoding and playback.

[0247] To summarize, at least two sub-bands corresponding to the target audio are obtained through frequency band segmentation, and sub-band energy data and sub-band energy distribution data corresponding to different sub-bands are obtained. The designated sub-band that does not meet the adjustment conditions is determined through the sub-band energy distribution data of the sub-band, and then the sub-band energy distribution data corresponding to the designated sub-band is used to determine the adjustment parameters corresponding to different designated sub-bands, so that the sub-band energy data of the designated sub-band is selectively adjusted using the adjustment parameters, and based on the adjusted sub-band energy data, the enhanced target enhanced audio is obtained, which improves the quality of speech enhancement while fully considering the characteristics of the target audio.

[0248] In an optional embodiment, the above-mentioned voice enhancement method is applied to a voice call scenario, and the method is applied to a terminal as an example for explanation. Fig. 9 As shown, the above-mentioned speech enhancement method can also be implemented as the following steps 910 to 950.

[0249] Step 910, obtaining call audio.

[0250] Optionally, in a real-time call scenario, the terminal obtains the call audio of the object in real time, and the call audio is used to indicate voice data with audio information. Schematically, the real-time call scenario includes a two-person call, a multi-person call, and the like.

[0251] Step 920: divide the call audio into frequency bands along the frequency domain dimension to obtain at least two sub-bands.

[0252] The frequency domain dimension is used to describe the dimensionality of the frequency characteristics of the target audio. By analyzing the call audio in the frequency domain dimension, the oscillation information of the call audio in the frequency domain dimension can be provided.

[0253] Step 920 has been described in the above steps 220 and 420 and will not be repeated here.

[0254] Step 930: Obtain sub-band energy data corresponding to at least two sub-frequency bands.

[0255] Among them, the sub-band energy data is used to indicate the frequency change of the audio frame in the call audio along the frequency domain dimension within the sub-band.

[0256] Schematically, the terminal obtains sub-band energy data corresponding to at least two sub-bands in real time according to the frequency change of the audio frame in the call audio along the frequency domain dimension within the sub-band, and thus performs energy data analysis using the sub-band energy data corresponding to the at least two sub-bands.

[0257] Step 930 has been described in the above steps 230 and 430 and will not be repeated here.

[0258] Step 940: Analyze the sub-band energy data corresponding to the at least two sub-frequency bands respectively along the time domain dimension to obtain sub-band energy distribution data corresponding to the at least two sub-frequency bands respectively.

[0259] Schematically, the time domain dimension is a dimension in which a time scale is used to record changes in the target audio over time.

[0260] Optionally, subband energy data corresponding to at least two sub-frequency bands are analyzed along the time domain dimension to obtain subband energy long-term distribution data, subband energy high-order distribution data and subband energy low-order distribution data corresponding to at least two sub-frequency bands.

[0261] Schematically, the terminal obtains subband energy distribution data corresponding to at least two subbands under different audio frames based on the subband energy data corresponding to at least two subbands obtained in real time, for example: obtaining subband energy long-time distribution data, subband energy high-order distribution data and subband energy low-order distribution data corresponding to at least two subbands.

[0262] Step 940 has been described in the above embodiment and will not be repeated here.

[0263] Step 950: When, among at least two sub-bands, sub-band energy distribution data corresponding to a specified sub-band reaches a preset hearing threshold, an adjustment parameter is determined based on the sub-band energy distribution data corresponding to the specified sub-band, and the sub-band energy data of the specified sub-band is adjusted to obtain speech enhanced audio.

[0264] Optionally, the adjustment condition is a pre-set condition, for example, the adjustment condition is a preset energy threshold; or, the adjustment condition is a condition determined in real time based on multiple sub-band energy distribution data, for example, the adjustment condition is an average value of multiple sub-band energy distribution data, etc.

[0265] Schematically, after obtaining the subband energy distribution data corresponding to at least two subbands in real time, the terminal compares the long-term distribution data of subband energy corresponding to at least two subbands with the preset hearing threshold. When the long-term distribution data of subband energy corresponding to a specified subband reaches the preset hearing threshold, the adjustment parameter corresponding to the specified subband is determined according to the high-order distribution data of subband energy, the low-order distribution data of subband energy and the preset hearing threshold corresponding to the specified subband, so as to adjust the subband energy data of the specified subband by adjusting the parameter, so that the subband energy data of the specified subband whose long-term distribution data of subband energy exceeds the preset hearing threshold is limited within the preset hearing threshold, and further, the subband energy data corresponding to at least two subbands are all limited within the preset hearing threshold, so as to obtain call enhanced audio.

[0266] To summarize, at least two sub-bands corresponding to the call audio are obtained through frequency band segmentation, and sub-band energy data and sub-band energy distribution data corresponding to different sub-bands are obtained. The designated sub-band that does not meet the adjustment conditions is determined through the sub-band energy distribution data of the sub-band, and then the adjustment parameters corresponding to different designated sub-bands are determined using the sub-band energy distribution data corresponding to the designated sub-band, so that the sub-band energy data of the designated sub-band is selectively adjusted using the adjustment parameters, so that the sub-band energy data corresponding to at least two sub-bands are limited within the preset hearing threshold. Under the limit of the preset hearing threshold, the occurrence of auditory fatigue is avoided, which not only improves the call quality of the subject during the call, but also reduces the discomfort of the subject during long-term calls, effectively improving the call effect.

[0267] Fig.10 is a speech enhancement device provided by an exemplary embodiment of the present application, such as Fig.10 As shown, the device includes the following parts:

[0268] The audio acquisition module 1010 is used to acquire target audio, where the target audio is audio data to be speech enhanced;

[0269] A frequency band segmentation module 1020, configured to segment the target audio frequency band along a frequency domain dimension to obtain at least two sub-frequency bands;

[0270] A data acquisition module 1030 is used to acquire sub-band energy data corresponding to the at least two sub-bands respectively, wherein the sub-band energy data is used to indicate a frequency change of an audio frame in the target audio along a frequency domain dimension within the sub-band;

[0271] A data analysis module 1040 is used to analyze the sub-band energy data corresponding to the at least two sub-frequency bands respectively along the time domain dimension to obtain sub-band energy distribution data corresponding to the at least two sub-frequency bands respectively, wherein the sub-band energy distribution data is used to indicate the frequency distribution of the target audio in the at least two sub-frequency bands;

[0272] The energy adjustment module 1050 is used to determine the adjustment parameters based on the sub-band energy distribution data corresponding to the specified sub-band in the at least two sub-bands, and adjust the sub-band energy data of the specified sub-band to obtain the target enhanced audio.

[0273] In an optional embodiment, the data analysis module 1040 is also used to obtain subband energy data corresponding to at least two subbands in the i-th audio frame and subband energy distribution data corresponding to at least two subbands in the i-1th audio frame, where i is a positive integer greater than 1; based on the subband energy data corresponding to at least two subbands in the i-th audio frame, the subband energy distribution data corresponding to at least two subbands in the i-1th audio frame and preset weights, the subband energy distribution data corresponding to at least two subbands in the i-th audio frame are obtained.

[0274] In an optional embodiment, the sub-band energy distribution data includes sub-band energy long-term distribution data, and the sub-band energy long-term distribution data is used to indicate a change in sub-band energy data of two adjacent audio frames;

[0275] The data analysis module 1040 is also used to perform weighted fusion on the subband energy data corresponding to at least two subbands in the i-th audio frame and the subband energy long-time distribution data corresponding to at least two subbands in the i-1-th audio frame with a first preset weight to determine the subband energy long-time distribution data corresponding to at least two subbands in the i-th audio frame.

[0276] In an optional embodiment, the energy adjustment module 1050 is also used to determine the adjustment parameter in response to the long-term distribution data of sub-band energy corresponding to a specified sub-band in the at least two sub-bands reaching a preset auditory threshold, and the preset auditory threshold is used to indicate the adjustment condition.

[0277] In an optional embodiment, the sub-band energy distribution data includes sub-band energy high-bit distribution data, and the sub-band energy high-bit distribution data is used to indicate a data comparison between the sub-band energy data of the i-th audio frame and the sub-band energy high-bit distribution data of the i-1-th audio frame;

[0278] The data analysis module 1040 is also used to obtain the sub-band energy high-order distribution data corresponding to the i-th audio frame based on the sub-band energy data corresponding to the i-th audio frame, the sub-band energy high-order distribution data corresponding to the i-1-th audio frame and the second preset weight.

[0279] In an optional embodiment, the sub-band energy distribution data includes sub-band energy low-bit distribution data, and the sub-band energy low-bit distribution data is used to indicate a data comparison between the sub-band energy data of the i-th audio frame and the sub-band energy low-bit distribution data of the i-1-th audio frame;

[0280] The data analysis module 1040 is also used to obtain the sub-band energy low-bit distribution data corresponding to the i-th audio frame based on the sub-band energy data corresponding to the i-th audio frame, the sub-band energy low-bit distribution data corresponding to the i-1-th audio frame and the third preset weight.

[0281] In an optional embodiment, the energy adjustment module 1050 is also used to determine the adjustment parameter corresponding to the specified sub-band based on the sub-band energy high-order distribution data corresponding to the specified sub-band, the sub-band energy low-order distribution data corresponding to the specified sub-band, and a preset auditory threshold, and the preset auditory threshold is used to assist in limiting the data range of the sub-band energy data corresponding to the specified sub-band.

[0282] In an optional embodiment, the energy adjustment module 1050 is also used to determine the adjustment parameter based on the sub-band energy distribution data corresponding to the specified sub-band; adjust the sub-band energy data corresponding to the specified sub-band by using the adjustment parameter to determine the energy adjustment gain corresponding to the specified sub-band, and the energy adjustment gain is used to indicate the data ratio adjustment of the sub-band energy data; perform speech enhancement on the target audio based on the energy adjustment gain to obtain the target enhanced audio.

[0283] In an optional embodiment, the energy adjustment module 1050 is also used to perform gain transformation on the energy adjustment gain corresponding to the specified sub-frequency band to determine the frequency gain corresponding to the frequency point in the specified sub-frequency band; and obtain the target enhanced audio based on the product of the frequency gain corresponding to the frequency point in the specified sub-frequency band and the frequency amplitude corresponding to the frequency point in the specified sub-frequency band.

[0284] In an optional embodiment, the energy adjustment module 1050 is also used to perform a time domain transform on the product of the frequency gain corresponding to the frequency point in the designated sub-frequency band and the frequency amplitude corresponding to the frequency point in the designated sub-frequency band to determine the adjusted designated sub-frequency band; and perform a frequency band splicing operation on at least one of the adjusted designated sub-frequency bands to determine the target enhanced audio.

[0285] In an optional embodiment, the frequency band segmentation module 1020 is further configured to perform frequency band segmentation on the target audio along the frequency domain dimension based on a preset frequency band division standard to obtain the at least two sub-frequency bands.

[0286] In an optional embodiment, the energy adjustment module 1050 is also used to adjust the sub-band energy data of the designated sub-band to obtain the adjusted sub-band energy data corresponding to the designated sub-band, and the at least two sub-bands also include candidate discarded sub-bands that have not been energy adjusted; for the audio frames in the target audio, in response to the fact that the frequency band share of the energy-adjusted designated sub-band in at least two sub-bands exceeds a preset share threshold, and the candidate discarded sub-band is outside the frequency domain range corresponding to the human voice, the sub-band energy data of the energy-adjusted designated sub-band is retained, and the sub-band energy data of the candidate discarded sub-band that has not been energy adjusted is discarded to obtain the target enhanced audio.

[0287] To summarize, through the above-mentioned device, after frequency band segmentation is performed to obtain at least two sub-bands corresponding to the target audio, sub-band energy data and sub-band energy distribution data corresponding to different sub-bands are obtained, and the designated sub-band that does not meet the adjustment conditions is determined through the sub-band energy distribution data of the sub-band, and then the sub-band energy distribution data corresponding to the designated sub-band is used to determine the adjustment parameters corresponding to different designated sub-bands, so that the sub-band energy data of the designated sub-band is selectively adjusted using the adjustment parameters, and based on the adjusted sub-band energy data, the enhanced target enhanced audio is obtained, while fully considering the characteristics of the target audio, the quality of speech enhancement is improved.

[0288] It should be noted that the speech enhancement device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the speech enhancement device provided in the above embodiment and the speech enhancement method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0289] Fig.11 The schematic diagram of the structure of a server provided by an exemplary embodiment of the present application is shown. The server 1100 includes a central processing unit (CPU) 1101, a system memory 1104 including a random access memory (RAM) 1102 and a read-only memory (ROM) 1103, and a system bus 1105 connecting the system memory 1104 and the central processing unit 1101. The server 1100 also includes a mass storage device 1106 for storing an operating system 1113, an application program 1114, and other program modules 1115.

[0290] The mass storage device 1106 is connected to the central processing unit 1101 through a mass storage controller (not shown) connected to the system bus 1105. The mass storage device 1106 and its associated computer-readable media provide non-volatile storage for the server 1100. That is, the mass storage device 1106 may include a computer-readable medium (not shown) such as a hard disk or a compact disc read only memory (CD-ROM) drive.

[0291] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other solid-state storage technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above. The above-mentioned system memory 1104 and mass storage device 1106 can be collectively referred to as memory.

[0292] According to various embodiments of the present application, the server 1100 can also be connected to a remote computer on the network through a network such as the Internet. That is, the server 1100 can be connected to the network 1112 through the network interface unit 1111 connected to the system bus 1105, or the network interface unit 1111 can be used to connect to other types of networks or remote computer systems (not shown).

[0293] The above-mentioned memory also includes one or more programs, and the one or more programs are stored in the memory and configured to be executed by the CPU.

[0294] An embodiment of the present application also provides a computer device, which includes a processor and a memory, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the speech enhancement method provided by the above-mentioned method embodiments.

[0295] An embodiment of the present application also provides a computer-readable storage medium, on which is stored at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the speech enhancement method provided by the above-mentioned method embodiments.

[0296] The embodiments of the present application also provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device performs the speech enhancement method described in any of the above embodiments.

[0297] Optionally, the computer readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0298] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0299] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A speech enhancement method, characterized in that: The method comprises: Acquire target audio, where the target audio is audio data to be speech enhanced; Dividing the target audio frequency band along the frequency domain dimension to obtain at least two sub-bands; Acquire sub-band energy data corresponding to the at least two sub-bands respectively, where the sub-band energy data is used to indicate frequency changes of audio frames in the target audio along the frequency domain dimension within the sub-bands; Analyzing the sub-band energy data corresponding to the at least two sub-frequency bands respectively along the time domain dimension to obtain sub-band energy distribution data corresponding to the at least two sub-frequency bands respectively, wherein the sub-band energy distribution data is used to indicate the frequency distribution of the target audio in the at least two sub-frequency bands; If, among the at least two sub-frequency bands, sub-band energy distribution data corresponding to a designated sub-frequency band meets the adjustment condition, an adjustment parameter is determined based on the sub-band energy distribution data corresponding to the designated sub-frequency band, and the sub-band energy data of the designated sub-frequency band is adjusted to obtain the target enhanced audio.

2. The method according to claim 1, characterized in that The analyzing the sub-band energy data corresponding to the at least two sub-frequency bands respectively along the time domain dimension to obtain the sub-band energy distribution data corresponding to the at least two sub-frequency bands respectively includes: Obtain subband energy data corresponding to at least two subbands in an i-th audio frame and subband energy distribution data corresponding to at least two subbands in an i-1-th audio frame, where i is a positive integer greater than 1; Based on the subband energy data corresponding to at least two subbands in the i-th audio frame, the subband energy distribution data corresponding to at least two subbands in the i-1-th audio frame and the preset weights, the subband energy distribution data corresponding to at least two subbands in the i-th audio frame are obtained.

3. The method according to claim 2, characterized in that The sub-band energy distribution data includes sub-band energy long-term distribution data, and the sub-band energy long-term distribution data is used to indicate the change of sub-band energy data of two adjacent audio frames; The obtaining, based on the subband energy data respectively corresponding to at least two subbands in the i-th audio frame, the subband energy distribution data respectively corresponding to at least two subbands in the i-th audio frame, and the preset weights, the subband energy distribution data respectively corresponding to at least two subbands in the i-th audio frame comprises: The subband energy data corresponding to at least two subbands in the i-th audio frame and the subband energy long-time distribution data corresponding to at least two subbands in the i-1-th audio frame are weightedly fused with a first preset weight to determine the subband energy long-time distribution data corresponding to at least two subbands in the i-th audio frame.

4. The method according to claim 3, characterized in that The step of determining the adjustment parameter based on the sub-band energy distribution data corresponding to the designated sub-band in the at least two sub-bands when the sub-band energy distribution data corresponding to the designated sub-band meets the adjustment condition comprises: In response to the fact that the sub-band energy long-term distribution data corresponding to the designated sub-band in the at least two sub-bands reaches a preset hearing threshold, the adjustment parameter is determined, and the preset hearing threshold is used to indicate the adjustment condition.

5. The method according to claim 2, characterized in that: The sub-band energy distribution data includes sub-band energy high-order distribution data, and the sub-band energy high-order distribution data is used to indicate a data comparison between the sub-band energy data of the i-th audio frame and the sub-band energy high-order distribution data of the i-1-th audio frame; The obtaining, based on the subband energy data respectively corresponding to at least two subbands in the i-th audio frame, the subband energy distribution data respectively corresponding to at least two subbands in the i-th audio frame, and the preset weights, the subband energy distribution data respectively corresponding to at least two subbands in the i-th audio frame comprises: Based on the sub-band energy data corresponding to at least two sub-bands in the i-th audio frame, the sub-band energy high-order distribution data corresponding to at least two sub-bands in the i-1-th audio frame, and the second preset weight, the sub-band energy high-order distribution data corresponding to at least two sub-bands in the i-th audio frame are obtained.

6. The method according to claim 2, characterized in that The sub-band energy distribution data includes sub-band energy low-bit distribution data, and the sub-band energy low-bit distribution data is used to indicate a data comparison between the sub-band energy data of the i-th audio frame and the sub-band energy low-bit distribution data of the i-1-th audio frame; The obtaining, based on the subband energy data respectively corresponding to at least two subbands in the i-th audio frame, the subband energy distribution data respectively corresponding to at least two subbands in the i-th audio frame, and the preset weights, the subband energy distribution data respectively corresponding to at least two subbands in the i-th audio frame comprises: Based on the sub-band energy data corresponding to at least two sub-bands in the i-th audio frame, the sub-band energy low-bit distribution data corresponding to at least two sub-bands in the i-1-th audio frame, and the third preset weight, the sub-band energy low-bit distribution data corresponding to at least two sub-bands in the i-th audio frame are obtained.

7. The method according to any one of claims 1 to 6, characterized in that: The sub-band energy distribution data includes sub-band energy high-order distribution data and sub-band energy low-order distribution data; The determining the adjustment parameter based on the sub-band energy distribution data corresponding to the designated sub-frequency band includes: Based on the high-order distribution data of sub-band energy corresponding to the specified sub-band, the low-order distribution data of sub-band energy corresponding to the specified sub-band and a preset auditory threshold, an adjustment parameter corresponding to the specified sub-band is determined, and the preset auditory threshold is used to assist in limiting the data range of the sub-band energy data corresponding to the specified sub-band.

8. The method according to any one of claims 1 to 6, characterized in that: The step of determining the adjustment parameter based on the sub-band energy distribution data corresponding to the designated sub-band, and adjusting the sub-band energy data of the designated sub-band to obtain the target enhanced audio includes: Determining the adjustment parameter based on sub-band energy distribution data corresponding to the designated sub-frequency band; The sub-band energy data corresponding to the designated sub-band is adjusted by using the adjustment parameter to determine the energy adjustment gain corresponding to the designated sub-band, where the energy adjustment gain is used to indicate the data ratio adjustment of the sub-band energy data; Performing gain transformation on the energy adjustment gain corresponding to the designated sub-frequency band to determine the frequency point gain corresponding to the frequency point in the designated sub-frequency band; The target enhanced audio is obtained based on the product of the frequency gain corresponding to the frequency point in the designated sub-frequency band and the frequency amplitude corresponding to the frequency point in the designated sub-frequency band.

9. The method according to any one of claims 1 to 6, characterized in that: The step of adjusting the sub-band energy data of the designated sub-band to obtain target enhanced audio includes: Adjusting the subband energy data of the designated subband to obtain adjusted subband energy data corresponding to the designated subband, wherein the at least two subbands also include candidate discarded subbands for which energy adjustment has not been performed; For the audio frames in the target audio, in response to the fact that the frequency band proportion of the energy-adjusted designated sub-band in at least two sub-bands exceeds a preset proportion threshold, and the candidate discarded sub-band is outside the frequency domain range corresponding to the human voice, the sub-band energy data of the energy-adjusted designated sub-band is retained, and the sub-band energy data of the candidate discarded sub-band that has not been energy adjusted is discarded to obtain the target enhanced audio.

10. A speech enhancement device, characterized in that: The device comprises: An audio acquisition module, used to acquire target audio, where the target audio is audio data to be speech enhanced; A frequency band segmentation module, used for performing frequency band segmentation on the target audio along a frequency domain dimension to obtain at least two sub-frequency bands; A data acquisition module, used to acquire sub-band energy data corresponding to the at least two sub-bands respectively, wherein the sub-band energy data is used to indicate a frequency change of an audio frame in the target audio along a frequency domain dimension within the sub-band; a data analysis module, configured to analyze the sub-band energy data corresponding to the at least two sub-frequency bands respectively along the time domain dimension to obtain sub-band energy distribution data corresponding to the at least two sub-frequency bands respectively, wherein the sub-band energy distribution data is used to indicate the frequency distribution of the target audio in the at least two sub-frequency bands; An energy adjustment module is used to determine an adjustment parameter based on the sub-band energy distribution data corresponding to the specified sub-band in the at least two sub-bands, if the sub-band energy distribution data corresponding to the specified sub-band meets the adjustment conditions, and adjust the sub-band energy data of the specified sub-band to obtain the target enhanced audio.

11. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the speech enhancement method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the speech enhancement method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that The method comprises a computer program or an instruction, which, when executed by a processor, implements the speech enhancement method as claimed in any one of claims 1 to 9.

Images