Spectrum unmasking method, device and equipment for reference driving and medium
Patent Information
- Application Number
- CN202611154694.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-31
- Publication Date
- 2026-10-16
AI Technical Summary
其四，相位失真与抖动伪影问题突出
[0014] This application provides a reference-driven spectrum demasking method, apparatus, device, and medium. First, energy measurements are performed on reference vocals and reference accompaniment in each frequency band to obtain the energy of the reference vocals and accompaniment. Then, based on the energy of the reference vocals and accompaniment in each frequency band, the target sharpness corresponding to each frequency band is determined. Simultaneously, energy measurements are performed on the vocals and accompaniment to be processed in each frequency band to obtain the energy of the vocals and accompaniment to be processed. Then, based on the energy of the vocals and accompaniment to be processed in each frequency band, the actual sharpness corresponding to each frequency band is determined. Based on this, the target sharpness and actual sharpness of each frequency band are compared, and targeted energy reduction is performed on the accompaniment to be processed based on the difference between the two, ultimately outputting the demasked accompaniment. By using this method, the clarity target of each frequency band can be matched with the mixing style of the specific song, avoiding the mismatch problem caused by manually setting a fixed threshold. Furthermore, the backing track is reduced according to the clarity difference, so that the reduction operation is accurately applied to the specific location where the masking occurs. This effectively avoids the hollowing out of the backing track and the imbalance of the arrangement caused by blind reduction, and preserves the original form of the backing track and the integrity of the arrangement to the greatest extent. It improves the naturalness of the sound, reduces the amount of manual adjustment in the mixing production, and improves the automation and intelligence of the spectrum masking process.
Smart Images
Figure CN122929995A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of audio processing technology, and more specifically, to a reference-driven spectrum demasking method, apparatus, device, and medium. Background Technology
[0002] In music mixing, vocals often fail to be clearly presented due to the energy masking of the accompaniment in certain frequency bands. Even if the overall loudness is sufficient, the sound will still be blurry and lack penetration. To improve this problem, the industry widely adopts spectrum ducking technology, which uses vocals as a sidechain signal to apply multi-band dynamic equalization to the accompaniment. Gain attenuation is applied to the accompaniment in the frequency bands where vocals are active, thereby freeing up spectral space for vocals.
[0003] However, existing spectrum avoidance techniques have several limitations. First, the attenuation target is blind and lacks objective reference. Existing solutions usually determine the attenuation amount based on the amount of vocal energy or a fixed threshold, leading to excessive attenuation of accompaniment and hollowing out of instrumental arrangements in frequency bands that do not need processing, while insufficient attenuation may occur in frequency bands that do need enhancement. Second, it fails to adapt to the stylistic differences of different songs. Different music genres have significantly different requirements for vocal clarity. Rap works require highly prominent vocals, while ballads pursue a sense of integration between vocals and accompaniment. Fixed strategies cannot flexibly adapt to diverse needs. Third, it is prone to false triggering and unnecessary processing. If the rules are still followed in pure accompaniment sections, the accompaniment timbre will be changed unnecessarily; if attenuation is still performed in the low-frequency range where vocals are basically silent, it will simply destroy the low-frequency foundation of the accompaniment. Fourth, phase distortion and jitter artifacts are prominent problems. Multi-band crossovers are prone to introducing phase distortion, and frame-by-frame hard switching of gain produces an audible pumping sensation and jitter, which damages the naturalness of the listening experience. Summary of the Invention
[0004] This application provides a reference-driven spectrum demasking method, apparatus, device, and medium, which can improve the automation and intelligence level of spectrum masking processing.
[0005] In a first aspect, embodiments of this application provide a reference-driven spectrum demasking method, the method comprising: Based on the reference vocal and reference accompaniment, the energy of the reference vocal and reference accompaniment in each frequency band is measured respectively. The target intelligibility of each frequency band is determined based on the reference vocal energy and the reference accompaniment energy. Based on the vocal and accompaniment to be processed, the energy of the vocal and accompaniment to be processed in each frequency band is measured respectively. The actual clarity of each frequency band is determined based on the vocal energy to be processed and the accompaniment energy to be processed. Based on the target sharpness and the actual sharpness of each frequency band, the accompaniment to be processed is energy-reduced to obtain the demasked accompaniment.
[0006] Optionally, determining the target intelligibility of each frequency band based on the reference vocal energy and the reference accompaniment energy includes: In each of the frequency bands, the active frames corresponding to the reference human voice are selected; Extract the energy of the reference vocals and the reference accompaniment in the active frame, and calculate the energy ratio of the active frame; The target sharpness is determined based on the energy ratio of each active frame.
[0007] Optionally, determining the actual intelligibility of each frequency band based on the vocal energy to be processed and the accompaniment energy to be processed includes: In each frequency band, for all time frames, the vocal energy and accompaniment energy to be processed corresponding to each time frame are extracted, and the energy ratio of each time frame is calculated. The actual sharpness is determined based on the energy ratio of each time frame.
[0008] Optionally, the step of performing energy reduction on the accompaniment to be processed based on the target sharpness and the actual sharpness of each frequency band to obtain the demasked accompaniment includes: Based on the target sharpness and the actual sharpness of each frequency band, determine the sharpness gap of each frequency band; Calculate the bandwidth reduction based on the resolution gap of each bandwidth; Based on the bandwidth reduction amount, the energy of the accompaniment to be processed is reduced to obtain the demasked accompaniment.
[0009] Optionally, calculating the bandwidth reduction based on the resolution gap of each of the frequency bands includes: Based on the energy distribution of the reference human voice in each of the frequency bands, an occupancy mask is generated, which is used to mark the frequency band occupied by the reference human voice; The vocal activity level is obtained based on the energy distribution of the vocal voice to be processed in each frame; Based on the occupancy mask and the vocal activity level, the sharpness gap is filtered to obtain the filtered sharpness gap; The weight of each frequency band is determined based on the degree of influence of each frequency band on the intelligibility of human voice. The bandwidth reduction is calculated based on the resolution gap after filtering and the weight of each bandwidth.
[0010] Optionally, before performing energy reduction on the accompaniment to be processed based on the bandwidth reduction amount, the method further includes: The bandwidth reduction is smoothed and limited to obtain the final reduction. The step of reducing the accompaniment to be processed according to the bandwidth reduction amount to obtain the demasked accompaniment includes: The energy of the accompaniment to be processed is reduced according to the final reduction amount to obtain the unmasked accompaniment.
[0011] Optionally, the step of performing energy reduction on the accompaniment to be processed according to the frequency band reduction amount to obtain the demasked accompaniment includes: The bandwidth reduction is mapped to each frequency point of the time-frequency transformation to obtain the reduction gain at each frequency point; The accompaniment to be processed is subjected to time-frequency transformation to obtain the spectrum of the accompaniment to be processed; Based on the reduction gain, the amplitude of each frequency point in the accompaniment spectrum to be processed is adjusted to obtain the adjusted accompaniment spectrum; The adjusted accompaniment spectrum is inversely transformed to obtain the unmasked accompaniment.
[0012] Secondly, embodiments of this application also provide a reference-driven spectrum demasking device, the device comprising: The measurement module is used to measure the energy of the reference human voice and the energy of the reference accompaniment in each frequency band, based on the reference human voice and the reference accompaniment. A calculation module is used to determine the target intelligibility of each frequency band based on the reference vocal energy and the reference accompaniment energy. The measurement module is also used to measure the energy of the human voice to be processed and the energy of the accompaniment to be processed in each frequency band according to the human voice to be processed and the accompaniment to be processed. The calculation module is also used to determine the actual clarity of each frequency band based on the vocal energy to be processed and the accompaniment energy to be processed; The reduction module is used to reduce the energy of the accompaniment to be processed according to the target sharpness and the actual sharpness of each frequency band, so as to obtain the demasked accompaniment.
[0013] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a memory, and a bus, wherein the memory stores program instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and the processor executes the program instructions to perform the steps of the reference-driven spectrum demasking method as described in any of the first aspects. Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the reference-driven spectrum demasking method as described in any of the first aspects.
[0014] This application provides a reference-driven spectrum demasking method, apparatus, device, and medium. First, energy measurements are performed on reference vocals and reference accompaniment in each frequency band to obtain the energy of the reference vocals and accompaniment. Then, based on the energy of the reference vocals and accompaniment in each frequency band, the target sharpness corresponding to each frequency band is determined. Simultaneously, energy measurements are performed on the vocals and accompaniment to be processed in each frequency band to obtain the energy of the vocals and accompaniment to be processed. Then, based on the energy of the vocals and accompaniment to be processed in each frequency band, the actual sharpness corresponding to each frequency band is determined. Based on this, the target sharpness and actual sharpness of each frequency band are compared, and targeted energy reduction is performed on the accompaniment to be processed based on the difference between the two, ultimately outputting the demasked accompaniment. By using this method, the clarity target of each frequency band can be matched with the mixing style of the specific song, avoiding the mismatch problem caused by manually setting a fixed threshold. Furthermore, the backing track is reduced according to the clarity difference, so that the reduction operation is accurately applied to the specific location where the masking occurs. This effectively avoids the hollowing out of the backing track and the imbalance of the arrangement caused by blind reduction, and preserves the original form of the backing track and the integrity of the arrangement to the greatest extent. It improves the naturalness of the sound, reduces the amount of manual adjustment in the mixing production, and improves the automation and intelligence of the spectrum masking process. Attached Figure Description
[0015] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 A schematic flowchart of a reference-driven spectrum demasking method provided in an embodiment of this application; Figure 2 A schematic diagram illustrating the process of obtaining target sharpness in a reference-driven spectrum demasking method provided in an embodiment of this application; Figure 3 A schematic diagram illustrating the process of obtaining actual sharpness in a reference-driven spectrum demasking method provided in an embodiment of this application; Figure 4 A schematic diagram illustrating the process of obtaining the demasked accompaniment in a reference-driven spectrum demasking method provided in an embodiment of this application; Figure 5 A schematic diagram illustrating the process of obtaining the bandwidth reduction in a reference-driven spectrum demasking method provided in an embodiment of this application; Figure 6 A schematic diagram of the process for obtaining the demasked accompaniment in another reference-driven spectrum demasking method provided in this application embodiment; Figure 7 A schematic diagram of a reference-driven spectrum demasking device provided in an embodiment of this application; Figure 8 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation
[0017] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0018] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.
[0019] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0020] Before providing a detailed explanation of this application, let's first introduce its application scenarios.
[0021] In audio processing scenarios that rely on the coordinated handling of vocals and accompaniment, such as music production, post-production audio mixing, and film dubbing, the energy coverage of the accompaniment in specific frequency bands often leads to a decrease in the clarity and intelligibility of vocals. Even if the overall loudness meets listening requirements, the vocals may still sound blurry and lack penetration, affecting the overall quality of the work. For example, in vocal mixing scenarios, when producers replace newly recorded vocals with already released accompaniment, the matching degree between the new vocals and the original accompaniment often cannot reach the original level. It is necessary to improve the clarity of the vocals while preserving the original accompaniment.
[0022] Based on this, this application provides a reference-driven spectrum demasking method, apparatus, device, and medium. First, the energy of the reference vocal and accompaniment is measured in each frequency band to obtain the energy of the reference vocal and accompaniment. Then, based on the energy of the reference vocal and accompaniment in each frequency band, the target sharpness corresponding to each frequency band is determined. Simultaneously, the energy of the vocal and accompaniment to be processed are measured in each frequency band to obtain the energy of the vocal and accompaniment to be processed. Then, based on the energy of the vocal and accompaniment to be processed in each frequency band, the actual sharpness corresponding to each frequency band is determined. On this basis, the target sharpness and actual sharpness of each frequency band are compared, and the energy of the accompaniment to be processed is reduced accordingly based on the difference. Finally, the demasked accompaniment is output, preserving the original form and arrangement integrity of the accompaniment to the greatest extent, improving the naturalness of the listening experience, reducing the workload of manual adjustments in mixing production, and improving the automation and intelligence level of spectrum masking processing.
[0023] The following explanation, in conjunction with the accompanying drawings, uses several embodiments to illustrate the concepts.
[0024] The reference-driven spectrum demasking method provided in this application can be implemented by an electronic device pre-installed with a preset reference-driven spectrum demasking algorithm or software, by running the algorithm or software. The electronic device can be, for example, a server or a terminal, and the terminal can be a user computer.
[0025] Figure 1 A flowchart illustrating a reference-driven spectrum demasking method provided in this application embodiment is shown below. Figure 1 As shown, the reference-driven spectrum demasking method includes: S101 measures the energy of the reference human voice and the energy of the reference accompaniment in each frequency band based on the reference human voice and the reference accompaniment.
[0026] First, obtain the reference vocals and reference accompaniment. The reference vocals are the pure, original, dry vocals from the released version; the reference accompaniment is the accompaniment signal corresponding to the reference vocals, forming a paired reference material with the reference vocals.
[0027] Because the energy distribution of sound signals varies across frequency bands, and the sensitivity of the human ear to sound varies across different frequency bands, a uniform measurement across the entire frequency band would not accurately reflect the energy distribution characteristics of sound across the frequency bands. Specifically, approximately 24 frequency bands are divided according to the Bark scale within the range of approximately 90Hz to approximately 12kHz. The Bark scale is designed based on the frequency response characteristics of the basilar membrane of the human ear, with finer divisions in the low-frequency region and coarser divisions in the high-frequency region, making subsequent measurements and processing closer to the subjective hearing of the human ear. To facilitate independent processing of the energy of each frequency band in the frequency domain, it is also necessary to establish a mapping relationship between each Bark frequency band and the Short-Time Fourier Transform (STFT) frequency points, that is, to determine which STFT frequency point index ranges correspond to each Bark frequency band. In this embodiment, the STFT parameters are n_fft=2048 and hop=512.
[0028] After completing the frequency band division and frequency point mapping, short-time Fourier transforms are performed on the reference vocal and reference accompaniment frequencies respectively to obtain the reference vocal spectrum and reference accompaniment spectrum. Then, for each Bark frequency band, based on the corresponding STFT frequency point index set, the energy of all frequency points in the reference vocal spectrum falling within that frequency band is summed to obtain the reference vocal energy for that frequency band; the same summation operation is performed on the reference accompaniment spectrum to obtain the reference accompaniment energy for that frequency band. This process is repeated for all frequency bands to obtain the reference vocal energy and reference accompaniment energy for each band.
[0029] In one possible implementation, besides the Bark scale, the critical band division can also use the Equivalent Rectangular Bandwidth (ERB) scale or the Mel scale for frequency band division. The ERB scale is designed based on the filter bandwidth characteristics of the basilar membrane of the human ear, and can match the frequency selectivity characteristics of the human ear well at all frequencies; the Mel scale is designed based on the human ear's perception of pitch, with higher resolution in the low-frequency region and lower resolution in the high-frequency region. The above scales approximate the auditory characteristics of the human ear from different angles and can all serve as alternatives to the Bark scale. The appropriate scale can be flexibly selected based on the specific application scenario and actual processing effect.
[0030] S102 determines the target intelligibility of each frequency band based on the reference vocal energy and the reference accompaniment energy.
[0031] Among them, target clarity is used to characterize the prominence of the reference vocals relative to the reference accompaniment in each frequency band.
[0032] Specifically, for each Bark frequency band, the ratio of the total energy of the reference vocals to the total energy of the reference accompaniment is calculated. A larger ratio indicates that the vocals in that band are relatively prominent and have high clarity; a smaller ratio indicates that the vocals are severely masked by the accompaniment and have low clarity.
[0033] Considering the extreme cases that may exist in actual signals, the calculated ratio needs to be limited. On the one hand, when the energy of the reference accompaniment is extremely low, the ratio may be abnormally large, leading to an inflated target sharpness; on the other hand, when the energy of the reference vocals is extremely low, the ratio may be abnormally small, losing its reference value. To ensure numerical stability and clarity of physical meaning, reasonable upper and lower limits are set, and ratios exceeding the range are truncated to the boundary values. The value after limiting is the target sharpness for that frequency band.
[0034] By iterating through all Bark bands, the above calculation and clipping process is performed independently for each band, and finally the target sharpness corresponding to each band is obtained.
[0035] S103, based on the vocal and accompaniment to be processed, measures the energy of the vocal and accompaniment to be processed in each frequency band.
[0036] Among them, the vocal to be processed is the actual vocal signal to be mixed into the accompaniment. This signal may include reverb, delay and other effects processing, i.e., wet sound; the accompaniment to be processed is the actual accompaniment signal to be used.
[0037] Specifically, in this embodiment, the frequency band division method and the mapping relationship between each frequency band and the STFT frequency point are the same as in S101. Under the same frequency band division and mapping relationship, short-time Fourier transforms are performed on the vocal and accompaniment to be processed, respectively, to obtain the vocal spectrum and accompaniment spectrum. Subsequently, for each Bark frequency band, according to the STFT frequency point index set corresponding to that frequency band, the energy of all frequency points in the vocal spectrum falling within that frequency band is summed to obtain the vocal energy to be processed in that frequency band; the same summation operation is performed on the accompaniment spectrum to obtain the accompaniment energy to be processed in that frequency band. By traversing all frequency bands, the vocal energy and accompaniment energy to be processed in each frequency band are obtained respectively.
[0038] In the context of vocal overlay, the accompaniment to be processed is the same as the reference accompaniment. In this case, the energy data of the reference accompaniment already measured in S101 can be reused, eliminating the need to repeatedly perform time-frequency transformation and frequency band energy summation, thus saving computational resources. In general scenarios, if the accompaniment to be processed and the reference accompaniment are different audio materials, the above measurement process must be completed independently.
[0039] S104 determines the actual clarity of each frequency band based on the vocal energy and accompaniment energy to be processed.
[0040] Among them, actual clarity is used to characterize the degree of prominence of the vocals to be processed relative to the accompaniment to be processed in each frequency band.
[0041] Specifically, for each Bark frequency band, the ratio of the total energy of the vocals to be processed to the total energy of the accompaniment to be processed in that band is calculated. When the ratio is large, it indicates that the vocals in that band are relatively prominent and the actual clarity is high; when the ratio is small, it indicates that the vocals are severely masked by the accompaniment and the actual clarity is low.
[0042] Considering the extreme cases that may exist in actual signals, the calculated ratio needs to be limited. On the one hand, when the energy of the accompaniment to be processed is extremely low, the ratio may be abnormally large, leading to an inflated actual clarity; on the other hand, when the energy of the vocals to be processed is extremely low, the ratio may be abnormally small, losing its comparative significance. To ensure numerical stability and clarity of physical meaning, reasonable upper and lower limits are set, and ratios exceeding the range are truncated to the boundary values. The value after limiting is the actual clarity in that frequency band at that moment.
[0043] By iterating through all Bark bands, the above calculation and clipping process is performed independently for each band, and finally the actual resolution corresponding to each band is obtained.
[0044] S105 performs energy reduction on the accompaniment to be processed based on the target clarity and actual clarity of each frequency band, resulting in the demasked accompaniment.
[0045] Specifically, the target clarity of each frequency band is compared with the actual clarity of that band at each moment. When the actual clarity is lower than the target clarity, it means that the vocals in that frequency band are masked by the accompaniment to a greater extent than in the reference mix, and the vocals are not prominent enough, requiring processing to compensate for this gap; when the actual clarity has reached or exceeded the target clarity, it means that the clarity of that frequency band has met the standard at that moment, and no processing is required.
[0046] Based on the above comparison results, for each frequency band and time point requiring processing, the energy reduction amount for the accompaniment to be processed in that frequency band and at that time point is determined according to the size of the difference between the actual and target sharpness. The larger the difference, the more severe the masking, and the more accompaniment energy needs to be reduced; the smaller the difference, the less energy needs to be reduced. The above reduction amount is applied to the energy of the accompaniment to be processed in the corresponding frequency band and at the corresponding time point, reducing the energy of the accompaniment at that position, thereby reducing the masking effect on the vocals.
[0047] After reducing all frequency bands and times, the energy of all positions in the accompaniment that are actually less sharp than the target sharpness is reduced, while positions that are sharper than or equal to the target sharpness remain unchanged. Mixing the reduced accompaniment with the vocals restores the sharpness of the mix to a level comparable to the reference mix across all frequency bands and times.
[0048] This restores vocal clarity to the reference level, preserves the original form of the accompaniment to the greatest extent, and avoids the hollowing out of the accompaniment and the imbalance of the arrangement caused by blindly cutting back.
[0049] In this embodiment, the clarity target of each frequency band can be matched with the mixing style of the specific song, avoiding the target mismatch problem caused by manually setting a fixed threshold. Furthermore, the backing track to be processed is reduced according to the clarity difference, so that the reduction operation is accurately applied to the specific location where the masking occurs. This effectively avoids the backing track hollowing and arrangement imbalance caused by blind reduction, preserves the original form of the backing track and the integrity of the arrangement to the greatest extent, improves the naturalness of the listening experience, reduces the workload of manual adjustment in mixing production, and improves the automation and intelligence level of spectrum masking processing.
[0050] In the above Figure 1 Based on the corresponding embodiments, in order to more clearly demonstrate the process of obtaining target sharpness, this application also provides a possible implementation of obtaining target sharpness in a reference-driven spectrum demasking method. Figure 2 This is a schematic flowchart illustrating the process of obtaining target sharpness in a reference-driven spectral demasking method provided in an embodiment of this application. Figure 2 As shown, in S102 above, the target intelligibility of each frequency band is determined based on the reference vocal energy and the reference accompaniment energy, including: S210, in each frequency band, selects the active frames corresponding to the reference human voice.
[0051] Specifically, for each Bark frequency band, a complete sequence of the reference human voice energy changing over time is obtained, and the statistical distribution characteristics of the sequence are analyzed. Using the energy value corresponding to the global high percentile (e.g., the 99th percentile) of the reference human voice energy as a reference benchmark, the reference human voice energy of each frame is compared with this benchmark, and frames whose energy exceeds the benchmark by a certain percentage (e.g., 5%) are selected as active frames.
[0052] In this way, only frames in which the human voice energy is significantly higher than its global quiet level are included in the subsequent statistics, effectively eliminating the interference of silent segments and non-singing segments, and ensuring that the calculation samples for target clarity all come from the effective range of actual human voice production.
[0053] S220: Extract the reference vocal energy and reference accompaniment energy corresponding to each active frame, and calculate the energy ratio of the active frames.
[0054] Specifically, for each active frame, the energy value corresponding to that frame is extracted from both the reference vocal energy sequence and the reference accompaniment energy sequence, using the frame's time as an index. Then, the ratio of the reference vocal energy to the reference accompaniment energy for that frame is calculated. This ratio reflects the instantaneous prominence of the reference vocal relative to the reference accompaniment at that specific moment and frequency band. A larger ratio indicates that the vocal is relatively prominent and has higher clarity in that frequency band; a smaller ratio indicates that the vocal is severely masked by the accompaniment and has lower clarity.
[0055] Iterate through all active frames and perform the above ratio calculation independently for each active frame to obtain the energy ratio of each active frame.
[0056] S230 determines the target sharpness based on the energy ratio of each active frame.
[0057] Specifically, the first step is to determine whether the number of active frames is sufficient and whether the energy levels of each active frame are reliable. If the number of active frames is too small, it indicates that the frequency band is rarely occupied in the reference vocals, and the statistical sample size is insufficient to support reliable target sharpness estimation. If the median energy of the active frames is too low, it indicates that even when the vocals are active, the energy of this frequency band is still close to the noise floor level, and the reference vocals do not actually contribute meaningful energy to this frequency band. In both of these cases, the frequency band is marked as a band where the reference vocals are basically silent, and the occupancy mask is set to 0, the target sharpness is set to 1, and this frequency band is not involved in sharpness comparison and accompaniment reduction in subsequent processing.
[0058] If the number of active frames is sufficient and the median energy reaches a reliable level, the median of the energy ratios of all active frames in that frequency band is taken as the basic target sharpness. This median value is then clipped, with values exceeding the range truncated to boundary values to ensure numerical stability. The clipped value is the target sharpness for that frequency band. Simultaneously, the occupancy mask for that frequency band is set to 1, marking it as the frequency band actually occupied by the reference vocals. This frequency band is allowed to participate in sharpness comparison and accompaniment reduction in subsequent processing.
[0059] By traversing all Bark frequency bands, the occupancy mask markers for each band and their corresponding target sharpness are obtained. In this way, the occupancy mask effectively excludes frequency bands where human voices are barely heard, avoiding meaningless reduction in invalid frequency bands; the target sharpness is robustly extracted from the active frames of the valid frequency bands, providing an accurate and reliable reference standard for subsequent processing.
[0060] In one possible implementation, the target sharpness can also be taken as a high percentile (such as the 75th or 90th percentile) of the energy ratios of all active frames. Compared to the median, the high percentile better reflects the state where vocals are more prominent than the accompaniment in the reference mix; alternatively, the mean or median of the energy ratio sequence of active frames can be taken after smoothing.
[0061] In this embodiment, by filtering active frames to eliminate interference from silent segments, it is ensured that the statistical samples of target sharpness all come from the effective range of actual human voice output, which increases the effectiveness of the target sharpness calculation samples. The median statistics reduce the deviation of individual extreme ratios and improve the robustness of target sharpness.
[0062] In the above Figure 1 Based on the corresponding embodiments, in order to more clearly demonstrate the process of obtaining actual sharpness, this application also provides a possible implementation of obtaining actual sharpness in a reference-driven spectrum demasking method. Figure 3 This is a schematic flowchart illustrating the process of obtaining actual sharpness in a reference-driven spectral demasking method provided in an embodiment of this application. Figure 3 As shown, in S104 above, the determination of the actual clarity of each frequency band based on the vocal energy and accompaniment energy to be processed includes: S310: In each frequency band, for all time frames, extract the vocal energy and accompaniment energy to be processed for each time frame, and calculate the energy ratio of each time frame.
[0063] Actual clarity needs to reflect the energy contrast between the vocals and accompaniment at each moment, therefore, all time frames need to be traversed. Specifically, for each time frame, the energy of the vocals and accompaniment to be processed in each frequency band is extracted, and the ratio between the two is calculated. This ratio reflects the degree to which the vocals are masked by the accompaniment at that moment. A larger ratio indicates that the vocals in that frequency band are relatively prominent, and the actual clarity is high; a smaller ratio indicates that the vocals are severely masked by the accompaniment, and the actual clarity is low.
[0064] S320 determines the actual sharpness based on the energy ratio of each time frame.
[0065] After obtaining the energy ratios for each frequency band and each time frame, these energy ratios are directly used as the actual sharpness of each corresponding time frame in each frequency band, and each frame has a corresponding sharpness value in each frequency band.
[0066] In the context of a vocal track application, the accompaniment to be processed is the same as the reference accompaniment itself. In this case, the frequency band energy of the accompaniment to be processed in S310 can directly reuse the energy data of the reference accompaniment that has already been measured in S101, without having to repeatedly perform time-frequency transformation and frequency band energy summation, thus saving computational resources. In general scenarios, if the accompaniment to be processed and the reference accompaniment are different audio materials, then the energy measurement and extraction must be performed independently.
[0067] In this embodiment, the energy ratio is calculated frame by frame to avoid information loss. The actual clarity truly reflects the actual masking state of the current mix at each moment and in each frequency band, providing a comprehensive data foundation for subsequent comparison with the target clarity.
[0068] In the above Figure 1 Based on the corresponding embodiments, in order to more clearly demonstrate the process of obtaining the demasked accompaniment, this application also provides a possible implementation of obtaining the demasked accompaniment in a reference-driven spectrum demasking method. Figure 4 This is a schematic diagram illustrating the process of obtaining the demasked accompaniment in a reference-driven spectrum demasking method provided in an embodiment of this application. Figure 4 As shown, in step S105 above, based on the target sharpness and actual sharpness of each frequency band, energy reduction is performed on the accompaniment to be processed, resulting in the unmasked accompaniment including: S410 determines the resolution gap for each frequency band based on the target resolution and actual resolution of each frequency band.
[0069] Specifically, for each Bark band, the target sharpness of that band is compared with the actual sharpness of that band at each moment. Specifically, the sharpness gap (dB) = 10·log10(target sharpness / actual sharpness). After calculation, negative values are set to zero, and non-negative values are retained as the sharpness gap for that band and that moment. When the target sharpness is greater than the actual sharpness, the gap is positive, indicating that the prominence of human voices in that band at that moment does not meet the reference standard; when the target sharpness is less than or equal to the actual sharpness, the gap is negative or zero, indicating that the sharpness of that band at that moment has met the standard and no further processing is needed.
[0070] S420, calculates the bandwidth reduction based on the resolution gap in each frequency band.
[0071] Specifically, for each location with a sharpness gap (i.e., the actual sharpness of that frequency band at that moment is lower than the target sharpness), the corresponding frequency band reduction is determined based on the size of the gap. A larger sharpness gap indicates more severe masking at that location, requiring more accompaniment energy to be reduced; a smaller sharpness gap results in a correspondingly smaller reduction. For locations with zero sharpness gap, the frequency band reduction is zero, meaning no processing is needed. Through this method, the sharpness gaps of each frequency band and at each moment are converted into a frequency band reduction distribution that varies with frequency band and time.
[0072] S430 performs energy reduction on the accompaniment to be processed based on the bandwidth reduction amount, resulting in the demasked accompaniment.
[0073] After obtaining the frequency band reduction amount for each frequency band and each time point, the reduction amount is applied to the corresponding frequency band and time point of the accompaniment to be processed, reducing the energy of the accompaniment at that position. After the reduction is completed, the modified accompaniment signal is output, which is the demasked accompaniment.
[0074] In this embodiment, the amount of reduction is determined by the size of the clarity gap, so that the reduction range matches the degree of masking; the integrity of the accompaniment is improved, and reduction is only carried out in the frequency bands and times that do not meet the standards, while the positions that meet the standards are not affected; the arrangement imbalance caused by blind reduction is reduced, and the original form of the accompaniment is preserved to the greatest extent.
[0075] In the above Figure 4 Based on the corresponding embodiments, in order to more clearly demonstrate the process of obtaining the bandwidth reduction, this application also provides a possible implementation of obtaining the bandwidth reduction in a reference-driven spectrum demasking method. Figure 5 This is a schematic flowchart illustrating the process of obtaining bandwidth reduction in a reference-driven spectrum demasking method provided in an embodiment of this application. Figure 5 As shown, in the above S420, the calculation of the bandwidth reduction based on the resolution gap of each bandwidth includes: S510 generates an occupancy mask based on the energy distribution of the reference human voice in each frequency band.
[0076] The occupancy mask is used to mark the frequency band occupied by the reference human voice.
[0077] Specifically, for each Bark frequency band, the sequence of reference voice energy changing over time is obtained. When the reference voice energy in a certain frequency band is at an extremely low level, it indicates that the band is essentially silent in the reference voice, and the occupancy mask of that band is set to 0. Conversely, if there is an interval in the frequency band where the energy is significantly higher than the noise floor level, it indicates that the band is actually occupied by the reference voice, and the occupancy mask of that band is set to 1. By marking the occupancy mask, it can be ensured that the reduction operation is only performed on the frequency bands actually occupied by the reference voice.
[0078] S520 obtains the vocal activity level based on the energy distribution of the vocal voice in each frame.
[0079] Specifically, for each Bark band, the global high percentile value (e.g., the 99th percentile) of the vocal energy to be processed in that band is first obtained as a reference benchmark, and a relative floor value (e.g., 5% of the benchmark) is set. For each time frame, the ratio of the vocal energy to be processed in that frame to the floor value is calculated and subtracted by 1. The result is truncated to the interval between 0 and 1, which is taken as the vocal activity level of that frame in that band. When the energy level of a frame is lower than or close to the floor value, the vocal activity level is 0, indicating that the frame is a pure accompaniment segment and no cuts should be triggered; when the energy level is significantly higher than the floor value, the vocal activity level approaches 1, indicating that the frame is a vocal active segment and cuts are allowed; for states in between, the vocal activity level continuously takes values between 0 and 1 to achieve a smooth transition.
[0080] In one possible implementation, vocal activity can be combined with time-frequency domain features such as zero-crossing rate or spectral flatness to distinguish between voiced and unvoiced sounds, thereby more accurately locating the frequency band and time when the voice truly occupies the signal. Specifically, for each time frame, the zero-crossing rate or spectral flatness features of the vocal signal to be processed in that frame are extracted. The zero-crossing rate reflects the number of times the signal waveform crosses the zero level. Voiced segments are periodic and have a low zero-crossing rate, while unvoiced segments have noise-like characteristics and a high zero-crossing rate. Spectral flatness reflects the flatness of the signal spectrum. The spectral energy of voiced segments is concentrated at the fundamental frequency and its harmonics, resulting in a low spectral flatness, while the spectral energy distribution of unvoiced segments is more uniform, resulting in a high spectral flatness. After normalizing the above feature values, they are used as auxiliary judgment factors and comprehensively judged together with the vocal activity calculated based on the relative energy level. When the activity obtained based on the energy level is in the intermediate ambiguity range, the information provided by the zero-crossing rate or spectral flatness is used to assist in judging whether the frame is a vocal activity frame, thereby further improving the accuracy of vocal activity detection and reducing the risk of false triggering or missed triggering that may be caused by pure energy detection.
[0081] The S530 filters the sharpness gaps based on the occupancy mask and the activity level of human voices, resulting in the filtered sharpness gaps.
[0082] Specifically, the reference occupancy mask and vocal activity are used as two coefficients for multiplication. For frequency bands where the reference occupancy mask is 0, the gap is directly set to zero; for moments where vocal activity is 0, the gap is also directly set to zero; the gap is retained only when both are non-zero values. The resulting filtered sharpness gap has a non-zero value only when the reference vocal activity is actually present in the frequency band it occupies and when the vocal activity to be processed is actually active.
[0083] S540 determines the weight of each frequency band based on the degree of influence of each frequency band on the intelligibility of human voice.
[0084] Different frequency bands contribute significantly to vocal intelligibility. Vocal intelligibility is primarily concentrated in the mid-frequency range of approximately 1kHz to 5kHz, where key features such as sibilance and resonance are most concentrated, contributing the most to intelligibility. The low-frequency range mainly contributes to the fullness of the sound, contributing less to intelligibility; the high-frequency range mainly contributes to the airiness of the sound, also contributing limitedly. Therefore, it is necessary to determine the weighting values based on the degree of influence of each frequency band on vocal intelligibility.
[0085] In one specific embodiment, the frequency band weight is 0.3 for frequencies below 250Hz, 0.7 for frequencies from 250Hz to 1000Hz, 1.0 for frequencies from 1000Hz to 5000Hz, 0.7 for frequencies from 5000Hz to 8000Hz, and 0.4 for higher frequency bands. A higher weight value indicates that the frequency band is more important to human voice intelligibility and will have a greater weight in subsequent reduction calculations.
[0086] S550 calculates the bandwidth reduction based on the resolution gap after filtering and the weight of each frequency band.
[0087] Specifically, for each frequency band and each moment, the filtered clarity gap is multiplied by the corresponding frequency band's weight value, multiplied by the vocal activity level of the same frequency band at that moment, and then multiplied by a total intensity coefficient. Here, the clarity gap reflects the clarity difference in the current mix at that location; the weight reflects the importance of that frequency band to vocal intelligibility; vocal activity reflects whether the vocals are truly engaged at that moment; and the total intensity coefficient is a user-adjustable global parameter used to control the overall aggressiveness of the cut. The product of these four items is the frequency band cut amount for that frequency band and that moment.
[0088] By iterating through all frequency bands and all times, the above calculations are performed independently to finally obtain the frequency band reduction for each frequency band and each time.
[0089] In this embodiment, the reference occupancy mask and vocal activity filtering ensure that the reduction is performed only in the effective frequency band and when the vocals are active, avoiding false triggering of pure accompaniment segments and meaningless processing of invalid frequency bands; the perceptual weight ensures the priority of reduction and reduces the destruction of accompaniment caused by blind processing.
[0090] In the above Figure 4 Based on the corresponding embodiments, to more clearly demonstrate the process of smoothing and limiting the bandwidth reduction, this application also provides a possible implementation of bandwidth reduction smoothing and limiting in a reference-driven spectrum demasking method. Optionally, in S430, before energy reduction of the accompaniment to be processed according to the bandwidth reduction, the method further includes: S610 performs smoothing and limiting on the bandwidth reduction to obtain the final reduction.
[0091] Specifically, time smoothing is performed along the time axis. For each frequency band, its attenuation sequence in each time frame is arranged chronologically and processed using a single-pole smoother with an attack time constant and a release time constant. The attack time constant controls the response speed when the attenuation increases, and the release time constant controls the recovery speed when the attenuation decreases.
[0092] In one specific embodiment, the attack time constant is set to 12 milliseconds, and the release time constant is set to 120 milliseconds. When the cutoff amount in a frame is higher than the previous frame, smoothing is performed based on the attack time constant to allow the cutoff amount to build up quickly; when the cutoff amount in a frame is lower than the previous frame, smoothing is performed based on the release time constant to allow the cutoff amount to recover slowly. In this way, the abruptness of the sound caused by sudden changes in cutoff amount is avoided, while making the recovery process of the accompaniment energy more natural and smooth.
[0093] In one possible implementation, temporal smoothing can also employ look-ahead smoothing. Specifically, when calculating the reduction amount for the current frame, energy information from several future frames is obtained in advance, causing the rising edge of the reduction amount to be established earlier. This further suppresses the transient pumping sensation at the onset of the sound, resulting in a more natural smoothing effect. Look-ahead smoothing can maintain a good auditory experience even when the onset time constant is small.
[0094] Frequency smoothing is performed along the frequency band axis. For each time frame, the reduction amount in each frequency band is arranged from low to high frequency, and the reduction amount of adjacent frequency bands is smoothed to ensure a continuous transition between frequency bands, avoiding the comb filtering effect caused by the reduction amount of one frequency band being much higher than that of adjacent frequency bands.
[0095] Limiting is applied to the reduction amount for each frequency band and each moment. A single-band reduction upper limit is set (e.g., 4 to 9 dB, which can be adaptively adjusted according to the size of the clarity gap). If the reduction at a certain position exceeds this upper limit, it is truncated to the upper limit value. Limiting prevents excessive reduction due to extreme gap values and avoids the accompaniment energy being completely drained in a particular frequency band.
[0096] The reduction amount obtained after time smoothing, frequency smoothing and amplitude limiting is the final reduction amount.
[0097] Accordingly, S430 performs energy reduction on the accompaniment to be processed based on the bandwidth reduction amount, resulting in the demasked accompaniment, including: S620 performs energy reduction on the accompaniment to be processed based on the final reduction amount, resulting in the unmasked accompaniment.
[0098] The final reduction has been smoothed in both the time and frequency dimensions, and the reduction amplitude of each frequency band is limited to a reasonable range. When the energy of the accompaniment to be processed is reduced, the vocal clarity is restored to the reference level, while the accompaniment's timbre transitions naturally and without jitter, and no frequency band is excessively attenuated. After the reduction is completed, the modified accompaniment signal is output, which is the demasked accompaniment.
[0099] In this embodiment, by performing time and frequency smoothing on the bandwidth reduction amount and setting an upper limit on the bandwidth reduction amount of each bandwidth, the vocal clarity is restored to the reference level after the final reduction amount is applied to the accompaniment to be processed, while the timbre transition of the accompaniment is natural and the bandwidth energy structure is complete, which significantly improves the naturalness and smoothness of the listening experience.
[0100] In the above Figure 4 Based on the corresponding embodiments, in order to more clearly demonstrate the process of obtaining the demasked accompaniment, this application also provides a possible implementation of obtaining the demasked accompaniment in a reference-driven spectrum demasking method. Figure 6 This is a schematic diagram illustrating the process of obtaining the demasked accompaniment in another reference-driven spectrum demasking method provided in an embodiment of this application. For example... Figure 6 As shown, in S430 above, the energy of the accompaniment to be processed is reduced according to the bandwidth reduction amount, resulting in the unmasked accompaniment including: S710 maps the bandwidth reduction amount to each frequency point of the time-frequency conversion to obtain the reduction gain at each frequency point.
[0101] After obtaining the final reduction amount for each frequency band and at each time point, the reduction amount based on the Bark frequency band needs to be allocated to each frequency point of the Short Time Fourier Transform (STFT) in order to perform the actual energy reduction operation in the frequency domain. Since each Bark frequency band corresponds to multiple frequency points in the STFT domain, the reduction amount of a single frequency band needs to be mapped to all frequency points covered by that frequency band.
[0102] Specifically, for each Bark frequency band, the corresponding STFT frequency point index set is obtained, and the reduction amount of the frequency band is assigned to each frequency point in the index set. For frequency points at the boundary of adjacent frequency bands, a linear interpolation method can be used for smooth transition to avoid abrupt changes in the reduction amount between frequency points.
[0103] The reduction amount, measured in decibels (dB), represents the energy reduction required for the accompaniment at a given frequency band and time. To apply this reduction to the accompaniment's amplitude in the frequency domain, the decibel value must be converted to a linear amplitude ratio. Specifically, the reduction amount in dB for each frequency point is substituted into the conversion formula: Linear Gain = 10^(-dB / 20). For example, a 3dB reduction results in a linear gain of approximately 0.708, meaning the amplitude at that frequency point needs to be multiplied by 0.708 to achieve a 3dB energy reduction. Through this conversion, the reduction amount, originally measured in decibels, is transformed into a linear gain value ranging from 0 to 1.
[0104] Through the above mapping and transformation, the original reduction distribution with frequency band as the granularity is transformed into a linear reduction gain consistent with the number of STFT frequency points. Each frequency point has a corresponding reduction gain value in each time frame, with the value ranging from 0 to 1.
[0105] In one possible implementation, attenuation gain can also be applied in the constant Q transform (CQT) domain. When mapping the attenuation amount of each Bark band to the CQT frequency point, a method similar to STFT mapping can be used: obtain the CQT frequency point corresponding to each Bark band, assign the attenuation amount to the corresponding CQT frequency point; after converting the decibel value to a linear gain, multiply it frame by frame onto the CQT coefficients of each accompaniment channel, and then recover the time-domain signal through inverse CQT transformation.
[0106] S720 performs time-frequency transformation on the accompaniment to be processed to obtain the spectrum of the accompaniment.
[0107] Specifically, a short-time Fourier transform is performed on the accompaniment to be processed, using the same parameters as in S101 (n_fft=2048, hop=512), to obtain the spectrum of the accompaniment to be processed. This spectrum contains the amplitude and phase information of the accompaniment at each frequency point and each time frame.
[0108] S730 adjusts the amplitude of each frequency point in the accompaniment spectrum to be processed according to the reduction gain, and obtains the adjusted accompaniment spectrum.
[0109] Specifically, for each time frame and each frequency point, the amplitude value of the accompaniment spectrum to be processed at that position is multiplied by the corresponding reduction gain (the reduction gain is a linear amplitude ratio, ranging from 0 to 1). The phase information remains unchanged; only the magnitude of the amplitude is adjusted. The amplitude of frequency points with a reduction gain of 1 remains unchanged, while the amplitude of frequency points with a reduction gain less than 1 is reduced accordingly. By traversing all frequency points and all time frames, the amplitude adjustment for all positions that should be reduced is completed, resulting in the adjusted accompaniment spectrum.
[0110] S740 performs an inverse transformation on the adjusted accompaniment spectrum to obtain the unmasked accompaniment.
[0111] Specifically, an inverse short-time Fourier transform (ISTFT) is performed on the adjusted accompaniment spectrum to convert the frequency domain representation back to a time domain waveform. The inverse transform uses the same window function and frame shift parameter as the forward transform to ensure the quality of signal reconstruction. The time domain signal obtained after the inverse transform is the demasked accompaniment.
[0112] Unlike traditional multi-band frequency divider schemes, this embodiment completes the reduction operation entirely in the STFT frequency domain. That is, by directly applying the reduction gain to the amplitude of the STFT frequency point, and then recovering the time domain signal through inverse transformation, there is no frequency divider link of frequency division, filtering, and re-combining in the whole process, thus fundamentally avoiding the phase distortion problem introduced by multi-band frequency dividers.
[0113] In this embodiment, by mapping the bandwidth reduction amount to each frequency point of the STFT, the reduction operation can be performed on a frequency point basis. Finally, the time domain signal is recovered through inverse transformation, so that the output audio can be directly used for subsequent mixing. While achieving accurate spectrum reduction, the phase distortion introduced by traditional crossovers is avoided, and the timbre and listening effect of the output audio are improved.
[0114] The following describes a reference-driven spectrum demasking device and electronic device provided in this application, which are used to implement the application. The specific implementation process and technical effects are described above and will not be repeated below.
[0115] Figure 7 A schematic diagram of a reference-driven spectrum demasking device provided in an embodiment of this application is shown below. Figure 7 As shown, the reference-driven spectrum demasking device includes: The measurement module 1000 is used to measure the energy of the reference human voice and the energy of the reference accompaniment in each frequency band, based on the reference human voice and the reference accompaniment.
[0116] The calculation module 2000 is used to determine the target intelligibility of each frequency band based on the reference vocal energy and the reference accompaniment energy.
[0117] The measurement module 1000 is also used to measure the energy of the vocal and accompaniment to be processed in each frequency band, based on the vocal and accompaniment to be processed.
[0118] The calculation module 2000 is also used to determine the actual clarity of each frequency band based on the energy of the vocals and the accompaniment to be processed.
[0119] The reduction module 3000 is used to reduce the energy of the accompaniment to be processed based on the target clarity and actual clarity of each frequency band, so as to obtain the demasked accompaniment.
[0120] Optionally, the calculation module 2000 is specifically used to filter out the active frames corresponding to the reference human voice in each frequency band; extract the reference human voice energy and reference accompaniment energy corresponding to each active frame, calculate the energy ratio of the active frames; and determine the target sharpness based on the energy ratio of each active frame.
[0121] Optionally, the calculation module 2000 is specifically used to extract the vocal energy and accompaniment energy to be processed for each time frame in each frequency band for all time frames, calculate the energy ratio of each time frame, and determine the actual clarity based on the energy ratio of each time frame.
[0122] Optionally, the calculation module 2000 is further configured to determine the resolution gap of each frequency band based on the target resolution and actual resolution of each frequency band; and to calculate the frequency band reduction amount based on the resolution gap of each frequency band.
[0123] Optionally, the reduction module 3000 is specifically used to reduce the energy of the accompaniment to be processed according to the bandwidth reduction amount, so as to obtain the demasked accompaniment.
[0124] Optionally, the calculation module 2000 is specifically used to generate an occupancy mask based on the energy distribution of the reference human voice in each frequency band; the reference occupancy mask is used to mark the frequency band occupied by the reference human voice; the human voice activity is obtained based on the energy distribution of the human voice to be processed in each frame; the intelligibility gap is filtered based on the occupancy mask and the human voice activity to obtain the filtered intelligibility gap; the weight of each frequency band is determined based on the degree of influence of each frequency band on the intelligibility of the human voice; and the frequency band reduction is calculated based on the filtered intelligibility gap and the weight of each frequency band.
[0125] Optionally, the reference-driven spectrum demasking device further includes a smoothing and limiting module for smoothing and limiting the bandwidth reduction to obtain the final reduction.
[0126] Optionally, the reduction module 3000 is also used to reduce the energy of the accompaniment to be processed according to the final reduction amount, so as to obtain the unmasked accompaniment.
[0127] Optionally, the reduction module 3000 is specifically used to map the bandwidth reduction amount to each frequency point of the time-frequency transformation to obtain the reduction gain of each frequency point; to perform time-frequency transformation on the accompaniment to be processed to obtain the spectrum of the accompaniment to be processed; to adjust the amplitude of each frequency point in the spectrum of the accompaniment to be processed according to the reduction gain to obtain the adjusted accompaniment spectrum; and to perform inverse transformation on the adjusted accompaniment spectrum to obtain the demasked accompaniment.
[0128] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more digital signal processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).
[0129] Figure 8 This is a schematic diagram of an electronic device provided in an embodiment of this application. The device may be a computing device or a server with computing processing capabilities.
[0130] The electronic device 10 includes a processor 11, a storage medium 12, and a bus 13. The storage medium 12 stores program instructions executable by the processor 11. When the electronic device 10 is executed, the processor 11 communicates with the storage medium 12 via the bus 13, and the processor 11 executes the program instructions to perform the above-described method embodiment. The specific implementation and technical effects are similar and will not be described in detail here.
[0131] Optionally, this application also provides a program product, such as a computer-readable storage medium, including a program that, when executed by a processor, performs the above-described method embodiments.
[0132] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0133] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0134] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.
[0135] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0136] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A reference-driven spectral demasking method, characterized in that, The method includes: Based on the reference vocal and reference accompaniment, the energy of the reference vocal and reference accompaniment in each frequency band is measured respectively. The target intelligibility of each frequency band is determined based on the reference vocal energy and the reference accompaniment energy. Based on the vocal and accompaniment to be processed, the energy of the vocal and accompaniment to be processed in each frequency band is measured respectively. The actual clarity of each frequency band is determined based on the vocal energy to be processed and the accompaniment energy to be processed. Based on the target sharpness and the actual sharpness of each frequency band, the accompaniment to be processed is energy-reduced to obtain the demasked accompaniment.
2. The method according to claim 1, characterized in that, The step of determining the target intelligibility of each frequency band based on the reference vocal energy and the reference accompaniment energy includes: In each of the frequency bands, the active frames corresponding to the reference human voice are selected; Extract the energy of the reference vocals and the reference accompaniment in the active frame, and calculate the energy ratio of the active frame; The target sharpness is determined based on the energy ratio of each active frame.
3. The method according to claim 1, characterized in that, The step of determining the actual clarity of each frequency band based on the vocal energy to be processed and the accompaniment energy to be processed includes: In each frequency band, for all time frames, the vocal energy and accompaniment energy to be processed corresponding to each time frame are extracted, and the energy ratio of each time frame is calculated. The actual sharpness is determined based on the energy ratio of each time frame.
4. The method according to claim 1, characterized in that, The step of performing energy reduction on the accompaniment to be processed based on the target sharpness and the actual sharpness of each frequency band to obtain the demasked accompaniment includes: Based on the target sharpness and the actual sharpness of each frequency band, the sharpness gap of each frequency band is determined; Calculate the bandwidth reduction based on the resolution gap of each bandwidth; Based on the bandwidth reduction amount, the energy of the accompaniment to be processed is reduced to obtain the demasked accompaniment.
5. The method according to claim 4, characterized in that, The step of calculating the bandwidth reduction based on the resolution gap of each bandwidth includes: Based on the energy distribution of the reference human voice in each of the frequency bands, an occupancy mask is generated, which is used to mark the frequency band occupied by the reference human voice; The vocal activity level is obtained based on the energy distribution of the vocal voice to be processed in each frame; Based on the occupancy mask and the vocal activity level, the sharpness gap is filtered to obtain the filtered sharpness gap; The weight of each frequency band is determined based on the degree of influence of each frequency band on the intelligibility of human voice. The bandwidth reduction is calculated based on the resolution gap after filtering and the weight of each bandwidth.
6. The method according to claim 4, characterized in that, Before performing energy reduction on the accompaniment to be processed based on the bandwidth reduction amount, the method further includes: The bandwidth reduction is smoothed and limited to obtain the final reduction. The step of reducing the accompaniment to be processed according to the bandwidth reduction amount to obtain the demasked accompaniment includes: The energy of the accompaniment to be processed is reduced according to the final reduction amount to obtain the unmasked accompaniment.
7. The method according to claim 4, characterized in that, The step of reducing the energy of the accompaniment to be processed according to the bandwidth reduction amount to obtain the demasked accompaniment includes: The bandwidth reduction is mapped to each frequency point of the time-frequency transformation to obtain the reduction gain at each frequency point; The accompaniment to be processed is subjected to time-frequency transformation to obtain the spectrum of the accompaniment to be processed; Based on the reduction gain, the amplitude of each frequency point in the accompaniment spectrum to be processed is adjusted to obtain the adjusted accompaniment spectrum; The adjusted accompaniment spectrum is inversely transformed to obtain the unmasked accompaniment.
8. A reference-driven spectrum demasking device, characterized in that, The device includes: The measurement module is used to measure the energy of the reference human voice and the energy of the reference accompaniment in each frequency band, based on the reference human voice and the reference accompaniment. The calculation module is used to determine the target intelligibility of each frequency band based on the reference human voice energy and the reference accompaniment energy. The measurement module is also used to measure the energy of the human voice to be processed and the energy of the accompaniment to be processed in each frequency band according to the human voice to be processed and the accompaniment to be processed. The calculation module is also used to determine the actual clarity of each frequency band based on the vocal energy to be processed and the accompaniment energy to be processed; The reduction module is used to reduce the energy of the accompaniment to be processed according to the target sharpness and the actual sharpness of each frequency band, so as to obtain the demasked accompaniment.
9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores program instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus, and the processor executes the program instructions to perform the steps of the reference-driven spectrum demasking method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, performs the steps of the reference-driven spectrum demasking method as described in any one of claims 1 to 7.