Spectrogram generation method, device and equipment and computer readable storage medium

Abstract

The application discloses a kind of spectrogram generation method, device, equipment and computer readable storage medium;Specifically, the speech signal of target voice segment is framed and handled, and then multiple frames of target speech signals are extracted discontinuously;The amplitude value of the frequency domain signal corresponding to the multiple frames of target speech signals is used to generate the normalized amplitude spectrum corresponding to each frame of target speech signal, and the normalized amplitude spectrum is logarithmically operated to obtain the logarithmic amplitude spectrum corresponding to each frame of target audio signal, so as to generate the target spectrogram of target voice segment. Thus, the speech signal of target voice segment can be framed and handled first, and then the spectrogram pixel set of each frame of extracted speech signal is calculated, and the spectrogram pixel sets of multiple speech signals are spliced into the target spectrogram of target voice segment in time sequence;Therefore, the amount of calculation of spectrogram is reduced, the efficiency of spectrogram generation is improved, and the quality of spectrogram is guaranteed.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a spectrogram generation method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] The display quality of spectrograms has a significant impact on the identification of voiceprint features. In voiceprint detection, features in spectrograms, such as formants and spectral lines, can be used for analysis and identification. Furthermore, some features in voiceprints are closely related to indicators such as frequency distribution and energy distribution in spectrograms. Therefore, high-quality spectrograms facilitate rapid and accurate location of voiceprint features by identification personnel, while low-quality spectrograms may obscure certain voiceprint features, causing difficulties for identification personnel. Thus, continuous optimization of speech segment parameters is necessary to generate high-quality spectrograms. Related techniques can be used to manually adjust speech segment parameters using parameter adjustment software to generate high-quality spectrograms.

[0003] However, while the technology can generate spectrograms, it relies too heavily on the experience of technicians and requires constant manual adjustment of spectrogram parameters, which increases the complexity and difficulty of spectrogram generation, thereby reducing the efficiency of spectrogram generation and hindering the maintenance of spectrogram quality. Summary of the Invention

[0004] This application provides a spectrogram generation method, apparatus, device, and computer-readable storage medium, which can automatically adjust spectrogram parameters, generate high-quality spectrograms, reduce the complexity and difficulty of spectrogram generation, and improve the efficiency of spectrogram generation.

[0005] This application provides a spectrogram generation method, including:

[0006] The speech signal of the target speech segment is acquired, the speech signal is processed into frames to obtain multiple frames of speech signal, and multiple frames of target speech signal are extracted discontinuously from the multiple frames of speech signal.

[0007] The multi-frame target speech signal is frequency domain converted to obtain multi-frame frequency domain signal, and the target amplitude spectrum corresponding to each frame target speech signal is generated according to the amplitude value of the multi-frame frequency domain signal.

[0008] The target amplitude spectrum corresponding to each frame of the target audio signal is normalized to obtain a normalized amplitude spectrum, and a logarithmic operation is performed on the normalized amplitude spectrum to obtain the logarithmic amplitude spectrum corresponding to each frame of the target audio signal.

[0009] Based on the logarithmic amplitude spectrum corresponding to each frame of the target speech signal, a target spectrogram of the target speech segment is generated.

[0010] Accordingly, embodiments of this application provide a spectrogram generation apparatus, including:

[0011] The acquisition unit is used to acquire the speech signal of the target speech segment, perform frame segmentation processing on the speech signal to obtain multiple frames of speech signal, and non-continuously extract multiple frames of target speech signal from the multiple frames of speech signal.

[0012] The conversion unit is used to perform frequency domain conversion on the multi-frame target speech signal to obtain multi-frame frequency domain signal, and generate the target amplitude spectrum corresponding to each frame target speech signal according to the amplitude value of the multi-frame frequency domain signal.

[0013] The arithmetic unit is used to normalize the target amplitude spectrum corresponding to each frame of the target audio signal to obtain a normalized amplitude spectrum, and to perform logarithmic operation on the normalized amplitude spectrum to obtain the logarithmic amplitude spectrum corresponding to each frame of the target audio signal.

[0014] The generation unit is used to generate a target spectrogram of the target speech segment based on the logarithmic amplitude spectrum corresponding to each frame of the target speech signal.

[0015] In some embodiments, the spectrogram generation apparatus further includes a determining unit, configured to:

[0016] Determine the target number of frames based on the preset image resolution;

[0017] The acquisition unit includes:

[0018] Non-continuous extraction of the target number of target speech signals from the multi-frame speech signals. 。

[0019] In some embodiments, the conversion unit is further configured to:

[0020] Calculate the average amplitude value of the multi-frame frequency domain signal based on the amplitude values ​​of the multi-frame frequency domain signal;

[0021] The target amplitude spectrum corresponding to each frame of the target speech signal is obtained by subtracting the amplitude value of each frame of the frequency domain signal from the average amplitude value.

[0022] In some embodiments, the arithmetic unit is further configured to:

[0023] Obtain the amplitude value of each frequency point in the target amplitude spectrum corresponding to each frame of the target speech signal;

[0024] Based on the ratio of the amplitude value of each frequency point to the corresponding frame length, a normalized amplitude spectrum corresponding to each frame of the target speech signal is created.

[0025] In some embodiments, the generating unit is further configured to:

[0026] Based on the logarithmic amplitude spectrum corresponding to each frame of the target speech signal, the maximum logarithmic amplitude value is obtained;

[0027] Based on the maximum logarithmic amplitude value, the frequency amplitude in the logarithmic amplitude spectrum corresponding to each frame of the target speech signal is shifted to obtain the target logarithmic amplitude spectrum corresponding to each frame of the target speech signal.

[0028] Convert the target logarithmic amplitude spectrum corresponding to each frame of target speech signal into the corresponding spectrogram;

[0029] The multiple spectrograms are spliced ​​together according to their temporal relationship to obtain the target spectrogram of the target speech segment.

[0030] In some embodiments, the generating unit is further configured to:

[0031] The target logarithmic magnitude spectrum is converted into a visual image;

[0032] Based on the fluctuation range of each frame of the target speech signal in the corresponding target logarithmic amplitude spectrum, different colors are mapped onto the visualization image to obtain the spectrogram corresponding to each frame of the target speech signal.

[0033] In some embodiments, the spectrogram generation apparatus further includes an updating unit, configured to:

[0034] If the target speech segment is updated, the speech signal of the updated target speech segment is processed by frame segmentation to obtain multiple frames of speech signal to be confirmed.

[0035] Based on the multi-frame target speech signal, the candidate speech signal is obtained by deduplicating the multi-frame speech signal to be confirmed.

[0036] Calculate the candidate spectrogram corresponding to the candidate speech signal based on the candidate speech signal;

[0037] The target spectrogram is updated using the candidate spectrogram to obtain the updated target spectrogram of the target speech segment.

[0038] Furthermore, embodiments of this application also provide a computer device, including a processor and a memory, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the steps in the spectrogram generation method provided in embodiments of this application.

[0039] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the spectrogram generation methods provided in embodiments of this application.

[0040] Furthermore, embodiments of this application also provide a computer program product, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in any of the spectrogram generation methods provided in embodiments of this application.

[0041] This application embodiment acquires the speech signal of the target speech segment, performs frame-segmentation processing on the speech signal to obtain multiple frames of speech signal, and non-continuously extracts multiple frames of target speech signal from the multiple frames of speech signal; performs frequency domain transformation on the multiple frames of target speech signal to obtain multiple frames of frequency domain signal, and generates the target amplitude spectrum corresponding to each frame of target speech signal based on the amplitude values ​​of the multiple frames of frequency domain signal; performs normalization processing on the target amplitude spectrum corresponding to each frame of target audio signal to obtain a normalized amplitude spectrum, and performs logarithmic operation on the normalized amplitude spectrum to obtain the logarithmic amplitude spectrum corresponding to each frame of target audio signal; and generates the target spectrogram of the target speech segment based on the logarithmic amplitude spectrum corresponding to each frame of target speech signal. Therefore, this scheme first performs non-continuous frame extraction on the speech signal after the target speech segment is processed into frames. The extracted speech signal is then frequency-domain transformed to obtain the corresponding frequency domain signal. Then, the spectrogram pixel set of each extracted speech signal is calculated. The spectrogram pixel sets of multiple speech signals are then spliced ​​together in chronological order to form the target spectrogram of the target speech segment. In this way, the computational load of the spectrogram is minimized. At the same time, by adjusting the speech signal, the quality of the spectrogram is ensured, reducing the complexity and difficulty of spectrogram generation and improving the efficiency of spectrogram generation. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a schematic diagram of a scenario for the spectrogram generation system provided in an embodiment of this application;

[0044] Figure 2 This is a flowchart illustrating the steps of the spectrogram generation method provided in the embodiments of this application;

[0045] Figure 3 This is a schematic diagram of another step in the spectrogram generation method provided in the embodiments of this application;

[0046] Figure 4 This is a schematic diagram of the spectrogram generation apparatus provided in the embodiments of this application;

[0047] Figure 5 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] This application provides a spectrogram generation method, apparatus, device, and computer-readable storage medium. This application will describe the spectrogram generation apparatus from the perspective of the spectrogram generation apparatus itself. Specifically, the spectrogram generation apparatus can be integrated into a computer device, which can be a terminal device, specifically a terminal device mounted on a spectrogram generation tool, i.e., a voice terminal. Furthermore, the terminal device can also be other types of devices, such as a television, smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, smart wearable device, etc.; however, it is not limited to these.

[0050] For example, see Figure 1 This is a schematic diagram of a scenario for the spectrogram generation system provided in an embodiment of this application. The scenario includes a terminal or a server.

[0051] Specifically, the terminal can be a voice terminal, used to acquire the voice signal of the target voice segment, perform frame segmentation on the voice signal to obtain multiple frames of voice signal, and non-continuously extract multiple frames of target voice signal from the multiple frames of voice signal; perform frequency domain transformation on the multiple frames of target voice signal to obtain multiple frames of frequency domain signal, and generate the target amplitude spectrum corresponding to each frame of target voice signal based on the amplitude values ​​of the multiple frames of frequency domain signal; perform normalization processing on the target amplitude spectrum corresponding to each frame of target audio signal to obtain a normalized amplitude spectrum, and perform logarithmic operation on the normalized amplitude spectrum to obtain the logarithmic amplitude spectrum corresponding to each frame of target audio signal; and generate the target spectrogram of the target voice segment based on the logarithmic amplitude spectrum corresponding to each frame of target voice signal.

[0052] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0053] In this application embodiment, the description will be from the perspective of a spectrogram generation apparatus, which can be specifically integrated into a computer device such as a terminal device or a server. See also Figure 2 , Figure 2This is a schematic flowchart illustrating the steps of a spectrogram generation method provided in an embodiment of this application. Taking a terminal device as an example, the terminal device is a terminal mounted on a spectrogram generation tool. When the processor on the terminal device executes the program corresponding to the spectrogram generation method, the specific process of the spectrogram generation method is as follows:

[0054] 101. Obtain the speech signal of the target speech segment, perform frame segmentation on the speech signal to obtain multiple frames of speech signal, and extract multiple frames of target speech signal discontinuously from the multiple frames of speech signal.

[0055] In this embodiment of the application, in order to generate a spectrogram of a target speech segment, the speech signal of the target speech segment can be acquired and the speech signal can be processed by framing. This allows for frame extraction of the speech signal, thereby minimizing the computational load and increasing the computational speed. Subsequently, the corresponding spectrogram is generated by calculating the extracted speech signal.

[0056] The target speech segment can be the speech data that the user is currently obtaining from a microphone, audio file or other source, or it can be a segment of speech in an audio file or audio stream.

[0057] The speech signal can be an analog signal of the target speech segment, which is a continuous and smooth waveform. Its horizontal axis is the time axis and the vertical axis represents the intensity of the sound.

[0058] The target speech signal can be a single frame of speech signal extracted from all speech signal frames corresponding to the target speech segment.

[0059] Specifically, users can select an audio file or audio stream, convert it into a digital signal, and then use algorithms such as FFT (Fast Fourier Transform) to perform time-domain analysis on the digital signal to obtain the corresponding speech signal. The speech signal result can then be visualized as a time-domain graph. A time-domain graph is typically a waveform diagram on a time axis, with the horizontal axis representing time and the vertical axis representing the amplitude or energy of the speech signal. By locating the position of the preceding speech segment in this time-domain graph, the speech signal corresponding to the target speech segment can be obtained. Since speech signals are non-stationary, directly performing a Fourier transform on the speech signal is meaningless, as the frequency profile of the signal will be lost over time. Since speech signals are short-time stationary, after obtaining the speech signal corresponding to the target speech segment, it is necessary to divide the speech signal into short time frames. By performing a Fourier transform on a single frame of the speech signal within these short time frames, a good approximation of the signal's frequency profile can be obtained. After dividing the target speech segment into multiple frames, the number of frames to be extracted can be determined by framing the segmented speech signal. It's worth noting that the extraction order can be non-contiguous. For example, if a continuous speech signal A is divided into 10,000 single-frame speech signals, and 2,000 frames need to be extracted, one frame can be extracted every four frames. This reduces the computational burden by only calculating the extracted single-frame speech signals.

[0060] Therefore, in some embodiments, the method for generating the spectrogram can also determine the number of frames extracted by the image resolution of the spectrogram. For example, before step 101 "discontinuously extracting multiple frames of target speech signal from multiple frames of speech signal", it may also include:

[0061] Determine the target number of frames based on the preset image resolution;

[0062] Step 101, "non-continuously extracting multiple frames of target speech signals from multiple frames of speech signals," may include:

[0063] Non-continuous extraction of the target target speech signal from a multi-frame speech signal. 。

[0064] The preset image resolution can be the image pixels corresponding to the spectrogram, and can be related to hardware devices such as image displays.

[0065] The target number of frames to be extracted can be the number of signal frames that need to be extracted from a segment of speech signal frames.

[0066] Specifically, after the speech signal of the target speech segment is processed by framing, the sampling rate of each pixel in the image is calculated based on the resolution (number of pixels) and width and height of the preset spectrogram image. Then, the sampling rate of the speech signal is obtained, and the frame rate of the signal is calculated by dividing the signal's sampling rate by the sampling rate of each pixel. Finally, the target number of frames to be sampled can be determined based on the required number of frames and the signal's frame rate. Based on the target number of frames to be sampled, non-continuous frames are extracted from the segmented speech signal to obtain the corresponding multi-frame speech signal.

[0067] For example, if the spectrogram image resolution is 1920x1080 and the signal sampling rate is 44100Hz, and the signal needs to be framed at 30 frames per second, then the sampling rate per pixel can be calculated first: 44100 / (1920x1080) ≈ 0.024Hz. The calculated frame rate is 44100 / 0.024 ≈ 1837500 frames per second. Framed at 30 frames per second, the target number of frames is 1837500 / 30 ≈ 61250 frames. Therefore, in this case, the speech signal needs to be emulated 61250 frames, with a sampling rate of 0.024Hz per pixel.

[0068] It's worth noting that before acquiring the target speech segment's speech signal, the VAD (Voice Activity Detection) algorithm can be used to detect speech segments in the audio to obtain the target speech segment. Specifically, after acquiring the target audio, audio processing is required. The target audio can be speech data from one or more audio files or audio streams. In audio processing, a long speech signal needs to be segmented into many short speech segments, and each segment is processed separately, such as for speech recognition or speech enhancement. Therefore, features in the speech signal, such as short-time energy and zero-crossing rate, can be used to determine whether speech or other actual sounds exist within a given time period. The detected active speech segments can be used to reduce noise, remove irrelevant information, and acquire speech features. For example, in speech recognition tasks, only processing the actual pronunciation portion can correctly identify the speech content; for noise cancellation, only processing the signal containing the actual speech portion can minimize noise. The specific method utilizes the VAD (Voice Activity Detection) algorithm to detect speech segments in the target audio. First, the speech signal of the target audio can be preprocessed, such as by removing noise and filtering. Then, features are extracted from the processed speech signal, such as short-time energy, zero-crossing rate, frequency spectrum, and Mel-frequency cepstral coefficients. Based on the changes in these features, the speech signal of the target audio is segmented, dividing the long speech signal into several short speech segments. Classifiers or thresholding methods are then used to determine the active or inactive state of the segmented speech segments. Taking everyday human communication as an example, an active state can refer to human speech or other sounds, while an inactive state can refer to silence or background noise. Finally, brief inactive states are removed, and the detection threshold is adjusted to improve the accuracy and robustness of the detection, thus obtaining the active speech segments in the target audio. After obtaining the active speech segments in the target audio, target speech segments are selected as needed, and corresponding spectrograms are subsequently generated based on these target speech segments.

[0069] By using the above method, the speech signal of the target speech segment can be obtained and processed by framing. This allows for frame extraction of the speech signal, thereby minimizing the computational load and increasing the computational speed. Subsequently, the spectrogram corresponding to the target speech segment can be generated by calculating the extracted speech signal.

[0070] 102. Perform frequency domain conversion on multiple frames of target speech signals to obtain multiple frames of frequency domain signals, and generate the target amplitude spectrum corresponding to each frame of target speech signal based on the amplitude values ​​of the multiple frames of frequency domain signals.

[0071] In this embodiment of the application, in order to more intuitively demonstrate the influence of different frequency components on speech and more easily observe the differences and similarities between different speech signals, frequency domain conversion can be performed on multiple frames of target speech signals to obtain multiple frames of frequency domain signals. Based on the amplitude values ​​of the multiple frames of frequency domain signals, the target amplitude spectrum corresponding to each frame of target speech signal is generated, thereby making it more convenient and effective to analyze and process speech signals and providing accurate parameter information for generating spectrograms.

[0072] The frequency domain signal can be the energy distribution of the target speech segment at different frequencies. In a speech signal, sound waves of different frequencies appear with different intensities, and these changes in intensity affect the pitch, timbre, and other characteristics of the speech.

[0073] Specifically, after extracting multiple frames of target speech signals, these signals are converted into multiple digital signals to allow for more refined processing. Windowing is then applied to the signals to prevent oscillations. Fourier transform is used to convert the target speech signals from the time domain to the frequency domain, resulting in multiple frames of frequency domain signals. The amplitude value at each frequency point in the frequency domain signal is calculated to generate the amplitude spectrum corresponding to each frame of speech signal. It's worth noting that the frequency domain signal can be represented in complex form, i.e., a combination of amplitude and phase. The amplitude value of the frequency domain signal can be calculated by squaring the real and imaginary parts of the complex signal, adding the two parts together, and then taking the square root. The specific steps are as follows: Represent the frequency domain signal in complex form: F = A + jB, where A represents the real part, B represents the imaginary part, and j represents the imaginary unit; Calculate the square of the amplitude: |F|^2 = A^2 + B^2; Calculate the amplitude value: |F| = sqrt(A^2 + B^2).

[0074] In some implementations, high frequencies can be enhanced using a spectrum mean-reduction method, thereby solving the problem of high-frequency enhancement for speech with significant energy disparities between high and low frequencies. For example, step 102, "generating the target amplitude spectrum corresponding to each frame of the target speech signal based on the amplitude values ​​of multiple frames of frequency domain signals," may include:

[0075] (102.1) Calculate the average amplitude value of the multi-frame frequency domain signal based on the amplitude values ​​of the multi-frame frequency domain signal;

[0076] (102.2) Subtract the amplitude value of the frequency domain signal of each frame from the average amplitude value to obtain the target amplitude spectrum corresponding to the target speech signal of each frame.

[0077] The average amplitude value refers to the average amplitude of a frequency domain signal over a period of time. It is commonly used to describe the intensity of a signal or vibration. In speech signal processing, the average amplitude value is also known as the RMS (root mean square) value. Using the average amplitude value helps quantify the intensity of a signal and serves as an important parameter in speech signal processing.

[0078] The target spectrum can be the spectrum after removing the average amplitude value. The target spectrum not only eliminates the energy disparity between high and low frequencies, but also retains detailed information about the energy distribution at each frequency.

[0079] Specifically, after frequency domain transformation of the target speech signal to obtain its spectrum, a moving average filter is applied to the spectrum to calculate the average amplitude value within a certain range around each frequency point. This average amplitude value is then used as the benchmark for removing the average at that point. The amplitude of each point in the spectrum is subtracted from the corresponding average amplitude value to obtain the high-frequency enhanced target amplitude spectrum.

[0080] By performing frequency domain conversion on multiple frames of target speech signals, multiple frames of frequency domain signals can be obtained. Based on the amplitude values ​​of the multiple frames of frequency domain signals, the target amplitude spectrum corresponding to each frame of target speech signal can be generated. This makes it more convenient and effective to analyze and process speech signals, and provides accurate parameter information for generating spectrograms.

[0081] 103. Normalize the target amplitude spectrum corresponding to each frame of the target audio signal to obtain the normalized amplitude spectrum, and perform logarithmic operation on the normalized amplitude spectrum to obtain the logarithmic amplitude spectrum corresponding to each frame of the target audio signal.

[0082] In this embodiment of the application, since the energy distribution and dynamic range of different speech signals may vary greatly, in order to eliminate the influence of scale changes, the amplitude spectrum corresponding to each frame of target speech signal can be normalized to compress the dynamic range of the signal to a fixed range, so that the brightness or color of the subsequently generated spectrogram is more continuous and smooth, making the spectrogram more intuitive, easy to read and easy to interpret.

[0083] The normalized amplitude spectrum can be obtained by normalizing the amplitude values ​​in the target amplitude spectrum to values ​​between 0 and 1. The normalized amplitude spectrum can better represent the spectral characteristics of the target speech signal.

[0084] The logarithmic amplitude spectrum is obtained by taking the logarithm of the normalized amplitude spectrum. It can be used for the analysis and processing of speech signals. The logarithmic amplitude spectrum has a more balanced value range compared to the amplitude spectrum, and when generating a spectrogram, it can serve as the basis for parameters such as color level and contrast, helping to convert the spectral characteristics of the target speech signal into a two-dimensional image, thus more intuitively displaying the content and features of the audio signal.

[0085] Specifically, after calculating the target amplitude spectrum of the target speech signal, the target amplitude spectrum is linearly normalized, scaling all values ​​to the range of 0-1. This normalization allows for easier comparison of the spectra of different speech signals with varying spectral distributions, eliminating the effects of scaling. The normalized amplitude spectrum is then converted into a logarithmic amplitude spectrum. The specific formula for calculating the normalized logarithmic amplitude spectrum is as follows:

[0086]

[0087] The complex frequency domain amplitude value of each frequency point in the spectrum corresponding to the target speech signal is a+bi, where a and b are the real and imaginary parts of the complex number, respectively, and i is the imaginary unit. Here, the frame length frame_len is used as the denominator for normalization calculation.

[0088] In some implementations, the frame length of each frame can be used as the denominator for normalization calculation to normalize the target amplitude spectrum corresponding to the target audio signal of each frame. This achieves a unified measurement of the frequency energy of audio signals with different amplitudes, providing a unified standard for subsequent spectrogram color mapping. For example, step 103, "normalizing the target amplitude spectrum corresponding to the target audio signal of each frame to obtain the normalized amplitude spectrum," may include:

[0089] (103.1) Obtain the amplitude value of each frequency point in the target amplitude spectrum corresponding to each frame of target speech signal;

[0090] (103.2) Based on the ratio of the amplitude value of each frequency point to the corresponding frame length, create the normalized amplitude spectrum corresponding to each frame of the target speech signal.

[0091] Specifically, the amplitude value corresponding to each frequency point in the target amplitude spectrum of each frame of the target speech signal can be calculated. Since the frequency domain signal corresponding to a frequency point can be represented in complex form, that is, a combination of amplitude and phase, the amplitude value of the frequency point can be calculated by squaring the real and imaginary parts of the complex signal respectively, adding the two parts together, and finally taking the square root. Dividing the amplitude value corresponding to each frequency point by the frame length can normalize the target amplitude spectrum; thus, the influence of different frame lengths can be eliminated, making the amplitude values ​​of different frequency points comparable.

[0092] By normalizing the amplitude spectrum of each frame of the target speech signal, the dynamic range of the signal can be compressed into a fixed range, thereby eliminating the influence of different frame lengths and making the amplitude values ​​of different frequency points comparable. This results in a more continuous and smooth brightness or color of the subsequently generated spectrogram, making the spectrogram more intuitive, easy to read, and easy to interpret.

[0093] 104. Generate the target spectrogram of the target speech segment based on the logarithmic amplitude spectrum corresponding to each frame of the target speech signal.

[0094] In this embodiment of the application, in order to generate the spectrogram of the target speech segment, the logarithmic amplitude spectrum corresponding to each frame of the target speech signal can be generated to obtain the target spectrogram of the entire speech segment.

[0095] A target spectrogram is an image that visualizes the spectrum of a target speech signal over time. It uses time as the horizontal axis and frequency as the vertical axis, with frequency and time variations represented by color brightness. Target spectrograms help analysts identify different frequency components in a sound and their temporal variations, and can be used in signal processing, speech recognition, music analysis, and other fields.

[0096] Specifically, after obtaining the logarithmic amplitude spectrum corresponding to each frame of the target speech signal, the normalized logarithmic amplitude spectrum is visualized on the time and frequency axes. The time axis is usually horizontal, arranged according to the temporal sequence of the audio signal. The frequency axis is usually vertical, representing the frequency distribution of the audio signal. Optional spectral ranges include narrowband, broadband, and ultra-wideband. Further parameter adjustments and enhancements can be made as needed, such as enhancing contrast, setting axis scales, adding color bars and labels, etc., ultimately generating a spectrogram corresponding to each frame of the target speech signal. This provides a detailed description of the speech signal's spectrum and can be used for tasks such as recognition, transcription, and classification in speech processing.

[0097] It's worth noting that adaptive parameter configuration can be set to increase the operability of the spectrogram. The parameters for automatically calculating the spectrogram can be divided into global adaptive parameters and local adaptive parameters, both selectable by the user. Global parameters are calculated based on the raw data of the entire speech segment, while local parameters are calculated for the speech segments in the gain buffer. The adaptive parameters of the spectrogram and the spectrogram data, such as the average energy in the time domain and the average amplitude spectrum in the frequency domain, are calculated based on the speech segment data in the gain buffer. The energy dynamic range of the spectrogram is normalized based on the average energy, and then the spectrogram data is mean-reduced based on the average amplitude spectrum to enhance the high-frequency display effect of the spectrogram.

[0098] In some implementations, the maximum logarithmic amplitude value in the logarithmic amplitude spectrum corresponding to multiple frames of target speech signals can be obtained. Then, according to the maximum logarithmic amplitude value, the frequency amplitude in the logarithmic amplitude spectrum corresponding to each frame of target speech signal is shifted, so that speech with different amplitude ranges can use the same set of dynamic range parameter settings, and the generated spectrogram can achieve similar display effects. For example, step 104, "generating the target spectrogram of the target speech segment according to the logarithmic amplitude spectrum corresponding to each frame of target speech signal," may include:

[0099] (104.1) Obtain the maximum logarithmic amplitude value based on the logarithmic amplitude spectrum corresponding to each frame of the target speech signal;

[0100] (104.2) According to the maximum logarithmic amplitude value, the frequency amplitude in the logarithmic amplitude spectrum corresponding to each frame of the target speech signal is shifted to obtain the target logarithmic amplitude spectrum corresponding to each frame of the target speech signal;

[0101] (104.3) Convert the target logarithmic amplitude spectrum corresponding to each frame of target speech signal into the corresponding spectrogram;

[0102] (104.4) Multiple spectrograms are spliced ​​together according to temporal relationship to obtain the target spectrogram of the target speech segment.

[0103] Specifically, after obtaining the logarithmic amplitude spectrum of the speech signal, algorithms such as linear search and binary search can be used to find the maximum value in the logarithmic amplitude spectrum corresponding to all target speech signals. Then, using the maximum logarithmic amplitude value as the target shift, all frequency amplitudes in the logarithmic amplitude spectrum are shifted upwards by the target shift amount to obtain the target logarithmic amplitude spectrum corresponding to each frame of the target speech signal. For each time window, i.e., each frame of the speech signal, one column of its target logarithmic amplitude spectrum is taken and converted into a horizontal line segment in the spectrogram. The horizontal line segments of the spectrogram obtained from each time window are spliced ​​together according to the temporal relationship to obtain the complete spectrogram.

[0104] In some implementations, the spectrogram can be further processed, such as smoothing, enhancing contrast, and adding labels, to facilitate visualization or analysis. In the visualization of the spectrogram, different colors, line types, backgrounds, etc., can be used based on the amplitude spectrum's fluctuation range to enhance readability. For example, step (104.3), "generating the target spectrogram of the target speech segment based on the logarithmic amplitude spectrum corresponding to each frame of the target speech signal," can include:

[0105] (104.3.1) Convert the target logarithmic magnitude spectrum into a visual image;

[0106] (104.3.2) Based on the fluctuation range of the target speech signal in the corresponding target logarithmic amplitude spectrum of each frame, different colors are mapped in the visualization image to obtain the spectrogram of the target speech signal corresponding to each frame.

[0107] The visualization image is a graph representing the spectral energy distribution of an audio signal. The horizontal axis represents frequency, and the vertical axis represents amplitude. The amplitude values ​​are converted to decibels (dB) through logarithmic operations. The visualization image can more clearly show the spectral characteristics and energy distribution of the audio signal.

[0108] The fluctuation range refers to the difference between the lowest and highest frequencies of a speech signal within a certain period of time. It is usually expressed in Hz (Hertz).

[0109] Specifically, the process involves acquiring the target logarithmic amplitude spectrum for each frame of the target speech signal, determining the frequency range on the horizontal axis and the amplitude range on the vertical axis, and then determining the size and style of the visualization image. Mapping the frequency domain data from the logarithmic amplitude spectrum to the time domain can be achieved using a Short-time Fourier Transform (STFT). STFT segments the original audio signal for spectral analysis and maps the data to the time domain to obtain spectral data for different time periods. The spectral data obtained through STFT processing is then visualized, and the target speech signal is mapped to different colors in the image based on the frequency fluctuation range in the target logarithmic amplitude spectrum. It's worth noting that the spectral data of each frame of the target speech signal corresponds to a row of pixel values ​​in the image; the larger the frequency range, the richer the colors in the image.

[0110] Therefore, in some implementations, during the playback of audio, it is necessary to dynamically display the spectrogram of the current audio segment in real time. The method for generating this spectrogram can also, after the target audio segment is updated, perform frame extraction on the updated audio signal of the target audio segment and compare the extracted audio signal with the audio signal from the previous moment. Only the spectrograms of the different audio signals need to be calculated, thereby reducing the computational load. For example, after step 104, "Generate the target spectrogram of the target audio segment based on the logarithmic amplitude spectrum corresponding to each frame of the target audio signal," it can also include:

[0111] (104.b.1) If the target speech segment is updated, the speech signal of the updated target speech segment is processed by framing to obtain multiple frames of speech signals to be confirmed, and multiple candidate speech signals are extracted discontinuously from the multiple frames of speech signals to be confirmed.

[0112] (104.b.2) Based on the target speech signal, the candidate speech signal is obtained by deduplication of the candidate speech signal from the multiple frames of the target speech signal;

[0113] (104.b.3) Calculate the candidate spectrogram corresponding to the target candidate speech signal based on the target candidate speech signal;

[0114] (104.b.4) Update the target spectrogram using the candidate spectrogram to obtain the updated target spectrogram of the target speech segment.

[0115] The candidate speech signal can be a single frame of speech signal extracted from all speech signal frames corresponding to the updated target speech segment.

[0116] Specifically, when the target speech segment is updated, the speech signal in the updated target speech time domain diagram is framed to obtain multiple frames of speech signals to be confirmed. Then, multiple frames of target speech signals are extracted discontinuously from these multiple frames of speech signals to be confirmed, i.e., multiple frames of candidate speech signals are extracted discontinuously. Since the frame rate of the image used for dynamic spectrogram playback is high, there is a significant amount of data overlap between adjacent frames. Therefore, an incremental update method can be used to reduce the computational load. Specifically, the frame extraction information of adjacent images is compared. Overlapping speech frames are not recalculated; instead, the spectrum of the corresponding speech signal frame in the previous frame is shifted in time. Only newly added speech signal frames are calculated. In other words, the candidate speech signals are deduplicated based on the multiple frames of target speech signals to obtain the target candidate speech signal. Repeat steps 301 to 302 of the target candidate speech signal to generate the corresponding candidate spectrogram. Finally, update the target spectrogram according to the temporal relationship to obtain the target spectrogram corresponding to the updated target speech segment. This realizes the dynamic incremental update of the spectrogram and improves the relational efficiency of the spectrogram.

[0117] By using the above method, the target spectrogram of the entire speech segment can be obtained by generating the logarithmic amplitude spectrum corresponding to each frame of the target speech signal, thus ensuring the quality of the target spectrogram.

[0118] By implementing any one or a combination of implementation methods in the embodiments of this application, the application scenarios of the spectrogram generation process can be realized.

[0119] As can be seen from the above, the embodiments of this application can acquire the speech signal of the target speech segment, perform frame-segmentation processing on the speech signal to obtain multiple frames of speech signal, and non-continuously extract multiple frames of target speech signal from the multiple frames of speech signal; perform frequency domain transformation on the multiple frames of target speech signal to obtain multiple frames of frequency domain signal, and generate the target amplitude spectrum corresponding to each frame of target speech signal based on the amplitude values ​​of the multiple frames of frequency domain signal; perform normalization processing on the target amplitude spectrum corresponding to each frame of target audio signal to obtain a normalized amplitude spectrum, and perform logarithmic operation on the normalized amplitude spectrum to obtain the logarithmic amplitude spectrum corresponding to each frame of target audio signal; and generate the target spectrogram of the target speech segment based on the logarithmic amplitude spectrum corresponding to each frame of target speech signal. Therefore, this scheme first performs non-continuous frame extraction on the speech signal after the target speech segment is processed into frames. The extracted speech signal is then frequency-domain transformed to obtain the corresponding frequency domain signal. Then, the spectrogram pixel set of each extracted speech signal is calculated. The spectrogram pixel sets of multiple speech signals are then spliced ​​together in chronological order to form the target spectrogram of the target speech segment. In this way, the computational load of the spectrogram is minimized. At the same time, by adjusting the speech signal, the quality of the spectrogram is ensured, reducing the complexity and difficulty of spectrogram generation and improving the efficiency of spectrogram generation.

[0120] Based on the method described in the above embodiments, the following examples will provide further detailed explanations.

[0121] This application uses a spectrogram generation device as an example to further describe the spectrogram generation method provided in this application. Among them, Figure 3 This is a schematic diagram of another step in the spectrogram generation method provided in this application embodiment. For ease of understanding, this application embodiment combines... Figure 3 Describe it.

[0122] In this embodiment, the description will focus on a spectrogram generation apparatus, which can be integrated into a computer device. When the processor on the computer device executes the program instructions corresponding to the data transmission method, the specific flow of the spectrogram generation method is as follows:

[0123] 201. The server acquires the audio signal of the target audio segment, performs frame-segmentation on the audio signal, and obtains multiple frames of audio signal.

[0124] The target speech segment can be the speech data that the user is currently obtaining from a microphone, audio file or other source, or it can be a segment of speech in an audio file or audio stream.

[0125] The speech signal can be an analog signal of the target speech segment, which is a continuous and smooth waveform. Its horizontal axis is the time axis and the vertical axis represents the intensity of the sound.

[0126] Specifically, users can select an audio file or audio stream, which is then converted into a digital signal. The server can use algorithms such as FFT (Fast Fourier Transform) to perform time-domain analysis on the digital signal, obtaining the corresponding speech signal. The speech signal result is then visualized as a time-domain graph. A time-domain graph is typically a waveform diagram on a time axis, with the horizontal axis representing time and the vertical axis representing the amplitude or energy of the speech signal. By locating the position of the preceding speech segment in this time-domain graph, the speech signal corresponding to the target speech segment can be obtained. Since speech signals are non-stationary, directly performing a Fourier transform on the speech signal is meaningless, as the frequency contour of the signal will be lost over time. Since speech signals are short-time stationary, after obtaining the speech signal corresponding to the target speech segment, it is necessary to divide the speech signal into short time frames.

[0127] 202. The server determines the target number of frames to be extracted based on the preset image resolution, and extracts the target number of target audio signals non-continuously from the multi-frame audio signals.

[0128] The target speech signal can be a single frame of speech signal extracted from all speech signal frames corresponding to the target speech segment.

[0129] The preset image resolution can be the image pixels corresponding to the spectrogram, and can be related to hardware devices such as image displays.

[0130] The target number of frames to be extracted can be the number of signal frames that need to be extracted from a segment of speech signal frames.

[0131] Specifically, after the server processes the target speech segment's speech signal into frames, it calculates the sampling rate of each pixel in the image based on the preset spectrogram image resolution (number of pixels) and the image's width and height. Then, it obtains the speech signal's sampling rate, divides the signal's sampling rate by the sampling rate of each pixel, and calculates the signal's frame rate. Finally, it determines the target number of frames to be extracted based on the desired number of frames and the signal's frame rate. Based on the target number of frames to be extracted, it performs non-continuous frame extraction on the segmented speech signal to obtain the corresponding multi-frame speech signal.

[0132] For example, if the spectrogram image resolution is 1920x1080 and the signal sampling rate is 44100Hz, and the signal needs to be framed at 30 frames per second, then the sampling rate per pixel can be calculated first: 44100 / (1920x1080) ≈ 0.024Hz. The calculated frame rate is 44100 / 0.024 ≈ 1837500 frames per second. Framed at 30 frames per second, the target number of frames is 1837500 / 30 ≈ 61250 frames. Therefore, in this case, the speech signal needs to be emulated 61250 frames, with a sampling rate of 0.024Hz per pixel.

[0133] 203. The server performs frequency domain conversion on multiple frames of target speech signals to obtain multiple frames of frequency domain signals.

[0134] The frequency domain signal can be the energy distribution of the target speech segment at different frequencies. In a speech signal, sound waves of different frequencies appear with different intensities, and these changes in intensity affect the pitch, timbre, and other characteristics of the speech.

[0135] Specifically, after extracting multiple frames of target speech signals, the server converts these signals into multiple digital signals, allowing for more refined processing. Then, the signals are windowed to avoid issues such as signal oscillation. Finally, the target speech signals are converted from the time domain to the frequency domain using Fourier transform to obtain multiple frames of frequency domain signals.

[0136] 204. The server subtracts the amplitude value of the frequency domain signal of each frame from the average amplitude value of the frequency domain signals of multiple frames to obtain the target amplitude spectrum corresponding to each frame of target speech signal.

[0137] The average amplitude value refers to the average amplitude of a frequency domain signal over a period of time. It is commonly used to describe the intensity of a signal or vibration. In speech signal processing, the average amplitude value is also known as the RMS (root mean square) value. Using the average amplitude value helps quantify the intensity of a signal and serves as an important parameter in speech signal processing.

[0138] The target spectrum can be the spectrum after removing the average amplitude value. The target spectrum not only eliminates the energy disparity between high and low frequencies, but also retains detailed information about the energy distribution at each frequency.

[0139] Specifically, the server performs frequency domain transformation on the target speech signal to obtain its spectrum. Then, it performs a moving average filter on the spectrum, calculating the average amplitude value within a certain range around each frequency point, and using this average amplitude value as the benchmark for removing the mean at that point. The amplitude of each point in the spectrum is then subtracted from the corresponding average amplitude value to obtain the high-frequency enhanced target amplitude spectrum.

[0140] 205. The server normalizes the target amplitude spectrum corresponding to each frame of the target audio signal to obtain the normalized amplitude spectrum, and performs logarithmic operation on the normalized amplitude spectrum to obtain the logarithmic amplitude spectrum corresponding to each frame of the target audio signal.

[0141] The normalized amplitude spectrum can be obtained by normalizing the amplitude values ​​in the target amplitude spectrum to values ​​between 0 and 1. The normalized amplitude spectrum can better represent the spectral characteristics of the target speech signal.

[0142] The logarithmic amplitude spectrum is obtained by taking the logarithm of the normalized amplitude spectrum. It can be used for the analysis and processing of speech signals. The logarithmic amplitude spectrum has a more balanced value range compared to the amplitude spectrum, and when generating a spectrogram, it can serve as the basis for parameters such as color level and contrast, helping to convert the spectral characteristics of the target speech signal into a two-dimensional image, thus more intuitively displaying the content and features of the audio signal.

[0143] Specifically, after calculating the target amplitude spectrum of the target speech signal, the target amplitude spectrum is linearly normalized, scaling all values ​​to the range of 0-1. This normalization allows for easier comparison of the spectra of different speech signals with varying spectral distributions, eliminating the effects of scaling. The normalized amplitude spectrum is then converted into a logarithmic amplitude spectrum. The specific formula for calculating the normalized logarithmic amplitude spectrum is as follows:

[0144]

[0145] The complex frequency domain amplitude value of each frequency point in the spectrum corresponding to the target speech signal is a+bi, where a and b are the real and imaginary parts of the complex number, respectively, and i is the imaginary unit. Here, the frame length frame_len is used as the denominator for normalization calculation.

[0146] 206. Based on the logarithmic amplitude spectrum corresponding to each frame of the target speech signal, the server obtains the maximum logarithmic amplitude value, and performs frequency amplitude shifting processing on the logarithmic amplitude spectrum corresponding to each frame of the target speech signal according to the maximum logarithmic amplitude value, to obtain the target logarithmic amplitude spectrum corresponding to each frame of the target speech signal.

[0147] Specifically, after obtaining the logarithmic amplitude spectrum of the speech signal, algorithms such as linear search and binary search can be used to find the maximum value to obtain the maximum logarithmic amplitude value in the logarithmic amplitude spectrum corresponding to all target speech signals. Then, using the maximum logarithmic amplitude value as the target shift amount, all frequency amplitudes in the logarithmic amplitude spectrum are shifted upward according to the target shift amount to obtain the target logarithmic amplitude spectrum corresponding to each frame of the target speech signal.

[0148] 207. The server maps different colors onto the visualization image corresponding to the target logarithmic amplitude spectrum based on the fluctuation range of each frame of the target speech signal in the corresponding target logarithmic amplitude spectrum, thus obtaining the spectrogram corresponding to each frame of the target speech signal.

[0149] The fluctuation range refers to the difference between the lowest and highest frequencies of a speech signal within a certain period of time. It is usually expressed in Hz (Hertz).

[0150] Specifically, the process involves acquiring the target logarithmic amplitude spectrum for each frame of the target speech signal, determining the frequency range on the horizontal axis and the amplitude range on the vertical axis, and then determining the size and style of the visualization image. Mapping the frequency domain data from the logarithmic amplitude spectrum to the time domain can be achieved using a Short-time Fourier Transform (STFT). STFT segments the original audio signal for spectral analysis and maps the data to the time domain to obtain spectral data for different time periods. The spectral data obtained through STFT processing is then visualized, and the target speech signal is mapped to different colors in the image based on the frequency fluctuation range in the target logarithmic amplitude spectrum. It's worth noting that the spectral data of each frame of the target speech signal corresponds to a row of pixel values ​​in the image; the larger the frequency range, the richer the colors in the image.

[0151] 208. The server splices multiple spectrograms according to their temporal relationship to obtain the target spectrogram of the target speech segment.

[0152] Specifically, the server determines the spectrogram corresponding to each time window based on the target logarithmic amplitude spectrum of each frame of target speech signal, and then splices the horizontal segments of the spectrogram obtained from each time window according to the temporal relationship to obtain the complete spectrogram.

[0153] Through the above application scenarios, the following effects can be achieved: After acquiring the speech signal of the target speech segment, the speech signal is processed by frame segmentation, rather than continuously extracting multiple frames of target speech signal; after frequency domain transformation of multiple frames of target speech signal, multiple frames of frequency domain signal are obtained, thereby generating the target amplitude spectrum corresponding to each frame of target speech signal, and then obtaining the normalized logarithmic amplitude spectrum of the target amplitude spectrum corresponding to each frame of target audio signal. Finally, based on the logarithmic amplitude spectrum corresponding to each frame of target speech signal, the target spectrogram of the target speech segment is generated; in this way, the complexity and difficulty of spectrogram generation are reduced, the efficiency of spectrogram generation is improved, and the quality of spectrogram is guaranteed.

[0154] Therefore, this scheme first performs non-continuous frame extraction on the speech signal after the target speech segment is processed into frames. The extracted speech signal is then frequency-domain transformed to obtain the corresponding frequency domain signal. Then, the spectrogram pixel set of each extracted speech signal is calculated. The spectrogram pixel sets of multiple speech signals are then spliced ​​together in chronological order to form the target spectrogram of the target speech segment. In this way, the computational load of the spectrogram is minimized. At the same time, by adjusting the speech signal, the quality of the spectrogram is ensured, reducing the complexity and difficulty of spectrogram generation and improving the efficiency of spectrogram generation.

[0155] To better implement the above methods, this application also provides a spectrogram generation device, which can be integrated into computer equipment, such as a voice terminal or other computer equipment.

[0156] For example, such as Figure 4 As shown, the spectrogram generation device may include an acquisition unit 301, a conversion unit 302, a calculation unit 303, and a generation unit 304.

[0157] The acquisition unit 301 is used to acquire the speech signal of the target speech segment, perform frame segmentation processing on the speech signal to obtain multiple frames of speech signal, and non-continuously extract multiple frames of target speech signal from the multiple frames of speech signal.

[0158] The conversion unit 302 is used to perform frequency domain conversion on multiple frames of target speech signals to obtain multiple frames of frequency domain signals, and generate the target amplitude spectrum corresponding to each frame of target speech signal based on the amplitude values ​​of the multiple frames of frequency domain signals.

[0159] The arithmetic unit 303 is used to normalize the target amplitude spectrum corresponding to each frame of target audio signal to obtain a normalized amplitude spectrum, and to perform logarithmic operation on the normalized amplitude spectrum to obtain the logarithmic amplitude spectrum corresponding to each frame of target audio signal.

[0160] The generation unit 304 is used to generate a target spectrogram of the target speech segment based on the logarithmic amplitude spectrum corresponding to each frame of the target speech signal.

[0161] In some embodiments, the spectrogram generation apparatus further includes a determining unit, configured to:

[0162] Determine the target number of frames based on the preset image resolution;

[0163] The acquisition unit 301 includes:

[0164] Non-continuous extraction of the target target speech signal from a multi-frame speech signal. 。

[0165] In some embodiments, the conversion unit 302 is further configured to:

[0166] Calculate the average amplitude value of the multi-frame frequency domain signal based on the amplitude values ​​of the multi-frame frequency domain signal.

[0167] Subtracting the amplitude value of each frame's frequency domain signal from the average amplitude value yields the target amplitude spectrum corresponding to each frame's target speech signal.

[0168] In some embodiments, the arithmetic unit 303 is further configured to:

[0169] Obtain the amplitude value of each frequency point in the target amplitude spectrum corresponding to each frame of the target speech signal;

[0170] Based on the ratio of the amplitude value at each frequency point to the corresponding frame length, a normalized amplitude spectrum is created for each frame of the target speech signal.

[0171] In some embodiments, the generating unit 304 is further configured to:

[0172] The maximum logarithmic amplitude value is obtained based on the logarithmic amplitude spectrum corresponding to each frame of the target speech signal.

[0173] Based on the maximum logarithmic amplitude value, the frequency amplitude in the logarithmic amplitude spectrum corresponding to each frame of the target speech signal is shifted to obtain the target logarithmic amplitude spectrum corresponding to each frame of the target speech signal.

[0174] Convert the target logarithmic amplitude spectrum corresponding to each frame of the target speech signal into the corresponding spectrogram;

[0175] Multiple spectrograms are spliced ​​together according to their temporal relationship to obtain the target spectrogram of the target speech segment.

[0176] In some embodiments, the generating unit 304 is further configured to:

[0177] Convert the target logarithmic magnitude spectrum into a visual image;

[0178] Based on the fluctuation range of each frame of the target speech signal in the corresponding target logarithmic amplitude spectrum, different colors are mapped in the visualization image to obtain the spectrogram corresponding to each frame of the target speech signal.

[0179] In some embodiments, the spectrogram generation apparatus further includes an updating unit, configured to:

[0180] If the target speech segment is updated, the speech signal of the updated target speech segment is processed into frames to obtain multiple frames of speech signal to be confirmed.

[0181] Based on multiple frames of target speech signals, deduplication is performed on multiple frames of speech signals to be confirmed to obtain candidate speech signals;

[0182] Calculate the candidate spectrogram corresponding to the candidate speech signal based on the candidate speech signal;

[0183] The target spectrogram is updated using the candidate spectrogram to obtain the updated target spectrogram of the target speech segment.

[0184] As shown above, this scheme first performs non-continuous frame extraction on the speech signal after the target speech segment is processed into frames. The extracted speech signal is then transformed in the frequency domain to obtain the corresponding frequency domain signal. Then, the spectrogram pixel set of each extracted speech signal is calculated. The spectrogram pixel sets of multiple speech signals are then spliced ​​together in chronological order to form the target spectrogram of the target speech segment. In this way, the computational load of the spectrogram is reduced to the minimum. At the same time, by adjusting the speech signal, the quality of the spectrogram is guaranteed, reducing the complexity and difficulty of spectrogram generation and improving the efficiency of spectrogram generation.

[0185] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0186] This application also provides a computer device, such as... Figure 5 As shown, it illustrates a structural schematic diagram of the computer device involved in the embodiments of this application, specifically:

[0187] The computer device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 5 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0188] The processor 401 is the control center of the computer device. It connects various parts of the computer device via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 402, and by calling data stored in the memory 402, thereby providing overall monitoring of the computer device. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0189] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and spectrogram generation by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as voice playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0190] The computer device also includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0191] The computer device may also include an input unit 404, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0192] Although not shown, the computer device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the computer device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions, as follows:

[0193] The process involves acquiring the speech signal of the target speech segment, performing frame segmentation to obtain multiple frames of speech signal, and then non-continuously extracting multiple frames of target speech signal from these frames. Frequency domain transformation is then performed on these multiple frames of target speech signal to obtain multiple frames of frequency domain signal. Based on the amplitude values ​​of these frequency domain signals, a target amplitude spectrum corresponding to each frame of target speech signal is generated. The target amplitude spectrum corresponding to each frame of target audio signal is then normalized to obtain a normalized amplitude spectrum. Logarithmic operations are then performed on the normalized amplitude spectrum to obtain a logarithmic amplitude spectrum corresponding to each frame of target audio signal. Finally, a target spectrogram of the target speech segment is generated based on the logarithmic amplitude spectrum corresponding to each frame of target speech signal.

[0194] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0195] As shown above, this scheme first performs non-continuous frame extraction on the speech signal after the target speech segment is processed into frames. The extracted speech signal is then transformed in the frequency domain to obtain the corresponding frequency domain signal. Then, the spectrogram pixel set of each extracted speech signal is calculated. The spectrogram pixel sets of multiple speech signals are then spliced ​​together in chronological order to form the target spectrogram of the target speech segment. In this way, the computational load of the spectrogram is reduced to the minimum. At the same time, by adjusting the speech signal, the quality of the spectrogram is guaranteed, reducing the complexity and difficulty of spectrogram generation and improving the efficiency of spectrogram generation.

[0196] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0197] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the spectrogram generation methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0198] The process involves acquiring the speech signal of the target speech segment, performing frame segmentation to obtain multiple frames of speech signal, and then non-continuously extracting multiple frames of target speech signal from these frames. Frequency domain transformation is then performed on these multiple frames of target speech signal to obtain multiple frames of frequency domain signal. Based on the amplitude values ​​of these frequency domain signals, a target amplitude spectrum corresponding to each frame of target speech signal is generated. The target amplitude spectrum corresponding to each frame of target audio signal is then normalized to obtain a normalized amplitude spectrum. Logarithmic operations are then performed on the normalized amplitude spectrum to obtain a logarithmic amplitude spectrum corresponding to each frame of target audio signal. Finally, a target spectrogram of the target speech segment is generated based on the logarithmic amplitude spectrum corresponding to each frame of target speech signal.

[0199] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0200] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0201] This application also provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the spectrogram generation method provided in the various optional implementations of the above embodiments.

[0202] Since the instructions stored in the computer-readable storage medium can execute the steps in any of the spectrogram generation methods provided in the embodiments of this application, the beneficial effects that any of the spectrogram generation methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0203] The above provides a detailed description of a spectrogram generation method, apparatus, device, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A spectrogram generation method, characterized in that, include: The process involves acquiring the speech signal of the target speech segment, performing frame segmentation on the speech signal to obtain multiple frames of speech signal, determining the sampling rate of each pixel in the spectrogram based on the preset image resolution, width, and height of the spectrogram, acquiring the sampling rate of the speech signal, dividing the sampling rate of the speech signal by the sampling rate of each pixel to calculate the frame rate of the signal, determining the target number of frame extractions based on the required number of frame extractions and the frame rate of the signal, and non-continuously extracting multiple frames of target speech signal corresponding to the target number of frame extractions from the multiple frames of speech signal. The target speech signal is a single frame of speech signal extracted from all speech signal frames corresponding to the target speech segment, and the target number of frame extractions is the number of signal frames extracted from the speech signal frames of a target speech segment. The multi-frame target speech signal is frequency domain converted to obtain multi-frame frequency domain signal, and the target amplitude spectrum corresponding to each frame target speech signal is generated according to the amplitude value of the multi-frame frequency domain signal. The target amplitude spectrum corresponding to each frame of target speech signal is normalized to obtain a normalized amplitude spectrum, and a logarithmic operation is performed on the normalized amplitude spectrum to obtain the logarithmic amplitude spectrum corresponding to each frame of target speech signal. Based on the logarithmic amplitude spectrum corresponding to each frame of the target speech signal, a target spectrogram of the target speech segment is generated.

2. The method according to claim 1, characterized in that, The step of generating the target amplitude spectrum corresponding to each frame of the target speech signal based on the amplitude values ​​of the multi-frame frequency domain signals includes: Calculate the average amplitude value of the multi-frame frequency domain signal based on the amplitude values ​​of the multi-frame frequency domain signal; The target amplitude spectrum corresponding to each frame of the target speech signal is obtained by subtracting the amplitude value of each frame of the frequency domain signal from the average amplitude value.

3. The method according to claim 1, characterized in that, The step of normalizing the target amplitude spectrum corresponding to each frame of the target speech signal to obtain the normalized amplitude spectrum includes: Obtain the amplitude value of each frequency point in the target amplitude spectrum corresponding to each frame of the target speech signal; Based on the ratio of the amplitude value of each frequency point to the corresponding frame length, a normalized amplitude spectrum corresponding to each frame of the target speech signal is created.

4. The method according to claim 1, characterized in that, The step of generating the target spectrogram of the target speech segment based on the logarithmic amplitude spectrum corresponding to each frame of the target speech signal includes: Based on the logarithmic amplitude spectrum corresponding to each frame of the target speech signal, the maximum logarithmic amplitude value is obtained; Based on the maximum logarithmic amplitude value, the frequency amplitude in the logarithmic amplitude spectrum corresponding to each frame of the target speech signal is shifted to obtain the target logarithmic amplitude spectrum corresponding to each frame of the target speech signal. Convert the target logarithmic amplitude spectrum corresponding to each frame of target speech signal into the corresponding spectrogram; The multiple spectrograms are spliced ​​together according to their temporal relationship to obtain the target spectrogram of the target speech segment.

5. The method according to claim 4, characterized in that, The step of converting the target logarithmic magnitude spectrum corresponding to each frame of target speech signal into a corresponding spectrogram includes: The target logarithmic magnitude spectrum is converted into a visual image; Based on the fluctuation range of each frame of the target speech signal in the corresponding target logarithmic amplitude spectrum, different colors are mapped onto the visualization image to obtain the spectrogram corresponding to each frame of the target speech signal.

6. The method according to claim 1, characterized in that, After generating the target spectrogram of the target speech segment based on the logarithmic amplitude spectrum corresponding to each frame of the target speech signal, the method further includes: If the target speech segment is updated, the speech signal of the updated target speech segment is processed by frame segmentation to obtain multiple frames of speech signal to be confirmed. Based on the multi-frame target speech signal, the multi-frame unconfirmed speech signal is deduplicated to obtain candidate speech signal; Calculate the candidate spectrogram corresponding to the candidate speech signal based on the candidate speech signal; The target spectrogram is updated using the candidate spectrogram to obtain the updated target spectrogram of the target speech segment.

7. A spectrogram generation apparatus, characterized in that, include: An acquisition unit is configured to acquire the speech signal of a target speech segment, perform frame segmentation on the speech signal to obtain multiple frames of speech signal, determine the sampling rate of each pixel in the spectrogram based on the preset image resolution, width, and height of the spectrogram, acquire the sampling rate of the speech signal, divide the sampling rate of the speech signal by the sampling rate of each pixel to calculate the frame rate of the signal, determine the target number of frame extractions based on the required number of frame extractions and the frame rate of the signal, and non-continuously extract multiple frames of target speech signal corresponding to the target number of frame extractions from the multiple frames of speech signal. The target speech signal is a single frame of speech signal extracted from all speech signal frames corresponding to the target speech segment, and the target number of frame extractions is the number of signal frames extracted from a segment of speech signal frames. The conversion unit is used to perform frequency domain conversion on the multi-frame target speech signal to obtain multi-frame frequency domain signal, and generate the target amplitude spectrum corresponding to each frame target speech signal according to the amplitude value of the multi-frame frequency domain signal. The arithmetic unit is used to normalize the target amplitude spectrum corresponding to each frame of target speech signal to obtain a normalized amplitude spectrum, and to perform logarithmic operation on the normalized amplitude spectrum to obtain the logarithmic amplitude spectrum corresponding to each frame of target speech signal. The generation unit is used to generate a target spectrogram of the target speech segment based on the logarithmic amplitude spectrum corresponding to each frame of the target speech signal.

8. A computer device, characterized in that, It includes a processor and a memory, the memory storing a computer program, and the processor running the computer program in the memory to implement the steps in the spectrogram generation method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is computer-readable and stores a plurality of instructions adapted for loading by a processor to perform the steps of the spectrogram generation method according to any one of claims 1 to 6.

Images