Speech processing method, processing apparatus, electronic device, and storage medium
By automatically generating and aligning spectrograms of speech data, the inefficiency caused by manual alignment in speech identity verification is solved, achieving efficient speech identity verification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-10
- Publication Date
- 2026-03-31
AI Technical Summary
In existing technologies, the process of identifying speech identity requires manual alignment of spectrograms of different speech data, resulting in low identification efficiency.
By automatically generating spectrograms for the first and second segments and aligning them to generate the target spectrogram, manual alignment is eliminated, improving identification efficiency.
It achieves automated alignment for voice identity verification, improving verification efficiency and accuracy.
Smart Images
Figure CN115547339B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of audio processing technology, and more specifically, to a speech processing method, processing device, electronic device, and storage medium. Background Technology
[0002] Voice identification, a type of biometric technology, is also known as speech identity verification. During the identification process, when it's necessary to compare different speech data, such as to confirm whether different speech data belong to the same speaker, the similarity of the spectrograms corresponding to the different speech data can be compared.
[0003] Currently, when comparing the similarity of spectrograms corresponding to different speech data, it is necessary to manually align the spectrograms corresponding to different speech data. This process is quite cumbersome, resulting in low efficiency in speech identity identification. Summary of the Invention
[0004] In view of the above problems, this application proposes a speech processing method, processing device, electronic device and storage medium to overcome or at least partially solve the problems of the prior art.
[0005] In a first aspect, embodiments of this application provide a speech processing method, comprising: acquiring a first segment corresponding to a comparison speech and a second segment corresponding to a target speech; generating a first spectrogram corresponding to the first segment; generating a second spectrogram corresponding to the second segment; aligning the first spectrogram and the second spectrogram to generate a target spectrogram, wherein the target spectrogram includes an aligned spectrogram corresponding to the first spectrogram and the second spectrogram, and the time position of the aligned spectrogram is aligned with the time position of the first spectrogram.
[0006] Secondly, embodiments of this application provide a speech processing apparatus, comprising: an acquisition module for acquiring a first segment corresponding to a comparison speech and a second segment corresponding to a target speech; a first generation module for generating a first spectrogram corresponding to the first segment; a second generation module for generating a second spectrogram corresponding to the second segment; and a third generation module for aligning the first and second spectrograms to generate a target spectrogram, the target spectrogram including an aligned spectrogram corresponding to the first and second spectrograms, the boundary of the aligned spectrogram being aligned with the boundary of the first spectrogram.
[0007] Thirdly, embodiments of this application provide an electronic device including a memory; one or more processors coupled to the memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more application programs are configured to perform the voice processing method as provided in the first aspect above.
[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing program code, which can be called by a processor to execute the voice processing method provided in the first aspect above.
[0009] The solution provided in this application obtains the first segment corresponding to the comparison speech and the second segment corresponding to the target speech, generates a first spectrogram corresponding to the first segment, generates a second spectrogram corresponding to the second segment, and aligns the first and second spectrograms to generate a target spectrogram. The target spectrogram includes an aligned spectrogram corresponding to the first and second spectrograms, with the time position of the aligned spectrogram aligned with the time position of the first spectrogram. This achieves automatic generation of a target spectrogram, including the first spectrogram corresponding to the first segment and the aligned spectrogram corresponding to the second segment, based on the first segment corresponding to the comparison speech and the second segment corresponding to the target speech. This eliminates the need for manual alignment of the first and second spectrograms, thus improving the efficiency of speech identity identification. Attached Figure Description
[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0011] Figure 1 A schematic flowchart of a speech processing method provided in an embodiment of this application is shown.
[0012] Figure 2 This paper illustrates another flowchart of the speech processing method provided in an embodiment of this application.
[0013] Figure 3 This illustration shows another flowchart of the speech processing method provided in an embodiment of this application.
[0014] Figure 4 A structural block diagram of a speech processing apparatus provided in an embodiment of this application is shown.
[0015] Figure 5 A functional block diagram of an electronic device provided in an embodiment of this application is shown.
[0016] Figure 6 This application illustrates a computer-readable storage medium for storing or carrying program code of a speech processing method according to an embodiment of the present application. Detailed Implementation
[0017] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.
[0018] The following disclosure provides many different implementations or examples for carrying out different structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. Of course, these are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or reference letters may be repeated in different examples; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various implementations and / or arrangements discussed.
[0019] Voice identification, a type of biometric technology, is also known as speech identity verification. During the identification process, when it's necessary to compare different speech data, such as to confirm whether different speech data belong to the same speaker, the similarity of the spectrograms corresponding to the different speech data can be compared.
[0020] Currently, when comparing the similarity of spectrograms corresponding to different speech data, it is necessary to manually align the spectrograms corresponding to different speech data. This process is quite cumbersome, resulting in low efficiency in speech identity identification.
[0021] To address the aforementioned problems, the inventors, after extensive research, proposed the speech processing method, processing device, electronic device, and storage medium provided in the embodiments of this application. This method automatically generates a target spectrogram, including a first spectrogram corresponding to the first speech segment and an aligned spectrogram corresponding to the second speech segment, based on the first speech segment corresponding to the comparison speech segment and the second speech segment corresponding to the target speech segment. This eliminates the need for manual alignment of the first spectrogram corresponding to the first speech segment and the second spectrogram corresponding to the second speech segment, thereby improving the efficiency of speech identity verification.
[0022] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.
[0023] Please see Figure 1 The diagram illustrates a flowchart of a voice processing method provided in one embodiment of this application. In a specific embodiment, the voice processing method can be executed by an electronic device with processing capabilities, such as a desktop computer, laptop computer, or other terminal device. The voice processing method can also be interactively executed by a processing system including a server and a terminal. Figure 1As shown, the speech processing method may include steps S110 to S140.
[0024] Step S110: Obtain a first segment corresponding to the comparison speech and a second segment corresponding to the target speech.
[0025] In an embodiment of the present application, for two speech data that need to be identified for speech identity, the speech data with undetermined identity information is the comparison speech, and the speech data with clear identity information is the target speech.
[0026] By performing automatic speech recognition (ASR) on the comparison speech, the corresponding first segment is obtained, and by performing ASR on the target speech, the corresponding second segment is obtained.
[0027] Among them, the first segment may include a speech segment with start and end time information of a specific phoneme (for example, the initial and final consonants of Chinese, or the consonants and vowels of English, etc.) in the comparison speech, or a speech segment with start and end time information of a syllable (for example, a single Chinese character, or a single English syllable, etc.) in the comparison speech.
[0028] The second segment may include a speech segment with start and end time information of a specific phoneme (for example, the initial and final consonants of Chinese, or the consonants and vowels of English, etc.) in the target speech, or a speech segment with start and end time information of a syllable (for example, a single Chinese character, or a single English syllable, etc.) in the target speech.
[0029] In an embodiment of the present application, a phoneme is the smallest speech unit divided according to the natural attributes of speech, and is divided into two major categories: vowels and consonants. Phonemes are analyzed based on the pronunciation actions in a syllable. One action constitutes one phoneme. For example, the Chinese syllable "啊 (ā)" has only one phoneme "ā"; "爱 (ài)" has two phonemes, namely "à" and "i"; "代 (dài)" has three phonemes, namely "d", "à" and "i". Another example is that the English "a" has only one phoneme "a"; "an" has two phonemes, namely "a" and "n"; "red" has three phonemes, namely "r", "e" and "d".
[0030] It can be understood that due to the differences in language pronunciation, the pronunciation of the same letter is different. Therefore, the phoneme can be a phoneme in any language. For example, it can be a Chinese phoneme or an English phoneme, and no limitation is made here.
[0033] In this embodiment of the application, the obtained first phonological segment can be preprocessed, and the preprocessed first phonological segment can be configured with Matlab parameters to generate the corresponding first spectrogram.
[0034] The preprocessing process can include normalization, pre-emphasis, framing, and windowing. Normalizing the first segment reduces the differences between different speech segments, and pre-emphasis enhances the high-frequency components of the first segment, making the transition from low to high frequencies relatively flat. A first-order high-pass filter can be used to boost the high-frequency components. The normalized and pre-emphasized first segment is then framed and windowed. A window function of a certain length is multiplied by the first segment to obtain the windowed segment in each frame. Matlab parameters are configured for each windowed segment to generate the corresponding first spectrogram.
[0035] The window function can be a Hamming window, a Hanning window, or a rectangular window, etc.; the configured parameters can be at least one of the following: signal bandwidth parameters, dynamic range parameters, sampling range parameters, attenuation parameters, high-frequency boost parameters, and window type parameters.
[0036] A spectrogram is a graphical representation of the speech spectrum over time. The vertical axis represents frequency, and the horizontal axis represents time. The intensity of any given frequency component at a given moment is represented by the gray level or hue intensity of the corresponding point. A darker color indicates stronger speech energy at that point, and vice versa. Spectrograms can be divided into narrowband spectrograms and wideband spectrograms. Narrowband spectrograms clearly show the structure of harmonics and reflect the time-varying process of the fundamental frequency; wideband spectrograms clearly show the structure of formants and the spectral envelope, reflecting the rapid time-varying process of the spectrum, thus achieving higher time resolution.
[0037] In the embodiments of this application, the first spectrogram can be a broadband spectrogram. The first spectrogram describes the frequency domain features of each speech frame, such as frequency and speech energy, in chronological order. That is, the first spectrogram is time-related.
[0038] Step S130: Generate the second spectrogram corresponding to the second syllable segment.
[0039] In this embodiment, the acquired second phonological segment can be preprocessed, and the preprocessed second phonological segment can be configured with Matlab parameters to generate the corresponding second spectrogram. The process of preprocessing the second phonological segment and configuring the Matlab parameters is the same as the process of preprocessing the first phonological segment and configuring the Matlab parameters, and will not be described again here.
[0040] The second spectrogram can be a broadband spectrogram. The second spectrogram describes the frequency domain characteristics of each speech frame in chronological order, such as frequency and speech energy. That is, the second spectrogram is time-dependent.
[0041] Step S140: Align the first spectrogram and the second spectrogram to generate the target spectrogram.
[0042] In this embodiment, because different speech segments have different durations, the widths (i.e., time information) of the spectrograms for different speech segments differ at the same time resolution. The first and second spectrograms can be aligned to generate a target spectrogram. The target spectrogram can include the aligned spectrograms corresponding to the first and second spectrograms, with the time position of the aligned spectrograms aligned with the time position of the first spectrogram. This achieves automatic generation of a target spectrogram, including the first spectrogram corresponding to the first speech segment and the aligned spectrogram corresponding to the second speech segment, based on the first speech segment corresponding to the comparison speech and the second speech segment corresponding to the target speech. This eliminates the need for manual alignment of the first spectrogram corresponding to the first speech segment and the second spectrogram corresponding to the second speech segment, thus improving the efficiency of speech identity verification.
[0043] In some implementations, a second spectrogram can be aligned with a first spectrogram as a reference to obtain an aligned spectrogram. The first spectrogram and the aligned spectrogram are then aligned on the same canvas to generate the target spectrogram. Here, the canvas is a layer that displays the spectrogram.
[0044] As one implementation method, the ratio between the time position of the second spectrogram and the time position of the first spectrogram can be calculated with the first spectrogram as a reference to obtain the time position ratio. The second spectrogram can then be scaled according to the time position ratio to obtain the corresponding scaled spectrogram, which is the aligned spectrogram. The first spectrogram and the aligned spectrogram can be aligned in the same canvas to generate the target spectrogram.
[0045] Understandably, when the time position ratio is greater than 1, it means the time position of the second spectrogram is greater than that of the first spectrogram. Scaling down the second spectrogram according to the time position ratio yields the corresponding scaled-down spectrogram, which is the aligned spectrogram. When the time position ratio is equal to 1, it means the time position of the second spectrogram is equal to that of the first spectrogram. No scaling is needed for the second spectrogram; it is the aligned spectrogram. When the time position ratio is less than 1, it means the time position of the second spectrogram is less than that of the first spectrogram. Enlarging the second spectrogram according to the time position ratio yields the corresponding enlarged spectrogram, which is the aligned spectrogram.
[0046] As one implementation method, since the number of pixels in the spectrogram displayed in a specific image region is fixed, directly scaling the second spectrogram to obtain the aligned spectrogram will result in pixelation, thus affecting the image quality of the aligned spectrogram. Instead, the second spectrogram can be analyzed to obtain the corresponding horizontal pixels. These horizontal pixels correspond to the temporal positions of the second spectrogram. Following preset rules, the target Fourier transform spectrum corresponding to the horizontal pixels is calculated. The temporal position corresponding to the target Fourier transform spectrum is aligned with the temporal position of the first spectrogram. Based on the target Fourier transform spectrum, the corresponding aligned spectrogram is generated.
[0047] Among them, the preset rule can be the frame extraction calculation rule of the horizontal pixels. By performing frame extraction calculation on the horizontal pixels of the second spectrogram, the corresponding target Fourier transform spectrum is obtained. This can ensure that the horizontal pixels of the generated aligned spectrogram are always consistent with the horizontal pixels of the second spectrogram, thus avoiding mosaic in the generated aligned spectrogram and ensuring the image quality of the aligned spectrogram.
[0048] As an example, the vertical height of the second spectrogram is x pixels and the horizontal width is y pixels. According to the frame extraction calculation rule of the horizontal pixels, y frames are uniformly extracted from the audio segment data corresponding to the second spectrogram according to the frame length parameter. The N-point target Fourier transform spectrum of the speech signal of each frame is calculated to obtain a spectrogram energy data matrix of size y*(N / 2+1). The energy values in the spectrogram energy data matrix are mapped to the gray values of the image to obtain the mapped spectrogram, which is the aligned spectrogram.
[0049] The solution provided in this application obtains the first segment corresponding to the comparison speech and the second segment corresponding to the target speech, generates a first spectrogram corresponding to the first segment, generates a second spectrogram corresponding to the second segment, and aligns the first and second spectrograms to generate a target spectrogram. The target spectrogram includes an aligned spectrogram corresponding to the first and second spectrograms, with the time position of the aligned spectrogram aligned with the time position of the first spectrogram. This achieves automatic generation of a target spectrogram, including the first spectrogram corresponding to the first segment and the aligned spectrogram corresponding to the second segment, based on the first segment corresponding to the comparison speech and the second segment corresponding to the target speech. This eliminates the need for manual alignment of the first and second spectrograms, thus improving the efficiency of speech identity identification.
[0050] Please see Figure 2This illustrates a flowchart of a voice processing method provided in another embodiment of this application. In a specific embodiment, the voice processing method can be executed by an electronic device with processing capabilities, such as a desktop computer, laptop computer, or other terminal device. The voice processing method can also be interactively executed by a processing system including a server and a terminal. Figure 2 As shown, the speech processing method may include steps S210 to S260.
[0051] Step S210: Obtain the first segment corresponding to the comparison speech and the second segment corresponding to the target speech.
[0052] Step S220: Generate the first spectrogram corresponding to the first phonological segment.
[0053] In this embodiment, steps S210 and S220 can be referred to the corresponding steps in the foregoing embodiments, and will not be repeated here.
[0054] Step S230: Determine whether to adjust the time information corresponding to the segment data of the second segment.
[0055] In this embodiment, due to errors in the time information of the segments obtained by ASR of speech, the start and end times of the segments deviate from the image in the spectrogram. The segment data of the second segment can be analyzed to obtain the analysis results, and it can be determined whether to adjust the time information corresponding to the segment data of the second segment based on the analysis results. The segment data may include phoneme data and syllable data, etc.
[0056] In some implementations, the analysis result can be a time matching degree. The time information corresponding to the segment data of the second segment can be matched with preset time information to obtain the time matching degree. Based on the time matching degree, it can be determined whether to adjust the time information corresponding to the segment data of the second segment. The preset time information is the actual time information corresponding to the actual speech data of the second segment.
[0057] If the time matching degree is greater than or equal to the time matching degree threshold, then the time information corresponding to the segment data of the second segment is adjusted; if the time matching degree is less than the time matching degree threshold, then the time information corresponding to the segment data of the second segment is not adjusted.
[0058] Step S240: When the time information corresponding to the segment data of the second segment is determined, the time information corresponding to the segment data of the second segment is adjusted to obtain the adjusted segment.
[0059] In this embodiment, when the time information corresponding to the segment data of the second segment is determined, the time information corresponding to the segment data of the second segment can be adjusted to obtain the adjusted segment. The time information corresponding to the segment data of the adjusted segment matches the preset time information, thereby realizing the correction of the time information of the second segment and improving the accuracy of speech identity identification.
[0060] In some implementations, when the time information corresponding to the segment data of the second segment is determined, the second segment can be input into a pre-trained deep learning network model. The deep learning network model can be used to adjust the second segment to an adjusted segment whose time information corresponding to the segment data matches the preset time information, and receive the adjusted segment output by the deep learning network model.
[0061] The deep learning network model can be a Convolutional Neural Network (CNN) model, a Deep Belief Network (DBN) model, a Stacked Auto Encoder Network (SAE) model, a Recurrent Neural Network (RNN) model, a Deep Neural Network (DNN) model, a Long Short-Term Memory (LSTM) network model, or a Gated Recurring Unit (GRU) model, etc. The type of deep learning network model is not limited here, and can be set according to actual needs.
[0062] Step S250: Generate the spectrogram corresponding to the adjusted segment and use it as the second spectrogram corresponding to the second segment.
[0063] In this embodiment, the adjusted segment can be preprocessed, and the preprocessed adjusted segment can be configured with Matlab parameters to generate a spectrogram corresponding to the adjusted segment. This spectrogram can then be used as the second spectrogram corresponding to the second segment. The process of preprocessing the adjusted segment and configuring the Matlab parameters is the same as the process of preprocessing the first segment and configuring the Matlab parameters for the second segment, and will not be described again here.
[0064] Step S260: Align the first spectrogram and the second spectrogram to generate the target spectrogram.
[0065] In this embodiment, step S260 can be referred to the corresponding steps in the previous embodiments, and will not be repeated here.
[0066] The solution provided in this embodiment obtains the first segment corresponding to the comparison speech and the second segment corresponding to the target speech, generates a first spectrogram corresponding to the first segment, and determines whether to adjust the time information corresponding to the segment data of the second segment. When it is determined that the time information corresponding to the segment data of the second segment needs to be adjusted, the time information corresponding to the segment data of the second segment is adjusted to obtain the adjusted segment, and a spectrogram corresponding to the adjusted segment is generated and used as the second spectrogram corresponding to the second segment. The target spectrogram is generated by aligning the first spectrogram and the second spectrogram. This achieves automatic generation of a target spectrogram including the first spectrogram corresponding to the first segment and the aligned spectrogram corresponding to the second segment based on the first segment corresponding to the comparison speech and the second segment corresponding to the target speech. There is no need for manual alignment of the first spectrogram corresponding to the first segment and the second spectrogram corresponding to the second segment, which improves the identification efficiency of speech identity identification.
[0067] Furthermore, when determining the time information corresponding to the segment data of the second segment, the time information corresponding to the segment data of the second segment is adjusted to obtain the adjusted segment, thereby realizing the correction of the time information of the second segment and improving the accuracy of speech identity identification.
[0068] Please see Figure 3 This illustrates a flowchart of a voice processing method provided in another embodiment of this application. In a specific embodiment, the voice processing method can be executed by an electronic device with processing capabilities, such as a desktop computer, laptop computer, or other terminal device. The voice processing method can also be interactively executed by a processing system including a server and a terminal. Figure 3 As shown, the speech processing method may include steps S310 to S370.
[0069] Step S310: Obtain the first segment corresponding to the comparison speech and the second segment corresponding to the target speech.
[0070] Step S320: Generate the first spectrogram corresponding to the first phonological segment.
[0071] Step S330: Generate the second spectrogram corresponding to the second syllable segment.
[0072] Step S340: Align the first spectrogram and the second spectrogram to generate the target spectrogram.
[0073] In this embodiment, steps S310, S320, S330 and S340 can be referred to the corresponding steps in the previous embodiments, and will not be repeated here.
[0074] Step S350: Obtain the first low-frequency formant of the first spectrogram and the second low-frequency formant of the aligned spectrogram corresponding to the second spectrogram.
[0075] In this embodiment, since the consistency of pitch quality is clearly reflected in the spectrogram corresponding to the pitch segment, the trend of the low-frequency formant and the center frequency of the start and end times of the pitch segments with the same pitch quality are basically the same. Therefore, in the process of identifying the identity of speech, the two pitch segments used to identify the identity are generally two pitch segments with the same pitch quality.
[0076] The first low-frequency formants can be obtained by automatically calculating the low-frequency formants of each speech frame corresponding to the first spectrogram using a speech signal processing algorithm. Similarly, the second low-frequency formants can be obtained by automatically calculating the low-frequency formants of each speech frame corresponding to the aligned spectrogram using the same algorithm. The speech signal processing algorithm can be an autocorrelation algorithm, a cepstral algorithm, or a linear predictive coding (LPC) algorithm, etc.
[0077] Step S360: Extract the first target spectrogram corresponding to the first low-frequency formant on the first spectrogram, and the second target spectrogram corresponding to the second low-frequency formant on the aligned spectrogram.
[0078] In this embodiment, a first target spectrogram can be obtained by cropping the spectrogram corresponding to the first low-frequency formant from the first spectrogram, and a second target spectrogram can be obtained by cropping the spectrogram corresponding to the second low-frequency formant from the aligned spectrogram. The first target segment corresponding to the first target spectrogram and the second target segment corresponding to the second target spectrogram have matching tuning qualities.
[0079] Step S370: Determine the identity identification result between the compared speech and the target speech based on the spectrogram similarity between the first target spectrogram and the second target spectrogram.
[0080] In this embodiment, the spectrogram similarity between the first target spectrogram and the second target spectrogram can be obtained, and the identity identification result between the compared speech and the target speech can be determined based on the spectrogram similarity. When the spectrogram similarity is greater than or equal to the spectrogram similarity threshold, it is determined that the compared speech and the target speech are from the same person; when the spectrogram similarity is less than the spectrogram similarity threshold, it is determined that the compared speech and the target speech are not from the same person. This achieves automatic determination of the identity identification result between the compared speech and the target speech based on the first spectrogram of the compared speech and the aligned spectrogram of the target speech, eliminating the need for manual comparison and judgment of the first spectrogram of the compared speech and the aligned spectrogram of the target speech, thus improving the identification efficiency of speech identity.
[0081] In some implementations, the spectrogram similarity can be a first spectrogram similarity. The first spectrogram similarity between the first target spectrogram and the second target spectrogram can be calculated according to a first preset algorithm rule, and the identity verification result between the compared speech and the target speech can be determined based on the first spectrogram similarity.
[0082] If the first spectrogram similarity is greater than or equal to the spectrogram similarity threshold, then the compared speech and the target speech are determined to be from the same person; if the first spectrogram similarity is less than the spectrogram similarity threshold, then the compared speech and the target speech are determined to be from different people.
[0083] The first preset algorithm rule may include at least one of the following: histogram algorithm, grayscale distribution algorithm, image template matching algorithm, structural similarity algorithm, peak signal-to-noise ratio algorithm, perceptual hash algorithm, etc.
[0084] In some implementations, the spectrogram similarity can be a second spectrogram similarity. A first target spectrogram and a second target spectrogram can be input into a pre-trained spectrogram similarity detection model, and the second spectrogram similarity output by the model can be received. Based on the second spectrogram similarity, the identity verification result between the compared speech and the target speech can be determined.
[0085] If the similarity of the second spectrogram is greater than or equal to the spectrogram similarity threshold, then the compared speech and the target speech are determined to be from the same person; if the similarity of the second spectrogram is less than the spectrogram similarity threshold, then the compared speech and the target speech are determined to be from different people.
[0086] The spectrogram similarity detection model can be used to detect the similarity between the second spectrogram of the first target spectrogram and the second target spectrogram. The spectrogram similarity detection model can be a Convolutional Neural Network (CNN) model, a Deep Belief Network (DBN) model, a Stacked Auto Encoder Network (SAE) model, a Recurrent Neural Network (RNN) model, a Deep Neural Network (DNN) model, a Long Short-Term Memory (LSTM) network model, or a Gated Recurring Unit (GRU) model, etc. The type of spectrogram similarity detection model is not limited here; it can be set according to actual needs.
[0087] The solution provided in this embodiment obtains the first segment corresponding to the comparison speech and the second segment corresponding to the target speech, generates a first spectrogram corresponding to the first segment, generates a second spectrogram corresponding to the second segment, aligns the first and second spectrograms to generate a target spectrogram, obtains the first low-frequency formant of the first spectrogram and the second low-frequency formant of the aligned spectrogram corresponding to the second spectrogram, and extracts the first target spectrogram corresponding to the first low-frequency formant on the first spectrogram and the second target spectrogram corresponding to the second low-frequency formant on the second spectrogram, respectively. Based on the spectrogram similarity between the first and second target spectrograms, the identity identification result between the comparison speech and the target speech is determined. This achieves automatic generation of a target spectrogram including the first spectrogram corresponding to the first segment and the aligned spectrogram corresponding to the second segment, without the need for manual alignment of the first and second spectrograms, thus improving the efficiency of speech identity identification.
[0088] Furthermore, based on the spectrogram similarity between the first target spectrogram corresponding to the compared speech and the second target spectrogram corresponding to the target speech, the identity identification result between the compared speech and the target speech is determined. This achieves automatic determination of the identity identification result between the compared speech and the target speech based on the first spectrogram of the compared speech and the aligned spectrogram of the target speech, eliminating the need for manual comparison and judgment of the first spectrogram of the compared speech and the aligned spectrogram of the target speech, thus improving the identification efficiency of speech identity.
[0089] Please see Figure 4 It illustrates a voice processing apparatus 400 provided in one embodiment of this application, such as Figure 4 As shown, the voice processing device 400 may include an acquisition module 410, a first generation module 420, a second generation module 430, and a third generation module 440.
[0090] The acquisition module 410 can be used to acquire the first segment corresponding to the comparison speech and the second segment corresponding to the target speech; the first generation module 420 can be used to generate the first spectrogram corresponding to the first segment; the second generation module 430 can be used to generate the second spectrogram corresponding to the second segment; the third generation module 440 can be used to align the first spectrogram and the second spectrogram to generate the target spectrogram, which includes the aligned spectrogram corresponding to the first spectrogram and the second spectrogram, and the boundary of the aligned spectrogram is aligned with the boundary of the first spectrogram.
[0091] In some embodiments, the voice processing device 400 may further include a first determining module and a second determining module.
[0092] The first determining module can be used to determine whether to adjust the time information corresponding to the segment data of the second segment before the second generating module 430 generates the second spectrogram corresponding to the second segment. The segment data includes phoneme data and syllable data. The second determining module can be used to adjust the time information corresponding to the segment data of the second segment when it is determined to adjust the time information, thereby obtaining the adjusted segment. The time information corresponding to the segment data of the adjusted segment matches the preset time information.
[0093] In some implementations, the second generation module 430 may include the first generation unit.
[0094] The first generation unit can be used to generate the spectrogram corresponding to the adjusted segment, and serve as the second spectrogram corresponding to the second segment.
[0095] In some implementations, the first determining module may include a matching unit, a first determining unit, a second determining unit, and a third determining unit.
[0096] The matching unit can be used to match the time information corresponding to the segment data of the second segment with preset time information to obtain a time matching degree; the first determining unit can be used to determine whether to adjust the time information corresponding to the segment data of the second segment based on the time matching degree; the second determining unit can be used to determine to adjust the time information corresponding to the segment data of the second segment when the time matching degree is greater than or equal to the time matching degree threshold; the third determining unit can be used to determine not to adjust the time information corresponding to the segment data of the second segment when the time matching degree is less than the time matching threshold.
[0097] In some implementations, the second determining module may include an input unit and a receiving unit.
[0098] The input unit can be used to input the second pitch into a pre-trained deep learning network model when the time information corresponding to the pitch data of the second pitch is determined. The deep learning network model is used to adjust the second pitch so that the time information corresponding to the pitch data matches the preset time information. The receiving unit can be used to receive the adjusted pitch output by the deep learning network model.
[0099] In some implementations, the third generation module 440 may include an alignment unit and a second generation unit.
[0100] The alignment unit can be used to align the second spectrogram with the first spectrogram to obtain an aligned spectrogram; the second generation unit can be used to generate a target spectrogram based on the first spectrogram and the aligned spectrogram.
[0101] In some implementations, the alignment unit may include an acquisition subunit, a calculation subunit, and a generation subunit.
[0102] The acquisition subunit can be used to acquire the horizontal pixels corresponding to the second spectrogram, and the horizontal pixels correspond to the time positions of the second spectrogram; the calculation subunit can be used to calculate the target Fourier transform spectrum corresponding to the horizontal pixels according to preset rules, and the time positions corresponding to the target Fourier transform spectrum are aligned with the time positions of the first spectrogram; the generation subunit can be used to generate the corresponding aligned spectrogram based on the target Fourier transform spectrum.
[0103] In some implementations, the preset rule can be a frame-skipping calculation rule, and the calculation subunit can include a calculation subunit.
[0104] The computational subunit can be used to calculate the target Fourier transform spectrum corresponding to the horizontal pixel according to the frame extraction calculation rules, and obtain the corresponding spectrogram energy data matrix.
[0105] In some implementations, generating sub-units may include mapping sub-units.
[0106] The mapping sub-unit can be used to map the energy values in the spectrogram energy data matrix to the gray values of the image, thereby obtaining the mapped spectrogram, and then use the mapped spectrogram as the corresponding spectrogram.
[0107] The solution provided in this application obtains the first segment corresponding to the comparison speech and the second segment corresponding to the target speech, generates a first spectrogram corresponding to the first segment, generates a second spectrogram corresponding to the second segment, and aligns the first and second spectrograms to generate a target spectrogram. The target spectrogram includes an aligned spectrogram corresponding to the first and second spectrograms, with the time position of the aligned spectrogram aligned with the time position of the first spectrogram. This achieves automatic generation of a target spectrogram, including the first spectrogram corresponding to the first segment and the aligned spectrogram corresponding to the second segment, based on the first segment corresponding to the comparison speech and the second segment corresponding to the target speech. This eliminates the need for manual alignment of the first and second spectrograms, thus improving the efficiency of speech identity identification.
[0108] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For device embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to in the descriptions of the method embodiments. Any processing method described in the method embodiments can be implemented in the device embodiments through corresponding processing modules, and will not be elaborated upon further in the device embodiments.
[0109] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.
[0110] Please see Figure 5 The diagram illustrates a functional block diagram of an electronic device 500 provided in one embodiment of the present application. The electronic device 500 may include one or more of the following components: a memory 510, a processor 520, and one or more application programs. The one or more application programs may be stored in the memory 510 and configured to be executed by one or more processors 520. The one or more application programs are configured to perform the methods as described in the foregoing method embodiments.
[0111] The memory 510 may include random access memory (RAM) or read-only memory. The memory 510 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 510 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (e.g., acquiring a first segment, acquiring a second segment, generating a first spectrogram, generating a second spectrogram, aligning the first and second spectrograms, generating a target spectrogram, determining whether to adjust time information, determining to adjust time information, adjusting time information, acquiring an adjusted segment, generating a spectrogram corresponding to the adjusted segment, matching the time information of the second segment with preset time information, inputting the second segment to a pre-trained deep learning network model, receiving the adjusted segment, acquiring the aligned spectrogram, acquiring lateral pixels, calculating the target Fourier transform spectrum, acquiring the spectrogram energy data matrix, and mapping energy values in the spectrogram energy data matrix, etc.), and instructions for implementing the various method embodiments described below. The storage data area can also store data created by the electronic device 500 during use (such as comparison speech, first segment, target speech, second segment, first spectrogram, second spectrogram, target spectrogram, aligned spectrogram, time position, segment data, time information, phoneme data, syllable data, adjusted segments, preset time information, time matching degree, time matching degree threshold, pre-trained deep learning network model, aligned spectrogram, horizontal pixels, preset rules, target Fourier transform spectrum, frame extraction calculation rules, spectrogram energy data matrix, energy value, image, grayscale value, and mapped spectrogram), etc.
[0112] Processor 520 may include one or more processing cores. Processor 520 connects to various parts within the electronic device 500 using various interfaces and lines, and performs various functions and processes data of the electronic device 500 by running or executing instructions, programs, code sets, or instruction sets stored in memory 510, and by calling data stored in memory 510. Optionally, processor 520 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 520 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 520 and may be implemented separately using a communication chip.
[0113] Please refer to Figure 6 This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable storage medium 600 stores program code 610, which can be called by a processor to execute the methods described in the above method embodiments.
[0114] The computer-readable storage medium 600 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 600 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 600 has storage space for program code 610 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 610 may be compressed, for example, in a suitable form.
[0115] The solution provided in this application obtains the first segment corresponding to the comparison speech and the second segment corresponding to the target speech, generates a first spectrogram corresponding to the first segment, generates a second spectrogram corresponding to the second segment, and aligns the first and second spectrograms to generate a target spectrogram. The target spectrogram includes an aligned spectrogram corresponding to the first and second spectrograms, with the time position of the aligned spectrogram aligned with the time position of the first spectrogram. This achieves automatic generation of a target spectrogram, including the first spectrogram corresponding to the first segment and the aligned spectrogram corresponding to the second segment, based on the first segment corresponding to the comparison speech and the second segment corresponding to the target speech. This eliminates the need for manual alignment of the first and second spectrograms, thus improving the efficiency of speech identity identification.
[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A voice processing method, characterized by, The method comprises the following steps: obtaining a first phonetic segment corresponding to a comparison voice and a second phonetic segment corresponding to a target voice; generating a first spectrogram corresponding to the first phonetic segment; determining whether to adjust time information corresponding to phonetic data of the second phonetic segment, the phonetic data comprising phoneme data and syllable data; when it is determined to adjust the time information corresponding to the phonetic data of the second phonetic segment, adjusting the time information corresponding to the phonetic data of the second phonetic segment to obtain an adjusted phonetic segment, the time information corresponding to the phonetic data of the adjusted phonetic segment matching preset time information; generating a second spectrogram corresponding to the second phonetic segment: generating a spectrogram corresponding to the adjusted phonetic segment, and taking the spectrogram as the second spectrogram corresponding to the second phonetic segment; aligning the first spectrogram and the second spectrogram to generate a target spectrogram, the target spectrogram comprising an aligned spectrogram corresponding to the first spectrogram and the second spectrogram, the time position of the aligned spectrogram being aligned with the time position of the first spectrogram.
2. The voice processing method of claim 1, wherein, The method further comprises the following steps: matching the time information corresponding to the phonetic data of the second phonetic segment with preset time information to obtain a time matching degree; determining whether to adjust the time information corresponding to the phonetic data of the second phonetic segment according to the time matching degree; when the time matching degree is greater than or equal to a time matching degree threshold, it is determined to adjust the time information corresponding to the phonetic data of the second phonetic segment; when the time matching degree is less than the time matching degree threshold, it is determined not to adjust the time information corresponding to the phonetic data of the second phonetic segment.
3. The voice processing method of claim 1, wherein, The method further comprises the following steps: when it is determined to adjust the time information corresponding to the phonetic data of the second phonetic segment, inputting the second phonetic segment into a pre-trained deep learning network model, the deep learning network model being configured to adjust the second phonetic segment to an adjusted phonetic segment whose time information corresponding to phonetic data matches preset time information; receiving the adjusted phonetic segment output by the deep learning network model.
4. The voice processing method of claim 1, wherein, The method further comprises the following steps: aligning the second spectrogram with the first spectrogram to obtain the aligned spectrogram; generating the target spectrogram according to the first spectrogram and the aligned spectrogram.
5. The voice processing method of claim 4, wherein, The method further comprises the following steps: obtaining a horizontal pixel corresponding to the second spectrogram, the horizontal pixel corresponding to a time position of the second spectrogram; calculating a target Fourier transform frequency spectrum corresponding to the horizontal pixel according to a preset rule, the target Fourier transform frequency spectrum corresponding to a time position aligned with the time position of the first spectrogram; generating the aligned spectrogram corresponding to the target Fourier transform frequency spectrum.
6. The voice processing method of claim 5, wherein, The preset rule is a frame extraction calculation rule, and the method further comprises the following steps: According to the frame extraction calculation rule, the target Fourier transform spectrum corresponding to the transverse pixels is calculated to obtain a corresponding spectrogram energy data matrix; The generating of the corresponding alignment spectrogram according to the target Fourier transform spectrum comprises: The energy values in the spectrogram energy data matrix are mapped to the gray values of an image to obtain a mapped spectrogram, and the mapped spectrogram is taken as the corresponding spectrogram.
7. A speech processing device, characterized by Comprise: The acquisition module is configured to acquire a first audio segment corresponding to the comparison voice and a second audio segment corresponding to the target voice. The first generation module is configured to generate a first spectrogram corresponding to the first audio segment. The first determination module is configured to determine whether to adjust time information corresponding to audio segment data of the second audio segment, the audio segment data comprising phoneme data and syllable data. The second determination module is configured to, when it is determined to adjust the time information corresponding to the audio segment data of the second audio segment, adjust the time information corresponding to the audio segment data of the second audio segment to obtain an adjusted audio segment, the time information corresponding to the audio segment data of the adjusted audio segment matching preset time information. The second generation module is configured to generate a second spectrogram corresponding to the second audio segment, generate a spectrogram corresponding to the adjusted audio segment, and take the spectrogram as the second spectrogram corresponding to the second audio segment. The third generation module is configured to perform alignment according to the first spectrogram and the second spectrogram to generate a target spectrogram, the target spectrogram comprising an alignment spectrogram corresponding to the first spectrogram and the second spectrogram, and a boundary of the alignment spectrogram being aligned with a boundary of the first spectrogram.
8. An electronic device, comprising: Comprise: A memory; One or more processors coupled to the memory; One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more application programs are configured to execute the voice processing method according to any one of claims 1 to 6.
9. A computer readable storage medium, characterized in that, The computer readable storage medium stores program codes, and the program codes can be called and executed by the processor to execute the voice processing method according to any one of claims 1 to 6.
Citation Information
Patent Citations
Voice data comparison processing method and device, electronic equipment and storage medium
CN114038468A