Speech style recognition system based on large model

Abstract

The present application relates to the field of intelligent speech recognition, and specifically discloses a speech style recognition system based on a large model, which receives the speech input signal of the target user collected by a microphone, extracts the Mel spectrum therefrom, and uses the signal analysis and processing technology based on deep learning and large models to perform block processing and multi-scale semantic encoding on the extracted Mel spectrum graph of the speech input signal, so as to intelligently generate speech style recognition broadcast speech according to the dynamic feature aggregation representation between the local coding features of the Mel spectrum of each context obtained by encoding. In this way, the system can ensure that complex styles are fully characterized, and can better adapt to the significant differences in pronunciation, speech speed, pitch, etc. between different speakers, and improve generalization ability.

Description

Technical Field

[0001] The present application relates to the field of intelligent speech recognition, and more specifically, to a speech style recognition system based on a large model. Background Art

[0002] With the development of smart devices and human-computer interaction technology, users' demand for personalized services is growing. For example, in the fields of virtual assistants, intelligent customer service, psychological therapy, etc., it is particularly important for the system to dynamically adjust the conversation style and voice broadcast method according to the user's personal preferences. This not only improves the user experience, but also enhances the naturalness and friendliness of human-computer interaction.

[0003] Traditional manual feature extraction methods have significant limitations in speech style recognition. Specifically, these methods rely on pre-defined rules and feature sets, which may result in the inability to capture all subtle features, especially those nonlinear or implicit features that are crucial for style recognition, such as intonation changes, emotional color, and pronunciation habits, resulting in insufficient representation of complex styles. In addition, different speakers vary greatly in pronunciation, speaking speed, pitch, etc., and a single manual feature set is difficult to adapt to this diversity, limiting the system's generalization ability.

[0004] Therefore, a speech style recognition system based on a large model is desired, which can improve the ability to capture subtle features and enhance the adaptability and generalization ability of the system. Summary of the invention

[0005] In order to solve the above technical problems, this application is proposed.

[0006] According to one aspect of the present application, a speech style recognition system based on a large model is provided, which includes: a speech signal acquisition module, which is used to receive a speech input signal of a target user collected by a microphone; a speech input signal Mel spectrum extraction module, which is used to extract the Mel spectrum features of the speech input signal to obtain a speech input signal Mel spectrum graph; a Mel spectrum feature block encoding module, which is used to perform block processing and multi-scale semantic encoding on the speech input signal Mel spectrum graph to obtain a sequence of contextual Mel spectrum local encoding features; a Mel spectrum feature dynamic aggregation module, which is used to perform Mel spectrum local feature dynamic aggregation on the sequence of contextual Mel spectrum local encoding features. The mel spectrum feature dynamic aggregation module comprises: a context mel spectrum clustering unit, which is used to calculate the context mel spectrum pseudo-anchor aggregation center of the sequence of context mel spectrum local coding features; a context mel spectrum feature dynamic aggregation unit, which is used to perform feature significant aggregation encoding on the sequence of context mel spectrum local coding features and the context mel spectrum pseudo-anchor aggregation center to obtain the speech style spectral feature significant aggregation coding feature; a broadcast speech generation module, which is used to obtain the speech style recognition broadcast speech based on the speech style spectral feature significant aggregation coding feature.

[0007] Compared with the prior art, the present application provides a speech style recognition system based on a large model, which receives the speech input signal of the target user collected by a microphone, extracts the Mel spectrum from it, and uses signal analysis and processing technology based on deep learning and large models to perform block processing and multi-scale semantic encoding on the extracted Mel spectrum of the speech input signal, so as to intelligently generate speech style recognition broadcast speech based on the dynamic feature aggregation representation between the local encoding features of the Mel spectrum of each context obtained by encoding. In this way, the system can comprehensively capture the local detail features and global context information of the speech signal, ensure that the complex style is fully characterized, and can better adapt to the significant differences in pronunciation, speaking speed, pitch, etc. among different speakers, and improve generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0009] Figure 1 4 is a block diagram of a speech style recognition system based on a large model according to an embodiment of the present application.

[0010] Figure 2 Schematic diagram of data flow of a large model-based speech style recognition system according to an embodiment of the present application.

[0011] Figure 3 It is a block diagram of a Mel spectrum feature block encoding module in a large model-based speech style recognition system according to an embodiment of the present application.

[0012] Figure 4 It is a block diagram of a Mel-spectrogram feature dynamic aggregation module in a large model-based speech style recognition system according to an embodiment of the present application.

[0013] Figure 5 It is a block diagram of a contextual Mel-spectrogram clustering unit in a large model-based speech style recognition system according to an embodiment of the present application. DETAILED DESCRIPTION

[0014] The embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, but rather these embodiments are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0015] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0016] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0017] It should be noted that in this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where the application is located and with the authorization of the owner of the corresponding device.

[0018] With the development of smart devices and human-computer interaction technology, users' demand for personalized services is growing. For example, in the fields of virtual assistants, intelligent customer service, psychological therapy, etc., it is particularly important for the system to dynamically adjust the conversation style and voice broadcast method according to the user's personal preferences. This not only improves the user experience, but also enhances the naturalness and friendliness of human-computer interaction.

[0019] Traditional manual feature extraction methods have significant limitations in speech style recognition. Specifically, these methods rely on pre-defined rules and feature sets, which may result in the inability to capture all subtle features, especially those nonlinear or implicit features that are crucial for style recognition, such as intonation changes, emotional color, and pronunciation habits, resulting in insufficient representation of complex styles. In addition, different speakers vary greatly in pronunciation, speaking speed, pitch, etc., and a single manual feature set is difficult to adapt to this diversity, limiting the system's generalization ability.

[0020] Therefore, in response to the above technical problems, the technical concept of this application is to receive the target user's voice input signal collected by the microphone, and then extract the Mel spectrum from it, and use signal analysis and processing technology based on deep learning and large models to perform block processing and multi-scale semantic encoding on the extracted voice input signal Mel spectrum map, so as to intelligently generate voice style recognition broadcast voice based on the dynamic feature aggregation representation between the local encoding features of each context Mel spectrum obtained by encoding. In this way, the system can comprehensively capture the local detail features and global context information of the voice signal, ensure that the complex style is fully represented, and can better adapt to the significant differences in pronunciation, speaking speed, pitch, etc. between different speakers, and improve generalization ability.

[0021] Figure 1 4 is a block diagram of a speech style recognition system based on a large model according to an embodiment of the present application. Figure 2 FIG. 1 is a data flow diagram of a speech style recognition system based on a large model according to an embodiment of the present application. Figure 1 and Figure 2As shown, according to the embodiment of the present application, the large model-based speech style recognition system 100 includes: a speech signal acquisition module 110, which is used to receive the speech input signal of the target user collected by a microphone; a speech input signal Mel spectrum extraction module 120, which is used to extract the Mel spectrum features of the speech input signal to obtain a speech input signal Mel spectrum graph; a Mel spectrum feature block encoding module 130, which is used to perform block processing and multi-scale semantic encoding on the speech input signal Mel spectrum graph to obtain a sequence of contextual Mel spectrum local coding features; a Mel spectrum feature dynamic aggregation module 140, which is used to perform Mel spectrum local feature dynamic aggregation on the sequence of contextual Mel spectrum local coding features to obtain a speech style spectrum feature significantly aggregated coding feature; a broadcast voice generation module 150, which is used to obtain a speech style recognition broadcast voice based on the speech style spectrum feature significantly aggregated coding features.

[0022] In an embodiment of the present application, the voice signal acquisition module 110 is used to receive the voice input signal of the target user collected by the microphone. It should be understood that the voice input signal refers to the sound waveform emitted by the user captured by a microphone or other audio acquisition device. These sound waveforms may contain human speech, intonation, tone and other non-verbal sound elements (such as laughter, sighs, etc.). Therefore, in order to capture and understand all the information in the user's voice so as to provide the user with a more accurate service or experience, in the technical solution of the present application, the voice input signal of the target user collected by the microphone is received.

[0023] In smart devices, user interaction with the system is usually done through a microphone. When the user speaks, the sound is transmitted to the microphone as sound waves, and the sensor inside the microphone responds to these sound waves and converts them into electrical signals. This conversion process is completed through the diaphragm or capacitor element inside the microphone: the sound waves cause the diaphragm to vibrate, which in turn changes the resistance or capacitance value in the circuit, thereby generating a voltage signal that changes over time. This analog voltage signal directly reflects the changes in the original sound waveform, including information such as volume, frequency, and possible environmental noise.

[0024] In order to meet the requirements of digital processing, the analog signal must be converted into a discrete numerical representation, which is the process of digitization. Digitization is not just a simple signal format conversion, it also involves how to capture and save the key details of voice data in the most effective way to ensure the accuracy and efficiency of subsequent processing. This process begins with analog-to-digital conversion (ADC). The ADC device samples the continuous analog signal into a series of discrete points and assigns a corresponding value to each sample point, usually a 16-bit or 24-bit integer, depending on the audio format used. The sampling frequency determines how many samples are obtained per second. Common standards include 8kHz (telephone quality), 16kHz, 22.05kHz, 44.1kHz (CD quality) or 48kHz. A higher sampling rate can retain more high-frequency components, but it also increases the amount of data. For voice signals, a sampling rate of 16kHz can meet the needs in most cases because the main energy of human speech is concentrated in the lower frequency range. However, choosing a suitable sampling rate also requires comprehensive consideration of the specific requirements of the application scenario, such as whether it is necessary to capture higher-frequency sound components or to use a lower sampling rate to reduce the pressure of data transmission and storage. In addition to the choice of sampling rate, the number of quantization bits is also an important factor affecting the quality of digitization. The number of quantization bits determines the number of different amplitude levels that can be represented by each sample point. 16-bit quantization means that each sample point can have 65536 different values, which is enough to provide sufficient dynamic range to express the nuances of most speech signals. For professional-grade recording or high-fidelity music playback, 24-bit or even higher quantization accuracy may be used. However, for most speech recognition applications, 16-bit quantization is sufficient and can achieve a good balance between performance and resource consumption.

[0025] The sampled data stream then enters the pre-processing stage, where unwanted low-frequency and high-frequency noise are removed, volume levels are adjusted, and other necessary operations are performed to optimize audio quality.

[0026] Specifically, in order to focus on the human voice part, a bandpass filter is usually applied to remove components beyond the human voice frequency range. For example, the frequency band between 300Hz and 3400Hz contains most of the information of human voice, so this part of the signal can be enhanced by setting appropriate filter parameters while suppressing interference from other frequency bands. In addition, it is also possible to use a high-pass filter to remove low-frequency noise below a certain threshold, such as humming or wind, and a low-pass filter to cut high-frequency noise above the voice frequency range. Various background noises are inevitable in the actual environment, which can affect the accuracy of speech recognition. To this end, a variety of noise reduction algorithms, such as spectral subtraction, Wiener filtering, adaptive filtering, etc., can be used to weaken or eliminate unwanted noise components. Especially for non-stationary noise sources, such as traffic noise or crowd noise, modern noise reduction algorithms often combine machine learning models for more accurate noise estimation and suppression. The difference in distance and volume of different users speaking may lead to inconsistent received signal strength, which poses a challenge to speech recognition. Therefore, automatic gain control (AGC) is usually implemented on the audio signal to standardize the amplitude of the input signal so that it is within the ideal processing range. This not only avoids clipping distortion caused by excessive input signals, but also ensures that smaller sounds are not drowned out by background noise. It is very useful to find the locations where speech begins and ends in the speech signal (i.e., the "endpoints"). It can help the system determine which parts are real speech content and which are just silence or noise. Endpoint detection helps reduce the amount of data processed, improves efficiency, and is particularly important in real-time applications because it can quickly locate the user's voice input and respond to commands or queries in a timely manner. Therefore, these steps ensure that the final digital audio signal is both clean and suitable for further processing, providing a solid foundation for efficient and accurate speech style recognition.

[0027] In an embodiment of the present application, the voice input signal Mel spectrum extraction module 120 is used to extract the Mel spectrum features of the voice input signal to obtain the voice input signal Mel spectrum diagram. Accordingly, considering that the voice input signal reflects the voice styles of different people speaking, such as pronunciation habits, intonation patterns, etc., and these different voice styles reflect the strength of different frequency components, therefore, in order to be able to more accurately and clearly understand the frequency feature information in different voice styles, in the technical solution of the present application, the Mel spectrum features of the voice input signal are extracted to obtain the voice input signal Mel spectrum diagram. It is worth mentioning that the Mel spectrum feature is an acoustic feature extracted from the voice signal, which reflects the energy distribution of the sound at different frequencies, and is organized according to the way the human auditory system perceives the frequency, that is, the Mel scale. The Mel scale is a method of mapping frequency to a nonlinear scale, which is more in line with the sensitivity changes of the human ear to different frequencies. The changes in the low frequency band are more sensitive than the high frequency band, so the Mel scale is divided more finely in the low frequency area and gradually becomes coarser in the high frequency area. Therefore, the Mel spectrum feature can effectively capture the key attributes of the formant information, pitch, intonation, etc. in the voice input signal, which are important bases for identifying the content and style of the voice, so that the machine learning model can learn and use this information more effectively to generate more accurate broadcast voice. The obtained Mel spectrum is obtained by a series of processing steps on the voice signal, which reflects the energy distribution of the sound at different Mel frequencies.

[0028] In particular, in a specific embodiment of the present application, the extraction of the Mel spectrum features of the speech input signal can be achieved in the following ways: First, 1. Framing: The continuous speech input signal is divided into short time frames (usually 20-40 milliseconds per frame), and there is overlap between frames (such as 50% or 75%). The reason for framing is that the speech signal is non-stationary, but it can be regarded as quasi-stationary in a sufficiently short time. 2. Windowing: In order to reduce the discontinuity between adjacent frames, a window function (such as Hanning window, Hamming window, etc.) is applied to each frame to reduce spectrum leakage.

[0029] Next, FFT is applied to each frame of the windowed speech signal to convert it from the time domain to the frequency domain and obtain a complex spectrum. That is, the result of the FFT output is a series of complex values, representing the amplitude and phase information of different frequency components.

[0030] Then, the square of the amplitude of the FFT result is calculated (that is, the complex value at each frequency point is modulo and squared) to obtain the power spectrum to show the energy of each frequency component.

[0031] Then, a set of triangular filters designed according to the Mel scale is constructed. These filters cover the entire frequency band and are dense in the low-frequency area and sparse in the high-frequency area. Each filter performs a weighted summation of the spectrum values ​​within a specific frequency range to simulate the changes in the sensitivity of the human ear to different frequencies. The above power spectrum is passed through the Mel filter bank to calculate the output of each filter. This step maps the linear frequency axis to the Mel frequency axis to obtain the Mel spectrum.

[0032] Afterwards, the natural logarithm of the output of each Mel filter is taken to compress the dynamic range and better match the loudness perception characteristics of the human auditory system. The logarithmic Mel spectrum can more accurately reflect the human auditory system's perception of volume changes.

[0033] Finally, the Mel spectra of all frames after the above processing are arranged in chronological order to form a two-dimensional matrix, in which one dimension represents time and the other dimension represents the frequency after Mel scale conversion. This two-dimensional matrix is ​​the Mel spectrogram of the speech input signal.

[0034] In the embodiment of the present application, the Mel spectrum feature block encoding module 130 is used to perform block processing and multi-scale semantic encoding on the Mel spectrum graph of the speech input signal to obtain a sequence of contextual Mel spectrum local encoding features. Figure 3 The block diagram of the Mel spectrum feature block encoding module in the speech style recognition system based on the large model according to the embodiment of the present application. Specifically, in the embodiment of the present application, the Mel spectrum feature block encoding module 130 includes: a speech input signal block unit 131, which is used to block the Mel spectrum graph of the speech input signal to obtain a sequence of Mel spectrum graph blocks of the speech input signal; a Mel spectrum feature local encoding unit 132, which is used to perform Mel spectrum multi-scale semantic encoding on the sequence of Mel spectrum graph blocks of the speech input signal to obtain a sequence of contextual Mel spectrum local encoding feature vectors as the sequence of contextual Mel spectrum local encoding features.

[0035] Specifically, the speech input signal block unit 131 is used to block the speech input signal Mel spectrogram to obtain a sequence of speech input signal Mel spectrogram blocks. It should be understood that different local segments in the speech input signal Mel spectrogram contain spectrum feature information in different local time periods, such as the pronunciation characteristics of a word or syllable. If the entire continuous Mel spectrogram is directly processed, some important details may be ignored, especially those features that change rapidly over time. Based on this, in the technical solution of the present application, the speech input signal Mel spectrogram is block processed to divide the longer Mel spectrogram into shorter time periods (i.e., blocks) to obtain a sequence of speech input signal Mel spectrogram blocks. It can be understood that compared to processing the entire continuous Mel spectrogram, block processing allows the model to focus on only a small part of the data at a time, and each block represents the spectrum information within a period of time, which helps to capture the speech characteristics within a specific time period, thereby improving the ability to capture rapidly changing features in speech, such as intonation, stress, etc.

[0036] Specifically, the following is a specific process of performing block processing on the Mel-spectrogram of the speech input signal: The first step of the block processing is to determine the length of time covered by each block, which is the so-called "frame length". The choice of frame length directly affects the performance and efficiency of the system. If the frame length is too short, although more transient characteristics can be captured, it will also increase the computational burden and may cause the loss of important long-term dependencies; conversely, if the frame length is too long, it may not be able to fully capture rapidly changing details. Therefore, choosing an appropriate frame length requires a balance between capturing details and computing resources.

[0037] Typically, the frame length is set to a value between 20 and 40 milliseconds. This range is considered ideal because it is short enough to capture transient characteristics, but not too short to lose too much information. For example, most phonemes in human speech last about 30 to 50 milliseconds, so a frame length of 20 to 40 milliseconds can well match the speech characteristics on these time scales. In addition, such a frame length can also avoid generating too much data while maintaining a high resolution, thereby maintaining a reasonable computational cost.

[0038] In addition to the frame length, another key parameter is the frame shift, which is the time difference between two adjacent frames. The frame shift determines the degree of overlap between adjacent tiles and is generally set to half or three-quarters of the frame length. In this way, the amount of data can be reduced and the processing speed can be increased while ensuring sufficient information redundancy. The overlapping design helps to smooth the transition, making the connection between different frames more natural and avoiding distortion problems caused by boundary effects. Specifically, when the frame shift is set to half the frame length, it means that each new frame starts from the middle of the previous frame, which ensures that most of the content between adjacent frames is the same, but just slightly moved, thereby effectively reducing the impact of edge effects.

[0039] In order to further reduce the discontinuity and edge effects between adjacent frames, a window function is usually applied to each frame. The window function acts like a weighting factor, which gradually reduces the importance of samples close to the frame boundary, thereby making the transition between adjacent frames smoother. Commonly used window functions include Hanning window, Hamming window and Blackman window. These window functions have specific shapes. They give higher weights in the center and gradually reduce the weights near the edges, eventually approaching zero. Taking the Hanning window as an example, its mathematical expression is:

[0040] in represents the weight of the nth sample, and N is the total number of samples in the frame. It can be seen that the Hanning window assigns lower weights to the two ends of the frame and higher weights to the middle part. This design softens the edges of the frame and reduces the spectrum leakage caused by sudden truncation, that is, the energy originally concentrated on certain frequencies is spread to other frequencies. Spectral leakage will lead to inaccurate frequency domain representation, which will affect subsequent feature extraction and pattern recognition.

[0041] The process of applying the window function is performed before each block. First, a fixed-length time period is extracted from the original Mel-spectrogram as the current frame according to the selected frame length. Then, all sample points of the frame are multiplied by the corresponding window function value to obtain a weighted frame. Finally, all frames processed by the window function are arranged in order to form a sequence of multiple short-time sequence blocks, that is, a sequence of Mel-spectrogram blocks of the speech input signal. This not only improves the accuracy of the frequency domain representation, but also enhances the continuity between adjacent frames, making the representation of the entire Mel-spectrogram more coherent and smooth. Therefore, the block processing realizes the effective transformation of the original long-time sequence Mel-spectrogram by reasonably selecting the frame length and frame shift, and combining with an effective window function. This method not only simplifies the data structure, but also retains the key features of the speech signal, laying a solid foundation for subsequent advanced processing.

[0042] Specifically, the Mel spectrum feature local encoding unit 132 is used to perform Mel spectrum multi-scale semantic encoding on the sequence of Mel spectrum graph blocks of the speech input signal to obtain a sequence of context Mel spectrum local encoding feature vectors as the sequence of context Mel spectrum local encoding features. Specifically, in an embodiment of the present application, the Mel spectrum feature local encoding unit is used to: input the sequence of Mel spectrum graph blocks of the speech input signal into a Mel spectrum multi-scale semantic encoder including a convolution layer and a converter structure to obtain a sequence of context Mel spectrum local encoding feature vectors. Accordingly, in order to further capture and mine the local feature information in the sequence of Mel spectrum graph blocks of the speech input signal, such as information at the phoneme level, while also considering that there is a spectral correlation between the contexts of the Mel spectrum graph blocks of the speech input signal at different time distances, therefore, in the technical solution of the present application, the sequence of Mel spectrum graph blocks of the speech input signal is processed by a Mel spectrum multi-scale semantic encoder including a convolution layer and a converter structure to obtain a sequence of context Mel spectrum local encoding feature vectors. It is worth mentioning that the convolution layer slides on the input data and performs a dot multiplication operation through a series of filters or kernels to generate a feature map. These feature maps capture local patterns in the input data. The transformer structure allows the model to focus on all positions in the input sequence, not just adjacent positions, through the self-attention mechanism, so that long-range temporal dependencies can be captured. Therefore, the Mel-spectrogram multi-scale semantic encoder containing convolutional layers and transformer structures can extract rich time-frequency features in speech signals and capture long-range dependencies. Specifically, the convolutional layer can effectively extract local features from the Mel-spectrogram patches, and by applying convolution operations at different scales, the details of the speech signal can be captured from multiple levels. After the features extracted by the convolutional layer pass through the transformer's self-attention mechanism, they can combine the information between each image patch to build a richer contextual representation, so that meaningful features can be extracted from a low level, and the relationship between these features can be understood and utilized at a high level, thereby improving the understanding of continuous speech streams.

[0043] In an embodiment of the present application, the Mel spectrum feature dynamic aggregation module 140 is used to perform Mel spectrum local feature dynamic aggregation on the sequence of the context Mel spectrum local coding features to obtain a significant aggregation coding feature of the speech style spectrum feature. Furthermore, considering that the speech signal is essentially time-varying and contains rich dynamic information. For example, changes in intonation, transitions of phonemes, etc. are very important dynamic characteristics in speech, which are essential for understanding the speaker's intentions, emotional state and personal style. Therefore, in order to significantly improve the model's ability to understand the intrinsic structure and pattern of speech data, in the technical solution of the present application, the sequence of the context Mel spectrum local coding features is subjected to Mel spectrum local feature dynamic aggregation to obtain a significant aggregation coding feature of the speech style spectrum feature. In particular, dynamic aggregation evaluates and optimizes the distribution of feature vectors in high-dimensional space by simulating the dynamic principles of the physical world, so as to better identify and process complex nonlinear patterns, thereby enhancing its ability to understand complex patterns (such as variable intonations, different pronunciation habits). At the same time, by performing dynamic analysis on the feature vectors, these time-varying features and their relationships with each other can be captured and those feature vectors with significant dynamic characteristics can be screened out. That is, these features often carry key information about speech style, such as rhythm, tone changes, etc. By focusing on these significant features, the model can more accurately capture the uniqueness of each speaker, thereby generating a more personalized speech style recognition broadcast voice.

[0044] Figure 4 is a block diagram of a Mel spectrum feature dynamic aggregation module in a speech style recognition system based on a large model according to an embodiment of the present application. Specifically, Figure 4 As shown, the mel spectrum feature dynamic aggregation module 140 includes: a context mel spectrum clustering unit 141, which is used to calculate the context mel spectrum pseudo-anchor aggregation center of the sequence of context mel spectrum local coding features; a context mel spectrum feature dynamic aggregation unit 142, which is used to perform feature significant aggregation coding on the sequence of context mel spectrum local coding features and the context mel spectrum pseudo-anchor aggregation center to obtain the speech style spectrum feature significant aggregation coding feature.

[0045] Figure 5 is a block diagram of a contextual Mel spectrum clustering unit in a speech style recognition system based on a large model according to an embodiment of the present application. Specifically, Figure 5As shown, the context mel spectrum clustering unit 141 includes: a context mel spectrum feature static potential weight factor calculation unit 1411, which is used to calculate the static potential weight factor of each context mel spectrum local coding feature vector in the sequence of context mel spectrum local coding feature vectors to obtain a sequence of context mel spectrum local feature static potential weight factors; a mel spectrum feature weight weighting unit 1412, which is used to calculate the weighted sum of the sequence of context mel spectrum local coding feature vectors based on the sequence of context mel spectrum local feature static potential weight factors to obtain a context mel spectrum pseudo-anchor aggregation center representation vector as the context mel spectrum pseudo-anchor aggregation center.

[0046] Specifically, in an embodiment of the present application, the context Mel spectrum feature static potential energy weight factor calculation unit 1411 includes: a static potential energy measurement coefficient calculation secondary subunit, which is used to input each context Mel spectrum local encoding feature vector in the sequence of context Mel spectrum local encoding feature vectors into a feature static potential energy measurement network to obtain a sequence of context Mel spectrum local feature static potential energy measurement coefficients; a static potential energy measurement coefficient screening gating secondary subunit, which is used to input the sequence of context Mel spectrum local feature static potential energy measurement coefficients into a feature energy level screening gating unit to obtain a sequence of context Mel spectrum local feature static potential energy weight factors.

[0047] The above-mentioned static potential energy metric coefficient calculation process of the secondary subunit can be expressed by the following formula: ;in, is the sequence of the contextual Mel-spectrogram local encoding feature vectors, and are the first, second, and third in the sequence of the contextual Mel spectrum local coding feature vectors. and contextual Mel-spectrogram local encoding feature vector, It is The eigenvalues ​​of each position in the local encoding feature vector of the contextual Mel spectrum, It is The mean of the local encoding feature vectors of the contextual Mel-spectrogram, It is The standard deviation of the local encoding feature vector of the contextual Mel spectrum, It is The number of eigenvalues ​​in the local encoding feature vector of the contextual Mel spectrum, is the first in the sequence of static potential energy measurement coefficients of the local features of the context Mel spectrum The static potential energy measurement coefficient of the local feature of the context Mel spectrum.

[0048] More specifically, in an embodiment of the present application, the static potential energy measurement coefficient screening gated secondary sub-unit is used to: standardize each context Mel spectrum local feature static potential energy measurement coefficient in the sequence of context Mel spectrum local feature static potential energy measurement coefficients to obtain a sequence of standardized context Mel spectrum local feature static potential energy measurement coefficients; input the sequence of standardized context Mel spectrum local feature static potential energy measurement coefficients into a mask function for masking processing to obtain a sequence of context Mel spectrum local feature static potential energy weight factors.

[0049] The above static potential energy metric coefficient screening gated secondary subunit processing process can be expressed by the following formula: ;in, is the first in the sequence of static potential energy measurement coefficients of the local features of the context Mel spectrum The static potential energy measurement coefficient of the local feature of the context Mel spectrum, is the first in the sequence of the static potential energy measurement coefficients of the local features of the normalized context Mel spectrum. The static potential energy measurement coefficient of the local feature of the standardized context Mel spectrum, is the preset threshold, is a masking operation, is the first in the sequence of static potential weight factors of the local features of the contextual Mel spectrum The static potential weight factor of the local feature of the contextual Mel spectrum.

[0050] It should be understood that in order to evaluate the potential energy level of each contextual Mel spectrum local encoding feature vector in a static environment, so as to discover the intrinsic connection between the feature vectors and their correlation with the target result, and finally assign an index to each feature vector to reflect their importance and influence in the current data distribution, each contextual Mel spectrum local encoding feature vector is input into the feature static potential energy measurement network respectively to obtain the contextual Mel spectrum local feature static potential energy measurement coefficient of each vector to reflect the relative stability and importance of each feature in its spatial position.

[0051] Next, each static potential energy measurement coefficient is input into the feature energy level screening gating unit to adaptively screen out the most relevant feature information and obtain the static potential energy weight factor of each contextual Mel spectrum local feature. In this way, the model can respond more flexibly to the changes in the importance of different features, ensuring that those features that have a greater impact on the voice style receive higher weights.

[0052] Specifically, the context mel spectrum feature dynamic aggregation unit 142 is used to calculate the aggregation movement direction of each context mel spectrum local encoding feature vector in the sequence of the context mel spectrum local encoding feature vector relative to the context mel spectrum pseudo-anchor aggregation center representation vector to obtain a sequence of context mel spectrum aggregation running directions. This step can be expressed by the following formula: ;in, is the first in the sequence of the contextual Mel spectrum local encoding feature vector contextual Mel-spectrogram local encoding feature vector, is the first in the sequence of static potential weight factors of the local features of the contextual Mel spectrum The static potential weight factor of the local feature of the contextual Mel spectrum, is the number of vectors in the sequence of the contextual Mel-spectrogram local encoding feature vectors, is the contextual Mel spectrum pseudo-anchor aggregation center representation vector, is the length of the vector, is the inverse cosine function, is the sequence of the contextual Mel spectrum aggregation running direction The contextual Mel-spectrogram aggregation runs in the direction.

[0053] Based on the sequence of the context Mel spectrum aggregation running direction, the sequence of the context Mel spectrum local coding feature vectors is dynamically aggregated toward the context Mel spectrum pseudo-anchor aggregation center representation vector to obtain a speech style spectrum feature significant aggregation coding vector as the speech style spectrum feature significant aggregation coding feature. This step can be expressed by the following formula: ;in, is the first in the sequence of the contextual Mel spectrum local encoding feature vector contextual Mel-spectrogram local encoding feature vector, is the first in the sequence of static potential energy measurement coefficients of the local features of the context Mel spectrum The static potential energy measurement coefficient of the local feature of the context Mel spectrum, is the number of vectors in the sequence of the contextual Mel-spectrogram local encoding feature vectors, is the contextual Mel spectrum pseudo-anchor aggregation center representation vector, is the sequence of the contextual Mel spectrum aggregation running direction contextual Mel spectrum aggregation operation direction, is a sine function, is the first in the sequence of weight matrices A weight matrix, is the first in the sequence of bias vectors A bias vector, It is the significant aggregation coding vector of the speech style spectrum features.

[0054] Then, based on each static potential weight factor, the weighted sum of each contextual Mel spectrum local encoding feature vector is calculated to obtain a compact and recognizable representation that can represent the core features of the entire speech segment, namely the contextual Mel spectrum pseudo-anchored aggregation center representation vector. Through this method, it is possible to clearly observe the influence of the center point on each feature vector, and infer the change path of the feature vector in the feature space.

[0055] After that, the aggregate movement direction of each contextual Mel spectrum local encoding feature vector relative to the pseudo-anchor aggregation center representation vector is calculated to obtain a sequence of contextual Mel spectrum aggregation running directions. This operation aims to measure the change tendency of each Mel spectrum feature vector relative to the aggregation center. These change trends indicate the direction of each Mel spectrum feature vector converging or moving away from the aggregation center, reflecting the interaction between each vector in the feature space and its change law, which is of great significance for identifying the relative position relationship and collective dynamic pattern between feature vectors. In other words, the aggregate movement direction can help identify those feature changes that have a significant impact on the speech style or content. For example, a sudden emotional outburst in a smooth speech may show an obvious change in movement direction.

[0056] Finally, based on the aggregation running direction of each context Mel spectrum, each context Mel spectrum local encoding feature vector is dynamically aggregated toward the context Mel spectrum pseudo-anchored aggregation center representation vector to obtain a speech style spectrum feature significant aggregation encoding vector. In other words, based on the previously obtained aggregation movement directions and the static potential energy measurement coefficients of each context Mel spectrum local encoding feature vector, the context Mel spectrum local encoding feature vector is adjusted and optimized using a simulated dynamics model to make it closer to the pseudo-anchored aggregation center vector. In this way, the dynamic characteristics of speech, such as intonation changes, emotional expression, etc., can be better captured to dynamically adjust and optimize the spatial distribution of feature vectors, generate a more compact, representative and recognizable representation, and help improve the model's understanding of the overall style of the speech segment.

[0057] In the embodiment of the present application, the broadcast voice generation module 150 is used to obtain the voice style recognition broadcast voice based on the significant aggregation coding features of the voice style spectrum features. Specifically, the broadcast voice generation module is used to: input the voice style spectrum feature significant aggregation coding vector into the trained voice style recognizer based on the large model to obtain the voice style recognition broadcast voice. That is, the voice style spectrum feature significant aggregation coding features obtained by dynamic aggregation of the sequence of the context Mel spectrum local coding feature vectors are used for generation processing to intelligently generate the voice style recognition broadcast voice. In this way, the system can comprehensively capture the local detail features and global context information of the voice signal, ensure that the complex style is fully characterized, and can better adapt to the significant differences in pronunciation, speech speed, tone, etc. between different speakers, and improve the generalization ability. It should be understood that large models in the field of artificial intelligence and machine learning usually refer to deep learning models with very large parameters, high computing resource requirements, and the ability to handle complex tasks. These models can reach or exceed the performance level of humans in various natural language processing (NLP), computer vision, speech recognition and other tasks by utilizing a large amount of training data and powerful computing power. In particular, a trained large-model-based speech style recognizer is an advanced AI system trained through deep learning that can recognize and imitate the speech style of a specific speaker or type. The recognizer is usually composed of multiple modules, each of which is responsible for processing different aspects of the speech signal and ultimately integrating this information to generate an accurate description of the input speech style.

[0058] In particular, considering that each contextual Mel-spectrogram local coding feature vector in the sequence of contextual Mel-spectrogram local coding feature vectors respectively represents the image semantic features of the Mel-spectrogram of the speech input signal in the local image semantic space domain, which is based on the local image semantic features of the global association of the context associations between the local image semantic space domains, when performing feature aggregation based on feature dynamics analysis, the insufficient feature dynamic correspondence of each local image semantic feature will also lead to the lack of decision-making of the aggregation feature instance of the speech style spectrum feature significant aggregation coding vector, thereby affecting the speech quality of the speech style recognition broadcast speech obtained by the speech style recognizer based on the large model whose input training has been completed.

[0059] Preferably, the speech style spectrum feature significant aggregation coding vector is input into a trained large-model-based speech style recognizer to obtain a speech style recognition broadcast voice, which includes: calculating the distance between each pair of feature values ​​of the speech style spectrum feature significant aggregation coding vector, such as the L2 distance, and taking the square root of the distance to obtain a speech style spectrum feature significant aggregation distance representation matrix, that is, ;in, and are respectively the first and The eigenvalues ​​at the positions, is the speech style spectral feature significant aggregation coding vector, yes and The L2 distance between is the first in the speech style spectrum feature significant aggregation distance representation matrix The feature value at each position.

[0060] The speech style spectrum feature significant aggregation autocorrelation matrix of the speech style spectrum feature significant aggregation coding vector is obtained, that is, ( is a row vector).

[0061] The speech style spectrum feature significant aggregation coding vector is matrix-multiplied with the speech style spectrum feature significant aggregation distance representation matrix, that is, After that, it is further multiplied by the matrix product of the speech style spectrum feature significant aggregation distance representation matrix and the speech style spectrum feature significant aggregation autocorrelation matrix, that is, Matrix multiplication is performed to obtain a multi-level mapping vector of speech style spectral feature saliency aggregation.

[0062] in, is the speech style spectrum feature significant aggregation distance representation matrix, yes The transposed vector of It's matrix multiplication.

[0063] The speech style spectral feature significant aggregation multi-level mapping vector is interpolated with the speech style spectral feature significant aggregation associated eigenvector composed of the eigenvalues ​​of the speech style spectral feature significant aggregation autocorrelation matrix (if the eigenvalue is insufficient, interpolation or zero padding) to obtain an optimized speech style spectral feature significant aggregation coding vector.

[0064] The optimized speech style spectrum feature significant aggregation coding vector is input into the trained large model-based speech style recognizer to obtain the speech style recognition broadcast speech.

[0065] Therefore, through the linear target mapping representation based on the similarity distance representation matrix of the speech style spectral feature significant aggregation coding vector, the self-correlation complete similarity instantiation of the speech style spectral feature significant aggregation coding vector is instantiated by a secondary target mapping representation based on a multi-level distribution hierarchy, and the association mismatch negative influence factor is compensated by the association fusion kernel bias to improve the instance determination degree of the eigenvalue of the speech style spectral feature significant aggregation coding vector under the similarity constraint, that is, the significance degree of the eigenvalue as an instance for the generative regression judgment, so as to improve the speech quality of the speech style recognition broadcast speech obtained by the speech style spectral feature significant aggregation coding vector input trained based on the large model.

[0066] In summary, the speech style recognition system 100 based on the large model of the embodiment of the present application is explained, which receives the speech input signal of the target user collected by the microphone, then extracts the Mel spectrum therefrom, and uses the signal analysis and processing technology based on deep learning and large models to perform block processing and multi-scale semantic encoding on the extracted Mel spectrum of the speech input signal, so as to intelligently generate the speech style recognition broadcast voice according to the dynamic feature aggregation representation between the local encoding features of the Mel spectrum of each context obtained by encoding. In this way, the system can comprehensively capture the local detail features and global context information of the speech signal, ensure that the complex style is fully characterized, and can better adapt to the significant differences in pronunciation, speaking speed, pitch, etc. between different speakers, and improve the generalization ability.

[0067] As described above, the speech style recognition system 100 based on the large model according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with a speech style recognition algorithm based on the large model. In a possible implementation, the speech style recognition system 100 based on the large model according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the speech style recognition system 100 based on the large model can be a software module in the operating system of the wireless terminal, or can be an application developed for the wireless terminal; of course, the speech style recognition system 100 based on the large model can also be one of the many hardware modules of the wireless terminal.

[0068] Alternatively, in another example, the large model-based speech style recognition system 100 and the wireless terminal may also be separate devices, and the large model-based speech style recognition system 100 may be connected to the wireless terminal via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0069] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details of the above application are only for the purpose of illustration and ease of understanding, rather than limitation, and the above details do not limit the present application to the use of the above specific details to be implemented.

Claims

1. A speech style recognition system based on a large model, characterized in that: include: A speech signal acquisition module is used to receive a speech input signal of a target user collected by a microphone; a speech input signal Mel spectrum extraction module is used to extract the Mel spectrum features of the speech input signal to obtain a speech input signal Mel spectrum graph; a Mel spectrum feature block encoding module is used to perform block processing and multi-scale semantic encoding on the speech input signal Mel spectrum graph to obtain a sequence of contextual Mel spectrum local encoding features; a Mel spectrum feature dynamic aggregation module is used to perform Mel spectrum local feature dynamic aggregation on the sequence of contextual Mel spectrum local encoding features to obtain a speech style spectrum feature significant aggregation encoding feature, wherein the Mel spectrum feature dynamic aggregation module includes: a contextual Mel spectrum clustering unit, which is used to calculate the contextual Mel spectrum pseudo-anchor aggregation center of the sequence of contextual Mel spectrum local encoding features; a contextual Mel spectrum feature dynamic aggregation unit, which is used to perform feature significant aggregation encoding on the sequence of contextual Mel spectrum local encoding features and the contextual Mel spectrum pseudo-anchor aggregation center to obtain the speech style spectrum feature significant aggregation encoding feature; a broadcast speech generation module is used to obtain a speech style recognition broadcast speech based on the speech style spectrum feature significant aggregation encoding feature.

2. The speech style recognition system based on a large model according to claim 1, characterized in that: The Mel spectrum feature block encoding module includes: a speech input signal blocking unit, used to block the speech input signal Mel spectrum map to obtain a sequence of speech input signal Mel spectrum map blocks; a Mel spectrum feature local encoding unit, used to perform Mel spectrum multi-scale semantic encoding on the sequence of speech input signal Mel spectrum map blocks to obtain a sequence of context Mel spectrum local encoding feature vectors as the sequence of context Mel spectrum local encoding features.

3. The speech style recognition system based on a large model according to claim 2, characterized in that: The Mel spectrum feature local encoding unit is used to: input the sequence of Mel spectrum graph blocks of the speech input signal into a Mel spectrum multi-scale semantic encoder including a convolutional layer and a converter structure to obtain a sequence of context Mel spectrum local encoding feature vectors.

4. The speech style recognition system based on a large model according to claim 3 is characterized in that: The context mel spectrum clustering unit includes: a context mel spectrum feature static potential weight factor calculation unit, which is used to calculate the static potential weight factor of each context mel spectrum local encoding feature vector in the sequence of context mel spectrum local encoding feature vectors to obtain a sequence of context mel spectrum local feature static potential weight factors; a mel spectrum feature weight weighting unit, which is used to calculate the weighted sum of the sequence of context mel spectrum local encoding feature vectors based on the sequence of context mel spectrum local feature static potential weight factors to obtain a context mel spectrum pseudo-anchor aggregation center representation vector as the context mel spectrum pseudo-anchor aggregation center.

5. The speech style recognition system based on a large model according to claim 4 is characterized in that: The context Mel spectrum feature static potential energy weight factor calculation unit includes: a static potential energy measurement coefficient calculation secondary subunit, which is used to input each context Mel spectrum local encoding feature vector in the sequence of context Mel spectrum local encoding feature vectors into a feature static potential energy measurement network to obtain a sequence of context Mel spectrum local feature static potential energy measurement coefficients; a static potential energy measurement coefficient screening gating secondary subunit, which is used to input the sequence of context Mel spectrum local feature static potential energy measurement coefficients into a feature energy level screening gating unit to obtain a sequence of context Mel spectrum local feature static potential energy weight factors.

6. The speech style recognition system based on a large model according to claim 5, characterized in that: The static potential energy measurement coefficient screening gated secondary subunit is used to: standardize each context Mel spectrum local feature static potential energy measurement coefficient in the sequence of context Mel spectrum local feature static potential energy measurement coefficients to obtain a sequence of standardized context Mel spectrum local feature static potential energy measurement coefficients; input the sequence of standardized context Mel spectrum local feature static potential energy measurement coefficients into a mask function for masking processing to obtain a sequence of context Mel spectrum local feature static potential energy weight factors.

7. The speech style recognition system based on a large model according to claim 6, characterized in that: The contextual Mel spectrum feature dynamic aggregation unit is used to: calculate the aggregation movement direction of each contextual Mel spectrum local encoding feature vector in the sequence of contextual Mel spectrum local encoding feature vectors relative to the contextual Mel spectrum pseudo-anchor aggregation center representation vector to obtain a sequence of contextual Mel spectrum aggregation running directions; based on the sequence of contextual Mel spectrum aggregation running directions, dynamically aggregate the sequence of contextual Mel spectrum local encoding feature vectors toward the contextual Mel spectrum pseudo-anchor aggregation center representation vector to obtain a speech style spectrum feature significant aggregation coding vector as the speech style spectrum feature significant aggregation coding feature.

8. The speech style recognition system based on a large model according to claim 7, characterized in that: The broadcast speech generation module is used to: input the speech style spectrum feature significant aggregation coding vector into the trained large model-based speech style recognizer to obtain the speech style recognition broadcast speech.

Images