Speech style expression switching system based on large model

Abstract

The present application provides a speech style expression switching system based on a large model, which relates to the field of intelligent speech interaction. It uses artificial intelligence-based data processing technology to extract, block and embed code the original speech signal based on Mel spectrum features to obtain a sequence of original speech Mel spectrum block embedding coding features, and semantically encodes the natural language description of the target speech style to obtain the target speech style semantic coding features, and then performs cross-modal interactive optimization encoding of the sequence of original speech Mel spectrum block embedding coding features and the target speech style semantic coding features. The features obtained are input into the trained end-to-end model to generate a style switching speech signal. In this way, user needs can be accurately understood, so that the switched speech style is more in line with user expectations. At the same time, through refined processing, it helps to make the switched speech style more natural and fluent.

Description

Technical Field

[0001] The present application relates to the field of intelligent voice interaction, and more specifically, to a voice style expression switching system based on a large model. Background Art

[0002] Voice style switching refers to the ability to make voice output present different styles in an AI voice interaction system. It involves changes in voice intonation, speech speed, timbre, word usage, and other factors, and aims to provide voice presentation methods that are suitable for different usage scenarios, user needs, and conversation content.

[0003] The traditional method is not accurate enough in understanding the user's expectations for the target voice style, and only relies on limited preset style options for users to choose from. This is difficult to fully meet the unique voice style requirements in the user's mind, and when switching between some voice styles, the switched voice style appears stiff and unnatural.

[0004] Therefore, an optimized voice style expression switching solution is expected. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a speech style expression switching system based on a large model.

[0006] According to one aspect of the present application, a speech style expression switching system based on a large model is provided, which includes: a speech signal data acquisition module, which is used to obtain an original speech signal collected by a microphone; a target speech style data acquisition module, which is used to obtain a natural language description of the target speech style input by a user; an original speech signal analysis module, which is used to extract and analyze Mel spectrum features of the original speech signal to obtain a sequence of original speech Mel spectrum block embedding coding features; a target speech style data semantic encoding module, which is used to semantically encode the natural language description of the target speech style to obtain target speech style semantic coding features; a cross-modal feature interaction optimization module, which is used to perform cross-modal prompt gate-guided feature interaction optimization on the sequence of original speech Mel spectrum block embedding coding features and the target speech style semantic coding features to obtain a sequence of original speech Mel spectrum block semantic coding features under speech style modulation; a style switching speech signal generation module, which is used to generate a style switching speech signal based on the sequence of original speech Mel spectrum block semantic coding features under speech style modulation.

[0007] Compared with the prior art, the speech style expression switching system based on a large model provided by the present application uses artificial intelligence-based data processing technology to extract, block and embed the original speech signal based on Mel spectrum features to obtain a sequence of original speech Mel spectrum block embedding coding features, and semantically encodes the natural language description of the target speech style to obtain the target speech style semantic coding features, and then performs cross-modal interactive optimization encoding of the sequence of original speech Mel spectrum block embedding coding features and the target speech style semantic coding features to obtain the features input into the trained end-to-end model to generate a style switching speech signal. In this way, user needs can be accurately understood, so that the switched speech style is more in line with user expectations, and at the same time, through refined processing, it helps to make the switched speech style more natural and fluent. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0009] In the attached picture: Figure 1 It is a system block diagram of a speech style expression switching system based on a large model according to an embodiment of the present application.

[0010] Figure 2 Schematic diagram of data flow of a large model-based speech style expression switching system according to an embodiment of the present application.

[0011] Figure 3 It is a block diagram of the original speech signal analysis module in the speech style expression switching system based on the large model according to the embodiment of the present application.

[0012] Figure 4 It is a block diagram of a cross-modal feature interaction optimization module in a large model-based speech style expression switching system according to an embodiment of the present application. DETAILED DESCRIPTION

[0013] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0014] In the field of artificial intelligence voice interaction, voice style conversion technology can make the machine's voice output show diverse style characteristics. This technology covers the adjustment of multiple dimensions such as voice pitch, rhythm, sound quality and language habits, in order to adapt to different application scenarios, meet the personalized needs of users, match the content of the conversation, and provide appropriate voice expression.

[0015] However, existing methods have limitations in capturing users' specific expectations of ideal voice styles. They usually only provide a limited number of preset styles for users to choose from, which can hardly fully meet users' personalized requirements for unique voice styles. In addition, in some cases, the effect of voice style transfer may appear abrupt and lack a smooth and natural feeling.

[0016] In response to the above technical problems, the technical concept of the present application is to first obtain the natural language description of the original speech signal and the target speech style input by the user, and then use the speech signal processing technology and text analysis technology based on artificial intelligence to extract, block and embed the original speech signal based on Mel spectrum features to obtain a sequence of original speech Mel spectrum block embedding coding features, and perform semantic coding on the natural language description to obtain the target speech style semantic coding features, and then perform cross-modal interactive optimization coding on the sequence of original speech Mel spectrum block embedding coding features and the target speech style semantic coding features to obtain the original speech Mel spectrum block semantic coding features under speech style modulation, and finally input the features obtained by cross-modal interactive optimization coding into the trained end-to-end model to generate a style switching speech signal. In this way, user needs can be accurately understood, so that the switched speech style is more in line with user expectations, and at the same time, through refined processing, it helps to make the switched speech style more natural and fluent.

[0017] Figure 1 It is a system block diagram of a speech style expression switching system based on a large model according to an embodiment of the present application. Figure 2 FIG. 1 is a data flow diagram of a speech style expression switching system based on a large model according to an embodiment of the present application. Figure 1 and Figure 2As shown, in the speech style expression switching system 100 based on the large model, it includes: a speech signal data acquisition module 110, which is used to obtain the original speech signal collected by the microphone; a target speech style data acquisition module 120, which is used to obtain the natural language description of the target speech style input by the user; an original speech signal analysis module 130, which is used to extract and analyze the Mel spectrum features of the original speech signal to obtain a sequence of original speech Mel spectrum block embedding coding features; a target speech style data semantic encoding module 140, which is used to semantically encode the natural language description of the target speech style to obtain the target speech style semantic coding features; a cross-modal feature interaction optimization module 150, which is used to perform cross-modal prompt gate-guided feature interaction optimization on the sequence of original speech Mel spectrum block embedding coding features and the target speech style semantic coding features to obtain a sequence of original speech Mel spectrum block semantic coding features under speech style modulation; a style switching speech signal generation module 160, which is used to generate a style switching speech signal based on the sequence of original speech Mel spectrum block semantic coding features under speech style modulation.

[0018] In the embodiment of the present application, the voice signal data acquisition module 110 is used to obtain the original voice signal collected by the microphone. It should be understood that the collected original voice signal contains acoustic feature information such as pitch, sound intensity and timbre. In addition, it also contains voice content-related feature information such as voice word sequence and pause. In general, the original voice signal is the basic material for the entire voice style switching, which carries all the relevant features of the voice at the beginning. The voice content and various acoustic features contained in the original voice signal can allow the system to accurately know the state of the original voice. When switching styles, it can ensure that the semantic content of the original voice is not changed, avoiding the occurrence of semantic errors after switching styles; at the same time, the original acoustic features also provide a reference for the style switching amplitude, so as not to change excessively and cause the voice to lose some of its original recognition characteristics.

[0019] In the embodiment of the present application, the target voice style data acquisition module 120 is used to obtain the natural language description of the target voice style input by the user. It should be understood that the user will mention the style characteristics of the desired voice in the natural language description, such as whether he wants a "lively and cheerful" style, a "calm and solemn" style, or a "humorous and witty" style. These style characteristics cover the requirements of the overall feeling and atmosphere of the voice, and will affect the intonation, speaking speed, wording and other aspects. For example, the "lively and cheerful" style often means an upward tone, a relatively fast speaking speed, and the wording may be more relaxed and playful; while the "calm and solemn" style is usually a steady tone, a moderate and slightly slower speaking speed, and a more formal and standardized wording. In addition to the overall style and timbre, the user may also clearly express the specific details of the intonation and speaking speed. For example, it is mentioned that "the tone should have obvious ups and downs, and the emphasis should focus on improving the tone", or "the speaking speed should be slightly slower than normal, about 100 words per minute". These detailed requirements on intonation and speech speed allow the system to adjust the corresponding voice parameters more accurately, so that the final generated voice style is more in line with what the user wants. In other words, the natural language description of the target voice style is equivalent to setting a clear goal and direction for the style switching voice signal to be generated. By parsing these natural language descriptions, the system knows what style the original voice needs to be transformed into.

[0020] In the embodiment of the present application, the original speech signal analysis module 130 is used to extract and analyze the Mel spectrum features of the original speech signal to obtain a sequence of original speech Mel spectrum block embedding coding features. Specifically, Figure 3 FIG. 1 is a block diagram of an original speech signal analysis module in a speech style expression switching system based on a large model according to an embodiment of the present application. Figure 3 As shown, the original speech signal analysis module 130 includes: a speech signal Mel spectrum feature extraction unit 131, used to extract the Mel spectrum features of the original speech signal to obtain an original speech Mel spectrum graph; an original speech Mel spectrum graph block embedding coding unit 132, used to perform block processing and embedding coding on the original speech Mel spectrum graph to obtain a sequence of original speech Mel spectrum graph block embedding coding vectors as a sequence of original speech Mel spectrum graph block embedding coding features.

[0021] In the embodiment of the present application, the speech signal Mel spectrum feature extraction unit 131 is used to extract the Mel spectrum feature of the original speech signal to obtain the original speech Mel spectrum graph. Accordingly, considering that the human ear is relatively sensitive to the frequency change of low-frequency sound, while for high-frequency sound, the frequency change is not so obvious in perception. In order to analyze the characteristics of the original speech signal more in line with the actual feeling of the human ear hearing the sound, in the technical solution of the present application, it is necessary to extract the Mel spectrum feature of the original speech signal to obtain the original speech Mel spectrum graph. It should be explained that the Mel frequency scale remaps the physical frequency of the original speech signal according to the human ear perception characteristics. The Mel spectrum feature specifically refers to the energy distribution of the speech signal over time based on the Mel frequency scale. Since the Mel spectrum feature adopts the Mel frequency scale that conforms to the human ear perception, when the speech is subsequently adjusted in style, the generated style switching speech signal will sound more natural and reasonable to the human ear based on such features. Here, the original speech Mel spectrum graph obtained after extracting the Mel spectrum feature presents the complex acoustic information of the speech in a visual and quantifiable form. In the spectrum diagram, different color depths, stripe shapes, etc. can intuitively reflect the energy of speech at different times and different Mel frequencies, which is convenient for further extraction and analysis of valuable features. For example, it can be clearly understood which time periods in the speech are the parts with concentrated energy, and which Mel frequency areas play a key role in the overall timbre and intonation, which can provide an accurate data basis for more in-depth speech style modulation.

[0022] Extracting the Mel spectrum features of the original speech signal to obtain the original speech Mel spectrum graph is intended to convert the original speech signal into a spectrum graph form that is more in line with the auditory characteristics of the human ear and convenient for subsequent analysis and processing. The following is a detailed description of its implementation process: First, the original speech signal collected by the microphone needs to be sampled and quantized to convert it into a digital signal. Usually, a suitable sampling rate is selected according to specific needs, such as the common 16kHz or higher, and the corresponding quantization accuracy, such as 16 bits or higher. This step is the basis for subsequent digital processing, ensuring that the speech signal can be accurately recorded and processed in digital form.

[0023] After sampling and quantization, the digital speech signal is framed and windowed. Generally speaking, the duration of each frame is set at about 20-40 milliseconds, and there will be a certain overlap between adjacent frames, with a common overlap ratio of half. The purpose of framing is to divide the continuous speech signal into relatively independent segments so that each small segment can be analyzed. The windowing operation is to reduce the spectrum leakage of the signal in the frequency domain and make the spectrum analysis results more accurate. For example, for a speech signal with a sampling rate of 16kHz, each frame may contain 320 sampling points (assuming a frame length of 20 milliseconds), and adjacent frames overlap by 160 sampling points. In the windowing process, a suitable window function such as a Hamming window or a Hanning window is used to process each frame of the signal.

[0024] Then, a fast Fourier transform (FFT) is performed on each frame of the windowed speech signal. FFT is a key link in the entire process. It can convert the time domain signal into a frequency domain signal, thereby obtaining the component information of the signal at different frequencies. Taking a frame of signal with 320 sampling points as an example, its spectrum distribution in the frequency domain can be obtained through FFT, and usually 160 frequency domain data points are obtained (due to the symmetry of the FFT result). This step enables the speech signal to be analyzed from a frequency perspective to understand its frequency composition.

[0025] After obtaining the frequency domain signal, it is necessary to convert the linear frequency scale obtained by FFT into the Mel frequency scale according to the human ear perception characteristics. There is a specific conversion relationship between the Mel frequency scale and the actual frequency, and the approximate formula is melf=2595log 10 (1+f / 700), where f is the actual frequency. Through this conversion, the frequency points that were originally evenly distributed on the Hz scale will show a nonlinear distribution on the Mel scale. This nonlinear distribution can better reflect the difference in sensitivity of the human ear to sounds of different frequencies. For example, lower frequencies will be relatively stretched on the Mel scale, while higher frequencies will be relatively compressed. Such a conversion will help to analyze the speech signal in a more consistent way with the perception characteristics of the human ear.

[0026] Next, the energy spectrum of each frame of speech signal is calculated on the Mel frequency scale. In this process, the frequency domain amplitude obtained by FFT is first squared, and then the frequency is grouped according to the Mel frequency scale, and the sum of the energy in each Mel frequency band is calculated. Usually a set of Mel filter banks are defined, which are triangularly distributed on the Mel frequency scale and cover the main frequency range of the speech signal. Generally, 20-40 Mel filters may be used, and the center frequency of each filter is evenly distributed according to the Mel scale, and there is a certain overlap between adjacent filters. By multiplying the square of each frequency domain amplitude with the corresponding Mel filter coefficient and summing them, the energy value of each Mel frequency band can be obtained. This energy value constitutes a column vector in the original speech Mel spectrum diagram, which corresponds to the energy distribution of a frame of signal in different Mel frequency bands.

[0027] As the speech signal is processed frame by frame, the Mel spectrum vectors obtained from each frame are arranged in sequence to form a two-dimensional matrix. In this matrix, the horizontal axis represents time (frame number), the vertical axis represents Mel frequency, and the element value in the matrix represents the energy at the corresponding time and Mel frequency. This two-dimensional matrix is ​​the required original speech Mel spectrum map. For example, if the total duration of the speech signal is 3 seconds, there will be about 150 frames of signal calculated based on 20 milliseconds per frame and 16kHz sampling rate. If 30 Mel filters are used, the final Mel spectrum map size is 150×30 (assuming that other preprocessing factors are not considered). In the spectrum map, the color depth or grayscale value can intuitively represent the energy size. The darker the color (or the larger the grayscale value), the stronger the energy at that time-frequency point. Through such a Mel-spectrogram of the original speech, we can clearly observe the temporal changes of different frequency components in the speech signal, which provides an extremely important basis for the subsequent further analysis and processing of the speech signal. Processing based on such features will make the generated style switching speech signal sound more natural and reasonable to the human ear, and also provide an accurate data basis for deeper speech style modulation.

[0028] In an embodiment of the present application, the original speech Mel spectrogram block embedding coding unit 132 is used to perform block processing and embedding coding on the original speech Mel spectrogram to obtain a sequence of original speech Mel spectrogram block embedding coding vectors as a sequence of original speech Mel spectrogram block embedding coding features. It should be understood that the original speech Mel spectrogram often contains rich and large amounts of information, covering the energy distribution of speech at different times and Mel frequencies, etc., and the overall data volume is large and the complexity is high. Directly performing subsequent processing on such a large and complex data will be very high in terms of both computational cost and model processing difficulty. In order to facilitate subsequent further operations and analysis, it is necessary to perform block processing operations on the original speech Mel spectrogram. Each local image block after block processing can reflect the acoustic characteristics of the original speech from different angles and different positions. Different blocks may correspond to different time periods and different frequency intervals of speech, and they each carry unique information, such as some blocks reflecting the low-frequency characteristics of the initial stage of speech, and some blocks reflecting the high-frequency energy changes in the middle part. Combining the features of these different blocks can show the various characteristics of the original speech more comprehensively and richly than directly using the overall spectrogram, and enhance the overall representation ability of the original speech features. Then, considering that the original local image blocks obtained after block division carry certain acoustic feature information, this information may be relatively low-level and intuitive. In order to deeply explore the deeper semantic features in these original speech mel-spectrogram blocks, in the technical solution of this application, it is necessary to embed the sequence of the original speech mel-spectrogram local image blocks obtained after the block division process to obtain the sequence of embedded coding vectors of the original speech mel-spectrogram blocks.

[0029] The original speech Mel spectrogram is processed by block processing and embedded coding to obtain a sequence of embedded coding vectors of the original speech Mel spectrogram blocks, which aims to extract a representative feature sequence that is convenient for subsequent processing from the original speech Mel spectrogram. The following is a detailed description of its implementation process: The first is the block processing link. Before starting the block division, a suitable block strategy must be determined. The determination of this strategy depends on the size of the original speech Mel spectrogram and the expectation of feature representation in the specific application scenario. Generally, the original speech Mel spectrogram has a certain dimensional specification. It is assumed that its size is M×N, where M represents the number of time frames, reflecting the distribution of speech in the time dimension, and N represents the Mel frequency dimension, reflecting the distribution of different frequency components. Based on this, a suitable block size is selected for division according to needs, for example, it can be set to a small block size of m×n, where m and n correspond to the block size in the time and frequency directions respectively. In practical applications, the selection of these block sizes is not arbitrary, but needs to be carefully adjusted according to specific needs. When you want to capture the local subtle features of speech more accurately, you may tend to choose a relatively small block size, so that you can focus on the local details of the spectrum and dig out more hidden feature information; on the contrary, if you need to grasp the overall characteristics of speech from a macro perspective, a larger block size is more suitable, which can cover a wider time and frequency range and provide a more global feature description.

[0030] After determining the block strategy, the actual block operation phase begins. According to the selected block size, the Mel spectrum map can be divided into multiple sub-blocks. There are two common block methods, one is non-overlapping block, and the other is block with certain overlap. For the non-overlapping method, starting from the upper left corner of the spectrum map, each sub-block is taken out in turn according to the established m×n size, just like taking out non-overlapping small puzzle pieces one by one from a complete puzzle. When overlap is allowed, each time a block is taken, a certain step size (the step size is less than the block size) is moved in the time or frequency direction. This method can increase the diversity of data and make each block contain richer information, but it also inevitably increases the amount of calculation and data. Take a specific example to illustrate, assuming that the size of the original speech Mel spectrum map is M=100, N=80, select m=10, n=8 block size and adopt non-overlapping block method, then through simple calculation, it can be known that 10×10 blocks can be obtained in the end. Each of these tiles has a unique meaning, representing the local characteristics of the original speech within a specific time and frequency range.

[0031] After the block processing is completed, the next stage is the embedding coding stage. In this stage, you first need to choose a suitable embedding coding method. The commonly used methods currently include neural network models (such as convolutional neural networks CNN, etc.) and methods based on mathematical transformations (such as discrete cosine transform DCT, etc.). Taking convolutional neural networks as an example, it has a strong automatic feature learning ability and is very suitable for processing image data (the Mel spectrum block here can be regarded as a special type of image data). When using convolutional neural networks for embedding coding, a series of operation steps are required. First, if it is an untrained model, it must be trained on a large amount of relevant data so that it can learn an effective feature encoding method. This training process involves multiple key elements, among which the definition of the network structure is crucial. Usually, a network structure containing different levels of combinations such as convolutional layers, pooling layers, and fully connected layers is constructed. The convolutional layer can effectively extract the local features of the block by sliding the convolution kernel on the block; the pooling layer mainly plays the role of reducing the data dimension, reducing the amount of data while retaining key information; the function of the fully connected layer is to integrate the features extracted from the previous levels and map them to the final encoding vector space. For example, a carefully trained CNN model may encode an m×n tile into a fixed-length (assumed to be k-dimensional) vector, which becomes the embedding code vector of the tile.

[0032] As for the methods based on mathematical transformation, such as discrete cosine transform DCT, their principles are different from those of neural networks. DCT does not require a training process like a neural network, but directly encodes the image blocks according to the transformation formula defined by it. Specifically, the pixel values ​​of the image blocks (the pixel values ​​here are actually the energy values ​​of the speech signal on the Mel spectrogram) are subjected to discrete cosine transform, and a coefficient vector is obtained after the transformation. In this coefficient vector, different coefficients represent the information of different frequency components. Usually, low-frequency coefficients often contain the main feature information of the image block, so some of the coefficients (such as low-frequency coefficients) can be selected as encoding vectors. In this way, whether using a neural network model or a method based on mathematical transformation, each Mel spectrogram block obtained after block processing can be converted into an embedded encoding vector. Finally, the encoding vectors of all the blocks are arranged in a certain order to form a sequence of embedded encoding vectors of the original speech Mel spectrogram blocks, which is also the sequence of the desired embedded encoding features of the original speech Mel spectrogram blocks. Compared with the original Mel-spectrogram, it can more effectively represent the characteristics of the original speech in different local areas, and provides a more suitable and easy-to-operate data form for further processing such as subsequent cross-modal interaction optimization.

[0033] In an embodiment of the present application, the target voice style data semantic encoding module 140 is used to semantically encode the natural language description of the target voice style to obtain the target voice style semantic encoding features. Specifically, in an embodiment of the present application, the target voice style data semantic encoding module is used to: input the natural language description of the target voice style into a voice style semantic encoder to obtain a target voice style semantic encoding feature vector as the target voice style semantic encoding feature. The voice style semantic encoder described here is a voice style semantic encoder including a Bert model. It should be understood that the natural language description of the target voice style input by the user may be relatively intuitive and simple, but there are many deep semantic associations hidden therein. For example, when the user describes and inputs "I hope the voice style is light and lively with a little playfulness", there is an inherent style connection between "light and lively" and "playfulness", and they have a comprehensive impact on multiple aspects such as the intonation and wording of the voice. In order to deeply explore these hidden semantic associations, extract more abstract and representative features, and accurately grasp the user's overall expectations for voice style, the technical solution of the present application needs to input the natural language description of the target voice style into a voice style semantic encoder containing a Bert model to obtain a target voice style semantic encoding feature vector. Those of ordinary skill in the art should know that the Bert model itself is a pre-trained language model based on the Transformer architecture. It has been pre-trained with a large amount of text data, and has learned a wealth of language semantic knowledge and grammatical structure information, and can well understand the various complex semantic relationships in natural language descriptions. And unlike traditional unidirectional language models, the Bert model uses a bidirectional encoding mechanism, which can simultaneously consider the contextual information of words when processing text, which enables a more comprehensive and accurate semantic encoding of the entire natural language description, avoiding the semantic understanding deviation that may be caused by only considering unidirectional information.

[0034] In an embodiment of the present application, the cross-modal feature interaction optimization module 150 is used to perform feature interaction optimization guided by a cross-modal prompt gate on the sequence of the original speech mel-spectrogram tile embedding coding features and the target speech style semantic coding features to obtain a sequence of original speech mel-spectrogram tile semantic coding features under speech style modulation. It should be understood that the sequence of original speech mel-spectrogram tile embedding coding vectors represents information from the perspective of speech acoustic features, while the target speech style semantic coding feature vector carries the speech style expectation information based on the user's natural language description, and they belong to data of different modalities. In order to make the acoustic features of the speech itself and the user's expected speech style requirements correlated with each other, break the modal barriers, and lay the foundation for subsequent speech style modulation, in the technical solution of the present application, it is necessary to perform feature interaction optimization guided by a cross-modal prompt gate on the sequence of the original speech mel-spectrogram tile embedding coding features and the target speech style semantic coding features to obtain a sequence of original speech mel-spectrogram tile semantic coding features under speech style modulation.

[0035] Specifically, Figure 4 FIG. 1 is a block diagram of a cross-modal feature interaction optimization module in a speech style expression switching system based on a large model according to an embodiment of the present application. Figure 4 As shown, the cross-modal feature interaction optimization module 150 includes: a speech style semantic interaction weight calculation unit 151, which is used to calculate the sequence of semantic interaction attention weights under speech style modulation based on the sequence of the target speech style semantic coding feature vector and the original speech mel-spectrogram block embedding coding vector; a speech style semantic cross-modal interaction coding unit 152, which is used to perform cross-modal interaction on the target speech style semantic coding feature vector and the sequence of the original speech mel-spectrogram block embedding coding vector to obtain a sequence of original speech mel-spectrogram interaction local feature vectors; a speech style semantic cross-modal feature interaction optimization unit 153, which is used to perform cross-modal interaction optimization on the sequence of semantic interaction attention weights under speech style modulation and the sequence of original speech mel-spectrogram interaction local feature vectors to obtain a sequence of original speech mel-spectrogram block semantic coding feature vectors under speech style modulation as a sequence of original speech mel-spectrogram block semantic coding features under speech style modulation.

[0036] Specifically, in an embodiment of the present application, the semantic interaction weight calculation unit is used to: perform a linear transformation on the target speech style semantic coding feature vector to obtain a target speech style semantic query coding vector and a target speech style semantic value coding vector; perform converter-based cross-modal prompt information encoding on each original speech mel-spectrogram block embedding coding vector in the sequence of the target speech style semantic query coding vector, the target speech style semantic value coding vector and the original speech mel-spectrogram block embedding coding vector to obtain a sequence of semantic interaction coding vectors under speech style modulation; perform information gating screening-based decoding processing on the sequence of semantic interaction coding vectors under speech style modulation to obtain a sequence of semantic interaction attention weights under speech style modulation.

[0037] Specifically, the target speech style semantic encoding feature vector is first linearly transformed to obtain a target speech style semantic query encoding vector and a target speech style semantic value encoding vector. The above process can be expressed as: ;in, is the semantic encoding feature vector of the target speech style, represents the query embedding encoding matrix, represents the target speech style semantic query encoding vector, represents the value embedding encoding matrix, Represents the target speech style semantic value encoding vector.

[0038] That is, through linear mapping, the target speech style semantic encoding feature vector is converted into a form suitable for attention mechanism calculation, where the target speech style semantic query encoding vector is used to find correlations with other modal features, while the target speech style semantic value encoding vector preserves important information of the original features. This conversion is the key to the subsequent implementation of the interaction with the original speech mel spectrogram tile embedding encoding data, which enables information from different modalities to be compared and fused in a unified way. It is worth mentioning that linear transformation is not just a simple mathematical operation, but by introducing trainable parameters to adjust the spatial distribution of feature representation, so that the converted query encoding vector and value encoding vector are more suitable for cross-modal comparison and fusion. Specifically, the target speech style semantic query encoding vector is used to guide the search direction in the attention mechanism, while the target speech style semantic value encoding vector preserves the information to be fused. This design can effectively promote the accurate matching and efficient integration of information between different modalities.

[0039] Next, the target speech style semantic query encoding vector, the target speech style semantic value encoding vector, and each original speech mel spectrogram block embedding encoding vector in the sequence of the original speech mel spectrogram block embedding encoding vector are respectively subjected to converter-based cross-modal prompt information encoding to obtain a sequence of semantic interaction encoding vectors under speech style modulation. The above process can be expressed as follows: ;in, is a sequence of embedding encoding vectors of the original speech mel-spectrogram blocks, and They represent the first, second and third in the sequence of the original speech Mel spectrogram block embedding encoding vectors. The mel-spectrogram block embedding encoding vector of the original speech, represents the target speech style semantic query encoding vector, To represent the semantic value encoding vector of the target speech style, express function, is the first in the sequence of the original speech Mel spectrogram patch embedding encoding vectors The original speech Mel spectrogram patch embedding encoding vector, express The transposed vector of express Length, The first in the sequence of semantic interaction encoding vectors modulated by speech style Semantic interaction encoding vector modulated by speech style.

[0040] In this process, the converter uses its core self-attention mechanism to effectively capture the complex interaction patterns between the semantic encoding features of the target speech style and the embedding encoding features of the original speech Mel-spectrogram patches, so that the sequence of semantic interaction encoding vectors generated under speech style modulation not only retains the information of the semantic encoding features of the target speech style, but also integrates the supplementary information from the original speech signal, thereby improving the richness and accuracy of the representation of the semantic encoding features of the target speech style.

[0041] Next, the sequence of semantic interaction coding vectors modulated by the speech style is decoded based on information gating screening to obtain a sequence of semantic interaction attention weights modulated by the speech style. The above process can be expressed as follows: ;in, express function, The first in the sequence of semantic interaction encoding vectors modulated by speech style The semantic interaction encoding vector modulated by speech style, for The corresponding semantic interaction attention weight under speech style modulation, Represents a decoder, Represents the decoding weight matrix.

[0042] That is, information screening is performed on the sequence of semantic interaction coding vectors modulated by the input speech style, which helps the model to flexibly allocate attention resources according to the context of the input information, avoid over-reliance on a certain modality and ignore other clues that may be equally important, and thus help to more accurately capture the complementarity and synergy between the original speech signal and the natural language description of the target speech style.

[0043] Specifically, in an embodiment of the present application, the speech style semantic cross-modal interaction encoding unit is used to: perform position point multiplication on the target speech style semantic feature vector with each original speech mel-spectrogram block embedding coding vector in the sequence of original speech mel-spectrogram block embedding coding vectors to obtain a sequence of original speech mel-spectrogram interaction local feature vectors.

[0044] Afterwards, the target speech style semantic coding feature vector is cross-modally interacted with the sequence of the original speech mel-spectrogram block embedding coding vectors to obtain a sequence of original speech mel-spectrogram interaction local feature vectors. Finally, the sequence of semantic interaction attention weights modulated by speech style and the sequence of original speech mel-spectrogram interaction local feature vectors are cross-modally optimized to obtain a sequence of original speech mel-spectrogram block semantic coding feature vectors modulated by speech style. The above two steps can be comprehensively expressed by the formula: ;in, Encode the feature vector for the target speech style semantics, They represent the first, second and third semantic interaction attention weights in the sequence under speech style modulation. cross-modal semantic interaction attention weights, and They represent the first, second and third in the sequence of the original speech Mel spectrogram block embedding encoding vectors. The mel-spectrogram block embedding encoding vector of the original speech, It means point multiplication by position. Represents a sequence of semantically encoded feature vectors of the original speech mel-spectrogram patches under speech style modulation.

[0045] That is, the direct interaction between the features of the two different modal data, the original speech signal and the natural language description, is achieved by positional dot multiplication. This method can effectively capture the complementary information of the two modalities at the same spatial position, thereby enhancing the spatial consistency of the feature representation. The final cross-modal interaction optimization combines the attention mechanism and the interaction features to generate a sequence of semantic encoding feature vectors of the original speech Mel spectrogram blocks under speech style modulation with stronger expressive power and less redundant information. It can not only maintain the fidelity to the original input speech signal and natural language description, but also help to serve the downstream style switching speech signal generation task.

[0046] In an embodiment of the present application, the style switching voice signal generation module 160 is used to generate a style switching voice signal based on a sequence of semantic coding features of the original voice Mel spectrogram block under the voice style modulation. Specifically, in an embodiment of the present application, the style switching voice signal generation module is used to: input the sequence of semantic coding feature vectors of the original voice Mel spectrogram block under the voice style modulation into the voice style switching engine based on the end-to-end large model trained to obtain the style switching voice signal. It should be understood that the end-to-end large model is a deep learning architecture mode, which has powerful feature learning and generation capabilities after training with a large amount of data. For the vector sequence under the input voice style modulation, it can accurately generate the expected voice signal based on the learned knowledge and pattern, according to the style requirements and original voice basic features contained in these vectors. This voice signal can accurately reflect the voice style expected by the user, achieve the timbre, appropriate intonation, expected speech speed and corresponding word habits required by the user, so that the voice is successfully transformed from the original state to a presentation form that meets the target style, and meets the user's needs for different styles of voice in the voice interaction scenario.

[0047] The end-to-end large model-based speech style switching engine trained by inputting the sequence of semantic encoding vectors of the original speech Mel spectrogram tiles modulated by speech style to generate style switching speech signals is a step involving multiple links and high technical requirements. The following is a detailed description of its implementation process: First, the model preparation stage. The key to this stage is to select a suitable end-to-end large model architecture, such as the common sequence-to-sequence model based on deep learning (especially the Transformer model and its variants), which is favored for its powerful feature learning and conversion capabilities. After selecting the model, it is necessary to use a large amount of speech data for training. These data cover the original speech samples and their corresponding speech samples of various styles, and may also contain relevant style labels or description information. During the training process, it is crucial to define a suitable loss function (such as mean square error) to measure the difference between the model prediction and the actual target speech signal. It is a key indicator for evaluating the degree of closeness between the acoustic features of the speech signal generated by the model and the target. At the same time, the model parameters are adjusted with the help of optimization algorithms (such as stochastic gradient descent and its variants Adagrad, Adadelta, Adam, etc.) to minimize the loss function, so as to enable the model to continuously optimize performance and accuracy. For example, the Adam optimization algorithm can dynamically adjust the parameter learning rate based on the first-order and second-order moment estimates of the gradient to accelerate model convergence. After training is completed, an independent validation data set is used to evaluate and verify the model, and by calculating speech quality evaluation indicators (such as PESQ, STOI, etc.), it is determined whether the generated speech signal meets the standard. If not, it is necessary to further adjust the model structure, training parameters or supplement training data.

[0048] Next is the input preprocessing phase. The sequence of semantic encoding vectors of the original speech Mel spectrogram tiles under speech style modulation needs to be formatted to fit the input specification of the selected end-to-end large model, which may involve adjusting the vector dimension operation to ensure that the input data format is unified and efficient.

[0049] Finally, the model inference and speech signal generation phase is entered. The preprocessed encoding vector sequence is input into the trained end-to-end large model, and the model processes the input data through a series of complex neural network layers. Taking the Transformer model as an example, the input first passes through the multi-head attention layer to calculate the vector correlation and capture the voice feature dependency, and then the feedforward neural network layer transforms the mapping features, and finally generates a new feature sequence containing the target voice style information based on the learned style mapping relationship. The generated new feature sequence needs to be converted into an actual voice signal through an inverse transform, such as first performing an inverse Mel spectrum transform to convert the Mel spectrum feature into a linear spectrum, and then performing an inverse Fourier transform to obtain a time domain waveform. In this process, the voice signal must be normalized to ensure a reasonable volume, and smoothed to eliminate discontinuities and noise, so that the final generated style switching voice signal presents the target voice style characteristics in terms of timbre, intonation, speaking speed, and word usage habits, meeting the diverse voice style requirements in voice interaction scenarios.

[0050] In particular, considering that the sequence of embedded coding vectors of the original speech mel-spectrogram blocks and the target speech style semantic coding feature vector respectively represent the embedded coding features in the local semantic space of the original speech mel-spectrogram and the full semantic coding features of the natural language description of the target speech style, when performing cross-modal interaction optimization based on cross-modal cue gates, insufficient responsiveness of the semantic cue gating of heteromodal data will also lead to the lack of cross-modal feature instance determination of the sequence of semantic coding vectors of the original speech mel-spectrogram blocks under the speech style modulation, thereby affecting the signal quality of the style switching speech signal obtained by the speech style switching engine based on the end-to-end large model that has been trained as input.

[0051] Preferably, inputting the sequence of original speech Mel-spectrogram block semantic coding feature vectors under speech style modulation into a speech style switching engine based on an end-to-end large model that has been trained to obtain a style switching speech signal comprises: cascading the sequence of original speech Mel-spectrogram block semantic coding feature vectors under speech style modulation into an original speech Mel-spectrogram block semantic cascade coding vector under speech style modulation; calculating the distance between each pair of eigenvalues ​​of the original speech Mel-spectrogram block semantic cascade coding vector under speech style modulation, such as the L2 distance, and taking the square root of the distance to obtain the original speech Mel-spectrogram block semantic cascade distance representation matrix under speech style modulation. The process can be expressed by the formula: ;in, represents the semantic concatenated encoding vector of the original speech Mel spectrogram block under speech style modulation, represents the semantic concatenated encoding vector of the original speech Mel spectrogram block under speech style modulation The eigenvalues ​​at the positions, represents the semantic concatenated encoding vector of the original speech Mel spectrogram block under speech style modulation The eigenvalues ​​at the positions, express and The distance between express and The square root of the distance between them.

[0052] The original speech Mel spectrogram block semantic cascade autocorrelation matrix under speech style modulation of the original speech Mel spectrogram block semantic cascade coding vector under speech style modulation is obtained. The process can be expressed by the formula: ;in, represents the semantic concatenated encoding vector of the original speech Mel spectrogram block under speech style modulation, and is a row vector, express The transposed vector of represents matrix multiplication, Represents the semantic concatenation autocorrelation matrix of the original speech Mel-spectrogram patches under speech style modulation.

[0053] After matrix multiplication of the original speech Mel-spectrogram block semantic concatenation encoding vector under speech style modulation and the original speech Mel-spectrogram block semantic concatenation distance representation matrix under speech style modulation, further matrix multiplication is performed with the matrix product of the original speech Mel-spectrogram block semantic concatenation distance representation matrix under speech style modulation and the original speech Mel-spectrogram block semantic concatenation self-correlation matrix under speech style modulation to obtain the original speech Mel-spectrogram block semantic concatenation multi-level mapping vector under speech style modulation. This process can be expressed by the formula: ;in, represents the semantic concatenated encoding vector of the original speech Mel spectrogram block under speech style modulation, Represents the semantic cascade distance representation matrix of the original speech Mel spectrogram patch under speech style modulation, represents matrix multiplication, Represents the semantic cascade autocorrelation matrix of the original speech Mel spectrogram block under speech style modulation, Represents the semantic cascade multi-level mapping vector of the original speech Mel-spectrogram patch under speech style modulation.

[0054] The optimized original speech mel-spectrogram block semantic cascade coding vector under speech style modulation is obtained by performing a dot operation on the multi-level mapping vector of the semantic cascade of the original speech mel-spectrogram block under speech style modulation and the semantic cascade association eigenvector of the original speech mel-spectrogram block under speech style modulation composed of the eigenvalues ​​of the semantic cascade self-association matrix of the original speech mel-spectrogram block under speech style modulation (if the eigenvalue is insufficient, interpolation or zero padding is performed).

[0055] The sequence of the original speech mel-spectrogram block semantic coding vectors under the optimized speech style modulation after the original speech mel-spectrogram block semantic cascade coding vectors under the optimized speech style modulation is restored and input into the trained end-to-end large model-based speech style switching engine to obtain the style switching speech signal.

[0056] That is, through the linear target mapping representation based on the similarity distance representation matrix of the original speech Mel-spectrogram block semantic cascade coding vector under the speech style modulation, the self-correlation of the original speech Mel-spectrogram block semantic cascade coding vector under the speech style modulation is instantiated with a secondary target mapping representation based on a multi-level distribution hierarchy, and the association mismatch negative influence factor is compensated by the association fusion kernel bias to improve the instance decision degree of the eigenvalue of the original speech Mel-spectrogram block semantic cascade coding vector under the similarity constraint under the speech style modulation, that is, the significance degree of the eigenvalue as an instance for the generative regression decision, so as to improve the signal quality of the style switching speech signal obtained by the speech style switching engine based on the end-to-end large model which is trained by the sequence input of the original speech Mel-spectrogram block semantic coding vector under the speech style modulation.

[0057] In summary, a large-model-based voice style expression switching system 100 based on the embodiment of the present application is explained, which uses artificial intelligence-based data processing technology to extract, block and embed code the original voice signal based on Mel spectrum features to obtain a sequence of original voice Mel spectrum block embedding coding features, and semantically encodes the natural language description of the target voice style to obtain the target voice style semantic coding features, and then cross-modal interactive optimization encoding of the original voice Mel spectrum block embedding coding features and the target voice style semantic coding features. The features obtained are input into the trained end-to-end model to generate a style switching voice signal. In this way, user needs can be accurately understood, so that the switched voice style is more in line with user expectations. At the same time, through refined processing, it helps to make the switched voice style more natural and fluent.

Claims

1. A speech style expression switching system based on a large model, characterized in that: include: A voice signal data acquisition module is used to obtain the original voice signal collected by a microphone; A target speech style data acquisition module is used to obtain a natural language description of the target speech style input by the user; an original speech signal analysis module is used to extract and analyze the Mel spectrum features of the original speech signal to obtain a sequence of original speech Mel spectrum block embedding coding features; a target speech style data semantic coding module is used to semantically encode the natural language description of the target speech style to obtain the target speech style semantic coding features; a cross-modal feature interaction optimization module is used to perform cross-modal cue gate-guided feature interaction optimization on the sequence of original speech Mel spectrum block embedding coding features and the target speech style semantic coding features to obtain a sequence of original speech Mel spectrum block semantic coding features under speech style modulation; a style switching speech signal generation module is used to generate a style switching speech signal based on the sequence of original speech Mel spectrum block semantic coding features under speech style modulation; Among them, the cross-modal feature interaction optimization module includes: a speech style semantic interaction weight calculation unit, which is used to calculate the sequence of semantic interaction attention weights under speech style modulation based on the sequence of target speech style semantic coding feature vectors and the original speech mel-spectrogram block embedding coding vectors; a speech style semantic cross-modal interaction coding unit, which is used to perform cross-modal interaction on the target speech style semantic coding feature vector and the sequence of original speech mel-spectrogram block embedding coding vectors to obtain a sequence of original speech mel-spectrogram interaction local feature vectors; a speech style semantic cross-modal feature interaction optimization unit, which is used to perform cross-modal interaction optimization on the sequence of semantic interaction attention weights under speech style modulation and the sequence of original speech mel-spectrogram interaction local feature vectors to obtain a sequence of original speech mel-spectrogram block semantic coding feature vectors under speech style modulation as the sequence of original speech mel-spectrogram block semantic coding features under speech style modulation.

2. The speech style expression switching system based on a large model according to claim 1 is characterized in that: The original speech signal analysis module includes: a speech signal Mel spectrum feature extraction unit, which is used to extract the Mel spectrum features of the original speech signal to obtain an original speech Mel spectrum graph; an original speech Mel spectrum graph block embedding coding unit, which is used to perform block processing and embedding coding on the original speech Mel spectrum graph to obtain a sequence of original speech Mel spectrum graph block embedding coding vectors as a sequence of original speech Mel spectrum graph block embedding coding features.

3. The speech style expression switching system based on a large model according to claim 2 is characterized in that: The target speech style data semantic encoding module is used to: input the natural language description of the target speech style into a speech style semantic encoder to obtain a target speech style semantic encoding feature vector as the target speech style semantic encoding feature.

4. The speech style expression switching system based on a large model according to claim 3 is characterized in that: The speech style semantic encoder is a speech style semantic encoder including a Bert model.

5. The speech style expression switching system based on a large model according to claim 4 is characterized in that: The semantic interaction weight calculation unit is used to: perform linear transformation on the target speech style semantic coding feature vector to obtain a target speech style semantic query coding vector and a target speech style semantic value coding vector; perform converter-based cross-modal prompt information encoding on each original speech mel-spectrogram block embedding coding vector in the sequence of the target speech style semantic query coding vector, the target speech style semantic value coding vector and the original speech mel-spectrogram block embedding coding vector to obtain a sequence of semantic interaction coding vectors under speech style modulation; The sequence of semantic interaction coding vectors under the speech style modulation is decoded based on information gating screening to obtain the sequence of semantic interaction attention weights under the speech style modulation.

6. The speech style expression switching system based on a large model according to claim 5 is characterized in that: The speech style semantic cross-modal interaction encoding unit is used to: perform position point multiplication on the target speech style semantic encoding feature vector and each original speech mel-spectrogram block embedding encoding vector in the sequence of original speech mel-spectrogram block embedding encoding vectors to obtain a sequence of original speech mel-spectrogram interaction local feature vectors.

7. The speech style expression switching system based on a large model according to claim 6 is characterized in that: The style switching speech signal generating module is used to: input the sequence of original speech mel-spectrogram block semantic encoding feature vectors modulated by the speech style into a trained speech style switching engine based on an end-to-end large model to obtain the style switching speech signal.

Images