End-to-end speech large model interaction system

Abstract

The present application relates to the field of intelligent interaction, and specifically discloses an end-to-end speech large model interaction system, which collects user input speech signals by sound sensors, and uses deep learning and end-to-end speech signal analysis and coding technology to perform speech signal segmentation and embedding fusion on the user input speech signals, and extracts intonation pattern features from the user input speech signals, thereby intelligently generating answering speech signals based on the fine-grained compensation fusion representation between the embedded fusion features of each user input speech syllable signal and the user input speech intonation features. In this way, the system can better understand the user's intentions and emotions, thereby providing a more humane and natural response. At the same time, it can adapt to different language environments, thereby enhancing the adaptability and generalization ability of the system.

Description

Technical Field

[0001] The present application relates to the field of intelligent interaction, and more specifically, to an end-to-end speech large model interaction system. Background Art

[0002] In the digital age, voice technology has become an important bridge for human-computer interaction. From simple text-to-speech conversion to complex voice interaction systems, technological advances are constantly driving innovations in the way we communicate with machines, in order to achieve natural, smooth, and efficient interaction between humans and machines, greatly improving the user experience.

[0003] However, traditional voice interaction systems rely on pre-set rules and simple feature extraction methods, making it difficult to deeply understand complex contextual information, especially in capturing subtle emotional changes and intonation information in speech. For example, when a user expresses dissatisfaction, he or she may use a higher pitch, faster speaking speed, and more pauses to emphasize his or her emotions, while traditional systems can often only recognize the basic emotion of "dissatisfaction" but cannot accurately capture the user's specific emotional intensity and changes. In addition, the rule-based approach of traditional systems has limited adaptability when faced with new vocabulary or rapidly changing language habits, and requires frequent updates of models and dictionaries. These problems limit the widespread application of traditional voice interaction systems in diverse application scenarios.

[0004] Therefore, an optimized voice intelligent interaction system is desired. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides an end-to-end speech large model interaction system, which collects user input speech signals by sound sensors, and uses deep learning and end-to-end speech signal analysis and coding technology to perform speech signal segmentation and embedding fusion on the user input speech signals, and extracts intonation pattern features from the user input speech signals, so as to intelligently generate answering speech signals according to the fine-grained compensation fusion representation between the embedded fusion features of each user input speech syllable signal and the user input speech intonation features. In this way, the emotional changes and intonation information in the speech can be captured more accurately, so that the system can better understand the user's intentions and emotions, thereby providing a more humane and natural response. At the same time, it can adapt to different language environments, reducing the need for a large amount of manual tuning, so that the system can still maintain a high recognition accuracy and generation quality when facing new vocabulary or background noise, thereby enhancing the adaptability and generalization ability of the system.

[0006] According to one aspect of the present application, an end-to-end speech large model interaction system is provided, which includes: a speech signal acquisition module, which is used to acquire a user input speech signal collected by a sound sensor; a speech signal segmentation and embedding module, which is used to perform speech signal segmentation and embedding fusion on the user input speech signal to obtain a user input speech syllable signal embedded coding cascade feature; a speech intonation coding module, which is used to extract intonation pattern features on the user input speech signal to obtain a user input speech intonation coding feature; a speech feature compensation fusion module, which is used to perform speech feature fine-grained compensation fusion on the user input speech syllable signal embedded coding cascade feature and the user input speech intonation coding feature to obtain a user input speech syllable semantic-intonation interaction compensation fusion. The combined representation is as follows, wherein the speech feature compensation fusion module includes: a speech common feature extraction unit, used to extract the common features between the user-input speech syllable signal embedded coding cascade features and the user-input speech intonation coding features to obtain a common feature representation between the user-input speech features; a common feature guided fusion unit, used to perform common feature guided complementary fusion of the user-input speech syllable signal embedded coding cascade features and the user-input speech intonation coding features based on the common feature representation between the user-input speech features to obtain the user-input speech syllable semantic-intonation interactive compensation fusion representation; an answer speech signal generation module, used to obtain an answer speech signal based on the user-input speech syllable semantic-intonation interactive compensation fusion representation.

[0007] In the above-mentioned end-to-end speech large model interaction system, the speech signal segmentation and embedding module includes: a speech signal segmentation unit, which is used to perform syllable-based speech signal segmentation on the user input speech signal to obtain a sequence of user input speech syllable signal segments; a speech signal embedding coding unit, which is used to use a syllable signal embedding coding matrix to embed and code each user input speech syllable signal segment in the sequence of user input speech syllable signal segments to obtain a sequence of user input speech syllable signal embedding coding vectors; a speech signal cascade unit, which is used to cascade the sequence of user input speech syllable signal embedding coding vectors to obtain a user input speech syllable signal embedding coding cascade vector as the user input speech syllable signal embedding coding cascade feature.

[0008] In the above-mentioned end-to-end speech large model interaction system, the speech intonation encoding module is used to: input the user input speech signal into the intonation pattern feature extractor based on the void convolutional neural network model to obtain the user input speech intonation encoding feature vector as the user input speech intonation encoding feature.

[0009] In the above-mentioned end-to-end speech large model interaction system, the speech common feature extraction unit is used to: embed the user input speech syllable signal into the encoding cascade vector and the user input speech intonation encoding feature vector into the common feature extraction network to obtain a common feature representation vector between the user input speech features as the common feature representation between the user input speech features.

[0010] In the above-mentioned end-to-end speech large model interaction system, the speech common feature extraction unit is used to: embed the coding cascade vector of the user-input speech syllable signal and the user-input speech intonation coding feature vector according to the position point to obtain the user-input speech syllable intonation point-added vector; multiply the user-input speech syllable intonation point-added vector with the weight matrix, and add the obtained user-input speech syllable intonation weight vector and the bias vector according to the position to obtain the user-input speech syllable intonation commonality vector; input the user-input speech syllable intonation commonality vector into the tanh function for processing to obtain the commonality feature representation vector between the user-input speech features.

[0011] In the above-mentioned end-to-end speech large model interaction system, the common feature guided fusion unit includes: a speech feature probabilization subunit, which is used to input the user input speech syllable signal embedding coding cascade vector, the user input speech intonation coding feature vector and the common feature representation vector between the user input speech features into a vector probabilization unit based on the Sigmoid function to obtain the probabilistic user input speech syllable signal embedding coding cascade vector, the probabilistic user input speech intonation coding feature vector and the common feature representation vector between the probabilistic user input speech features; a speech syllable information compensation calculation subunit, which is used to calculate the user input speech syllable information fine-grained compensation vector of the probabilistic user input speech syllable signal embedding coding cascade vector relative to the probabilistic user input speech feature common feature representation vector; a speech intonation information compensation calculation subunit, which is used to calculate the probabilistic user input speech intonation coding feature vector relative to the probabilistic user input speech feature common feature representation vector between the probabilistic user input speech features. A fine-grained compensation vector of user input speech intonation information of a common feature representation vector; a speech syllable feature compensation subunit, used to embed the fine-grained compensation vector of the user input speech syllable information and the user input speech syllable signal into a coding cascade vector input fine-grained compensation module to obtain a user input speech syllable fine-grained feature compensation vector; a speech intonation feature compensation subunit, used to input the fine-grained compensation vector of the user input speech intonation information and the user input speech intonation coding feature vector into the fine-grained compensation module to obtain a user input speech intonation fine-grained feature compensation vector; a speech syllable speech intonation compensation fusion subunit, used to cascade the user input speech syllable fine-grained feature compensation vector, the user input speech intonation fine-grained feature compensation vector and the common feature representation vector between the user input speech features to obtain a user input speech syllable semantic-intonation interaction compensation fusion representation vector as the user input speech syllable semantic-intonation interaction compensation fusion representation.

[0012] In the above-mentioned end-to-end speech large model interaction system, the speech syllable information compensation calculation subunit is used to: calculate the position point division of the probabilistic user input speech syllable signal embedding coding cascade vector and the common feature representation vector between the probabilistic user input speech features to obtain the user input speech syllable contribution vector; calculate the logarithmic function value with base two of the absolute value of each eigenvalue of the user input speech syllable contribution vector to obtain the user input speech syllable contribution logarithmic vector; calculate the position point multiplication of the probabilistic user input speech syllable signal embedding coding cascade vector and the user input speech syllable contribution logarithmic vector to obtain the point product vector, and calculate the exponential function with the natural constant e as the base, and each point product value in the point product vector is the exponent to obtain the user input speech syllable compensation vector; input the user input speech syllable compensation vector into the softmax function to obtain the fine-grained compensation vector of the user input speech syllable information.

[0013] In the above-mentioned end-to-end speech large model interaction system, the speech syllable feature compensation subunit is used to: calculate the point-by-point multiplication of the user-input speech syllable signal embedded coding cascade vector and the user-input speech syllable information fine-grained compensation vector to obtain the user-input speech syllable fine-grained feature compensation vector.

[0014] In the above-mentioned end-to-end speech large model interaction system, the answer speech signal generation module is used to: input the user input speech syllable semantic-intonation interaction compensation fusion representation vector into the speech interaction model based on the end-to-end model to obtain the answer speech signal.

[0015] Compared with the prior art, the present application provides an end-to-end speech large model interaction system, which uses a sound sensor to collect user input speech signals, and uses deep learning and end-to-end speech signal analysis and coding technology to perform speech signal segmentation and embedding fusion on the user input speech signals, and extracts intonation pattern features from the user input speech signals, so as to intelligently generate answering speech signals based on the fine-grained compensation fusion representation between the embedded fusion features of each user input speech syllable signal and the user input speech intonation features. In this way, the emotional changes and intonation information in the speech can be captured more accurately, so that the system can better understand the user's intentions and emotions, thereby providing a more humane and natural response. At the same time, it can adapt to different language environments, reducing the need for a large amount of manual tuning, so that the system can still maintain a high recognition accuracy and generation quality when facing new vocabulary or background noise, thereby enhancing the adaptability and generalization ability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0017] Figure 1 It is a block diagram of an end-to-end speech large model interaction system according to an embodiment of the present application.

[0018] Figure 2 Schematic diagram of data flow of an end-to-end speech large model interaction system according to an embodiment of the present application.

[0019] Figure 3 It is a block diagram of a speech signal segmentation and embedding module in an end-to-end speech large model interaction system according to an embodiment of the present application.

[0020] Figure 4 It is a block diagram of a speech feature compensation fusion module in an end-to-end speech large model interaction system according to an embodiment of the present application.

[0021] Figure 5 This is a block diagram of a common feature guided fusion unit in an end-to-end speech large model interaction system according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] The embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, but rather these embodiments are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0023] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0024] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0025] It should be noted that in this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where the application is located and with the authorization of the owner of the corresponding device.

[0026] In the digital age, voice technology has become an important bridge for human-computer interaction. From simple text-to-speech conversion to complex voice interaction systems, technological advances are constantly driving innovations in the way we communicate with machines, in order to achieve natural, smooth, and efficient interaction between humans and machines, greatly improving the user experience.

[0027] However, traditional voice interaction systems rely on pre-set rules and simple feature extraction methods, making it difficult to deeply understand complex contextual information, especially in capturing subtle emotional changes and intonation information in speech. For example, when a user expresses dissatisfaction, he or she may use a higher pitch, faster speaking speed, and more pauses to emphasize his or her emotions, while traditional systems can often only recognize the basic emotion of "dissatisfaction" but cannot accurately capture the user's specific emotional intensity and changes. In addition, the rule-based approach of traditional systems has limited adaptability when faced with new vocabulary or rapidly changing language habits, and requires frequent updates of models and dictionaries. These problems limit the widespread application of traditional voice interaction systems in diverse application scenarios.

[0028] Therefore, in response to the above technical problems, the technical concept of the present application is to collect user input voice signals by sound sensors, and use deep learning and end-to-end voice signal analysis and coding technology to perform voice signal segmentation and embedding fusion on the user input voice signals, and extract intonation pattern features from the user input voice signals, so as to intelligently generate answer voice signals based on the fine-grained compensation fusion representation between the features of each user input voice syllable signal embedded and fused and the user input voice intonation features. In this way, the emotional changes and intonation information in the voice can be captured more accurately, so that the system can better understand the user's intentions and emotions, thereby providing a more humane and natural response. At the same time, it can adapt to different language environments, reducing the need for a large amount of manual tuning, so that the system can still maintain a high recognition accuracy and generation quality when facing new vocabulary or background noise, thereby enhancing the adaptability and generalization ability of the system.

[0029] Figure 1 It is a block diagram of an end-to-end speech large model interaction system according to an embodiment of the present application. Figure 2 FIG. 1 is a schematic diagram of data flow in an end-to-end speech large model interaction system according to an embodiment of the present application. Figure 1 and Figure 2As shown, according to the embodiment of the present application, the end-to-end speech large model interaction system 100 includes: a speech signal acquisition module 110, which is used to acquire the user input speech signal collected by the sound sensor; a speech signal segmentation and embedding module 120, which is used to perform speech signal segmentation and embedding fusion on the user input speech signal to obtain the user input speech syllable signal embedded coding cascade feature; a speech intonation coding module 130, which is used to extract the intonation pattern feature of the user input speech signal to obtain the user input speech intonation coding feature; a speech feature compensation fusion module 140, which is used to perform speech feature fine-grained compensation fusion on the user input speech syllable signal embedded coding cascade feature and the user input speech intonation coding feature to obtain the user input speech syllable semantic-intonation interactive compensation fusion representation; an answer speech signal generation module 150, which is used to obtain the answer speech signal based on the user input speech syllable semantic-intonation interactive compensation fusion representation.

[0030] In an embodiment of the present application, the voice signal acquisition module 110 is used to acquire the user input voice signal collected by the sound sensor. It should be understood that considering that the user input voice signal contains multiple aspects of information, such as syllable information in the voice signal, that is, representing specific vocabulary or phonemes and intonation information, that is, the emotional state of the user when speaking. In order to accurately understand and process the user's voice input, thereby generating high-quality answers, in the technical solution of the present application, the user input voice signal collected by the sound sensor is acquired, and the syllable information can be extracted by analyzing and understanding it, and the words and sentence content spoken by the user can be understood, thereby accurately identifying the user's intentions, and at the same time, the user's emotional state can be identified, and a more emotionally consistent answer can be generated, thereby improving the user's satisfaction and experience.

[0031] In the embodiment of the present application, the speech signal segmentation and embedding module 120 is used to perform speech signal segmentation and embedding fusion on the user input speech signal to obtain the user input speech syllable signal embedding coding cascade feature. Figure 3The block diagram of the speech signal segmentation and embedding module in the end-to-end speech large model interaction system according to the embodiment of the present application. Specifically, in the embodiment of the present application, the speech signal segmentation and embedding module 120 includes: a speech signal segmentation unit 121, which is used to perform syllable-based speech signal segmentation on the user input speech signal to obtain a sequence of user input speech syllable signal segments; a speech signal embedding coding unit 122, which is used to embed and code each user input speech syllable signal segment in the sequence of user input speech syllable signal segments using a syllable signal embedding coding matrix to obtain a sequence of user input speech syllable signal embedding coding vectors; a speech signal cascade unit 123, which is used to cascade the sequence of user input speech syllable signal embedding coding vectors to obtain a user input speech syllable signal embedding coding cascade vector as the user input speech syllable signal embedding coding cascade feature.

[0032] Specifically, the speech signal segmentation unit 121 is used to perform syllable-based speech signal segmentation on the user input speech signal to obtain a sequence of user input speech syllable signal segments. Accordingly, considering that different local signal segments in the user input speech signal contain different syllable characteristics, in order to more carefully capture and analyze the subtle changes in the speech in each signal segment, in the technical solution of the present application, the user input speech signal is segmented based on syllables to obtain a sequence of user input speech syllable signal segments. That is, syllables are one of the basic units that constitute language, and are also an important element of emotional expression. Through the detailed analysis of syllables, the speaker's emotional state, such as happiness, sadness, anger, etc., can be more accurately captured, and the speaker's intentions and expressed content can be more accurately identified, providing support for the generation of subsequent answering speech.

[0033] Specifically, the speech signal embedding coding unit 122 is used to embed and code each user input speech syllable signal segment in the sequence of the user input speech syllable signal segments using a syllable signal embedding coding matrix to obtain a sequence of user input speech syllable signal embedding coding vectors. It should be understood that in order to better capture and extract the main features of the syllables expressed in each user input speech syllable signal segment, such as frequency, duration, energy, etc., so as to more clearly understand the semantic content in each syllable signal segment, in the technical solution of the present application, the syllable signal embedding coding matrix is ​​used to embed and code each user input speech syllable signal segment in the sequence of the user input speech syllable signal segments to obtain a sequence of user input speech syllable signal embedding coding vectors. That is, each user input speech syllable signal segment contains a large amount of information, which is usually high-dimensional, and through the syllable signal embedding coding matrix, each user input speech syllable signal segment can be converted into a fixed-length vector, which can capture the main features and details of each syllable, thereby providing better data representation for subsequent processing.

[0034] Specifically, the speech signal cascade unit 123 is used to cascade the sequence of the user-input speech syllable signal embedded coding vector to obtain the user-input speech syllable signal embedded coding cascade vector as the user-input speech syllable signal embedded coding cascade feature. Accordingly, considering that the speech signal is essentially a time series, the temporal arrangement order of each syllable is very important for understanding the meaning of the entire sentence. Therefore, in order to retain the sequential relationship between each syllable and to perform context understanding and semantic analysis more coherently, in the technical solution of the present application, the sequence of the user-input speech syllable signal embedded coding vector is cascaded to obtain the user-input speech syllable signal embedded coding cascade vector. In this way, a global representation containing rich information can be formed, so that the cascaded user-input speech syllable signal embedded coding cascade vector can better reflect the dependency relationship between the previous and next syllables, which helps to capture the overall structure and semantics of the sentence.

[0035] In the embodiment of the present application, the speech intonation encoding module 130 is used to extract the intonation pattern feature of the user input speech signal to obtain the user input speech intonation encoding feature. Specifically, in the embodiment of the present application, the speech intonation encoding module is used to: input the user input speech signal into the intonation pattern feature extractor based on the dilated convolutional neural network model to obtain the user input speech intonation encoding feature vector as the user input speech intonation encoding feature. It should be understood that considering that the user input speech signal also contains different semantic information, the intonation pattern usually spans a longer time range, for example, the beginning and end of a sentence may have different intonations. And considering that the dilated convolution expands the receptive field by inserting holes in the convolution kernel (i.e. skipping some input positions), it can effectively capture long-distance dependencies, which is very important for understanding the intonation pattern in speech. Therefore, in the technical solution of the present application, the user input speech signal is input into the intonation pattern feature extractor based on the dilated convolutional neural network model to refine and mine the intonation features between different time spans to obtain the user input speech intonation encoding feature vector. That is to say, compared with traditional pooling operations, dilated convolutions can expand the receptive field while maintaining a higher temporal resolution. This means that the model can capture a wider range of contextual information while maintaining temporal details, and subtle changes in speech (such as short pauses, stress, etc.) often carry important emotional and intention information.

[0036] In an embodiment of the present application, the speech feature compensation fusion module 140 is used to perform speech feature fine-grained compensation fusion on the embedded coding cascade features of the user input speech syllable signal and the user input speech intonation coding features to obtain a user input speech syllable semantic-intonation interactive compensation fusion representation. Figure 4 is a block diagram of a speech feature compensation fusion module in an end-to-end speech large model interaction system according to an embodiment of the present application. Figure 4As shown, the speech feature compensation fusion module 140 includes: a speech common feature extraction unit 141, which is used to extract the common features between the user input speech syllable signal embedded coding cascade features and the user input speech intonation coding features to obtain the common feature representation between the user input speech features; a common feature guided fusion unit 142, which is used to perform common feature guided complementary fusion on the user input speech syllable signal embedded coding cascade features and the user input speech intonation coding features based on the common feature representation between the user input speech features to obtain the user input speech syllable semantic-intonation interactive compensation fusion representation. Further, considering that the user input speech syllable signal embedded coding cascade vector mainly captures the semantic information of the speech content, and the user input speech intonation coding feature vector captures the emotion and tone information of the speech, if only one feature is used, some important information may be lost. For example, the same word may be expressed differently in different emotional states (such as happy, sad, angry), and it is difficult to distinguish these emotional differences by syllable embedded coding alone. Similarly, different words may be difficult to distinguish under the same intonation. Based on this, in order to make full use of the complementarity of the two types of information, improve the richness and accuracy of the overall representation, and more comprehensively understand the user's intentions and emotions, in the technical solution of the present application, the user input speech syllable signal embedding coding cascade vector and the user input speech intonation coding feature vector are subjected to speech feature fine-grained compensation fusion to obtain a user input speech syllable semantic-intonation interaction compensation fusion representation vector as a user input speech syllable semantic-intonation interaction compensation fusion representation.

[0037] Specifically, first, the common features between the user input speech syllable signal embedding encoding cascade vector and the user input speech intonation encoding feature vector are extracted to obtain the common feature representation vector between the user input speech features as the common feature representation between the user input speech features. Specifically, the common pattern and feature information between the two are captured by using a common feature extraction network, for example, by using a fully connected layer to mine and extract the common internal connection between the two, such as the association between certain specific syllable combinations and intonation patterns, to provide a basis for subsequent feature compensation.

[0038] Next, based on the common feature representation between the user input speech features, the user input speech syllable signal embedding coding cascade features and the user input speech intonation coding features are subjected to common feature guided complementary fusion to obtain the user input speech syllable semantic-intonation interactive compensation fusion representation. Specifically, in order to be able to convert the feature values ​​in these vectors into a weighted form to better evaluate and quantify the importance of each feature, a vector probabilization unit based on the Sigmoid function is used to probabilize the user input speech syllable signal embedding coding cascade vector and the user input speech intonation coding feature vector and common features. That is, the Sigmoid function can map any real value to the interval (0, 1) to generate a probability value to reflect the importance and reliability of the feature in a specific context. Afterwards, the fine-grained compensation vectors of the probabilistic user input speech syllable signal embedding coding cascade vector and the probabilistic user input speech intonation coding feature vector relative to the common feature representation vector between the probabilistic user input speech features are calculated respectively, and the fine-grained compensation vector of the user input speech syllable information and the fine-grained feature compensation vector of the user input speech intonation are obtained, so that the original unique feature information can be highlighted, that is, those details that may be ignored or not obvious in the common features but are crucial to the model processing. Then, the speech syllable information compensation vector and the user input speech syllable signal embedding coding cascade vector are respectively input into the fine-grained compensation module, and the speech intonation compensation vector and the user input speech intonation coding feature vector are simultaneously input into the fine-grained compensation module, so as to add the compensation vector to the original vector by means of penalty operation, so as to highlight the key information in the feature. Finally, the compensated user input speech syllable fine-grained feature compensation vector, user input speech intonation fine-grained feature compensation vector and common feature representation vector are cascaded to obtain the user input speech syllable semantic-intonation interactive compensation fusion representation vector, which includes the semantic information of the syllable, the emotional information of the intonation and the common features between them, and can more comprehensively understand the user's input speech.

[0039] Specifically, the speech common feature extraction unit 141 is used to: embed the user input speech syllable signal into the encoding cascade vector and the user input speech intonation encoding feature vector into the common feature extraction network to obtain a common feature representation vector between user input speech features as the common feature representation between user input speech features.

[0040] Specifically, in an embodiment of the present application, the speech common feature extraction unit is used to: embed the coding cascade vector of the user-input speech syllable signal and the user-input speech intonation coding feature vector by position point to obtain a user-input speech syllable intonation point-added vector; multiply the user-input speech syllable intonation point-added vector by a weight matrix, and add the obtained user-input speech syllable intonation weight vector and a bias vector by position to obtain a user-input speech syllable intonation commonality vector; input the user-input speech syllable intonation commonality vector into a tanh function for processing to obtain a commonality feature representation vector between user-input speech features.

[0041] Figure 5 1 is a block diagram of a common feature guided fusion unit in an end-to-end speech large model interaction system according to an embodiment of the present application. Specifically, Figure 5 As shown, the common feature guided fusion unit 142 includes: a speech feature probabilization subunit 1421, which is used to input the user input speech syllable signal embedding coding cascade vector, the user input speech intonation coding feature vector and the common feature representation vector between the user input speech features into a vector probabilization unit based on the Sigmoid function to obtain a probabilistic user input speech syllable signal embedding coding cascade vector, a probabilistic user input speech intonation coding feature vector and a probabilistic user input speech feature common feature representation vector; a speech syllable information compensation calculation subunit 1422, which is used to calculate the user input speech syllable information fine-grained compensation vector of the probabilistic user input speech syllable signal embedding coding cascade vector relative to the probabilistic user input speech feature common feature representation vector; a speech intonation information compensation calculation subunit 1423, which is used to calculate the probabilistic user input speech intonation coding feature vector relative to the probabilistic user input speech feature common feature representation vector. A fine-grained compensation vector of user input speech intonation information of the vector; a speech syllable feature compensation subunit 1424, used to embed the fine-grained compensation vector of the user input speech syllable information and the user input speech syllable signal into the encoding cascade vector input fine-grained compensation module to obtain the user input speech syllable fine-grained feature compensation vector; a speech intonation feature compensation subunit 1425, used to input the fine-grained compensation vector of the user input speech intonation information and the user input speech intonation encoding feature vector into the fine-grained compensation module to obtain the user input speech intonation fine-grained feature compensation vector; a speech syllable speech intonation compensation fusion subunit 1426, used to cascade the user input speech syllable fine-grained feature compensation vector, the user input speech intonation fine-grained feature compensation vector and the common feature representation vector between the user input speech features to obtain the user input speech syllable semantic-intonation interaction compensation fusion representation vector as the user input speech syllable semantic-intonation interaction compensation fusion representation.

[0042] Specifically, in an embodiment of the present application, the speech syllable information compensation calculation subunit 1422 is used to: calculate the position point division of the probabilistic user input speech syllable signal embedding coding cascade vector and the common feature representation vector between the probabilistic user input speech features to obtain a user input speech syllable contribution vector; calculate the base-two logarithmic function value of the absolute value of each eigenvalue of the user input speech syllable contribution vector to obtain a user input speech syllable contribution logarithmic vector; calculate the position point multiplication of the probabilistic user input speech syllable signal embedding coding cascade vector and the user input speech syllable contribution logarithmic vector to obtain a point product vector, and calculate an exponential function with the natural constant e as the base, with each point product value in the point product vector as the exponent to obtain a user input speech syllable compensation vector; input the user input speech syllable compensation vector into a softmax function to obtain a fine-grained compensation vector of the user input speech syllable information.

[0043] Specifically, in an embodiment of the present application, the speech syllable feature compensation subunit 1424 is used to: calculate the point-by-point multiplication of the user-input speech syllable signal embedded coding cascade vector and the user-input speech syllable information fine-grained compensation vector to obtain the user-input speech syllable fine-grained feature compensation vector.

[0044] In the embodiment of the present application, specifically, the speech feature fine-grained compensation fusion is performed on the syllable signal embedding coding cascade vector of the user input speech and the intonation coding feature vector of the user input speech, which can be expressed by the formula: .

[0045] in, and are respectively the user input speech syllable signal embedding coding concatenated vector and the user input speech intonation coding feature vector, To add by location, and are the weight matrix and bias vector respectively, yes function, is the common feature representation vector between the user input speech features, yes function, is the probabilistic user input speech syllable signal embedding encoding cascade vector, is the probabilistic user input speech intonation encoding feature vector, is the common feature representation vector between the probabilistic user input speech features, is the feature value of each position in the probabilistic user input speech syllable signal embedded in the encoding cascade vector, is the feature value of each position in the probabilistic user input speech intonation encoding feature vector, is the feature value of each position in the common feature representation vector between the probabilistic user input speech features, represents the logarithmic function value with base 2, Represents the exponential function value with the natural constant e as the base yes function, is the fine-grained compensation vector of the user input speech syllable information, is the fine-grained feature compensation vector of the user input speech intonation, It is the point multiplication by position. is the fine-grained feature compensation vector of the user input speech syllable, is the fine-grained feature compensation vector of the user input speech intonation, It is a cascade operation on vectors. It is the semantic-intonation interaction compensation fusion representation vector of the user input speech syllable.

[0046] In an embodiment of the present application, the answering voice signal generating module 150 is used to obtain an answering voice signal based on the semantic-intonation interaction compensation fusion representation of the user input voice syllable. Specifically, in an embodiment of the present application, the answering voice signal generating module is used to: input the semantic-intonation interaction compensation fusion representation vector of the user input voice syllable intonation ...ncencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencencence

[0047] In particular, in a specific embodiment of the present application, the user input speech syllable semantic-intonation interaction compensation fusion representation vector is input into a speech interaction model based on an end-to-end model to obtain an answering speech signal, which can be achieved by the following steps: First, an end-to-end model is selected, such as Transformer, LSTM, GRU, etc. Then, the model is trained using a large amount of annotated speech data to ensure that the model can accurately understand and generate speech signals. Next, the user input speech syllable semantic-intonation interaction compensation fusion representation vector is input as input into the end-to-end model. Afterwards, the model generates an answering speech text based on the input fusion representation vector. In particular, this step usually uses a decoder, and the decoder generates the corresponding text based on the input feature vector. Finally, the answering speech text generated by a speech synthesis module (such as Text-to-Speech, TTS) is converted into the final answering speech signal.

[0048] It should be understandable that when the user input speech syllable signal embedding coding cascade vector and the user input speech intonation coding feature vector respectively represent the time domain embedding coding features and signal waveform image semantic features of the user input speech syllable signal, when performing fine-grained compensation fusion of speech features, the common feature mask interactive compensation representation of the user input speech syllable semantic-intonation interactive compensation fusion representation vector will also have an interactive class regression understanding representation imbalance relative to the data heteromodal feature distribution field selective state space, thereby deviating from the temporal characteristics of the source data and the image semantic distribution representation class understanding, affecting the accuracy of the classification results.

[0049] Preferably, the method of inputting the user input speech syllable semantic-intonation interaction compensation fusion representation vector into a speech interaction model based on an end-to-end model to obtain an answer speech signal includes: determining the number of zero eigenvalues ​​in the user input speech syllable semantic-intonation interaction compensation fusion representation vector, and respectively calculating the reciprocal of the logarithm of the number of zero eigenvalues ​​with base two and the exponential value of the reciprocal of the number of zero eigenvalues ​​with base natural constants to obtain a first user input speech syllable semantic-intonation interaction compensation fusion representation value and a second user input speech syllable semantic-intonation interaction compensation fusion representation value; calculating a power function of each eigenvalue of the user input speech syllable semantic-intonation interaction compensation fusion representation vector with the reciprocal of the number of zero eigenvalues ​​as an exponent to obtain a user input speech syllable semantic-intonation interaction compensation fusion structure state vector; and comparing the user input speech syllable semantic-intonation interaction compensation fusion structure state vector with the The first user input speech syllable semantic-intonation interaction compensation fusion representation value is dot-multiplied to obtain the user input speech syllable semantic-intonation interaction compensation fusion information representation vector; the autocorrelation matrix of the user input speech syllable semantic-intonation interaction compensation fusion structure state vector is dot-multiplied with the second user input speech syllable semantic-intonation interaction compensation fusion representation value and weight hyperparameter respectively to obtain the user input speech syllable semantic-intonation interaction compensation fusion regression understanding matrix; the user input speech syllable semantic-intonation interaction compensation fusion information representation vector is matrix-multiplied with the user input speech syllable semantic-intonation interaction compensation fusion regression understanding matrix to obtain the optimized user input speech syllable semantic-intonation interaction compensation fusion representation vector; the optimized user input speech syllable semantic-intonation interaction compensation fusion representation vector is input into the speech interaction model based on the end-to-end model to obtain the answer speech signal.

[0050] That is, the semantics-intonation interaction compensation fusion representation value of the first user input speech syllable and the semantics-intonation interaction compensation fusion representation value of the second user input speech syllable are expressed as: .

[0051] in, is the number of zero eigenvalues ​​in the semantic-intonation interaction compensation fusion representation vector of the user input speech syllable, represents the logarithmic function with base 2, represents an exponential function with the natural constant e as the base, inputting a speech syllable semantic-intonation interactive compensation fusion representation value for the first user, A speech syllable semantics-intonation interaction compensation fusion representation value is input for the second user.

[0052] And the optimized user input speech syllable semantic-intonation interaction compensation fusion representation vector is expressed as: .

[0053] in, inputting a speech syllable semantic-intonation interactive compensation fusion representation value for the first user, The user inputs a speech syllable semantic-intonation interaction compensation fusion representation vector, and is a row vector, It means point multiplication by position. is the number of zero eigenvalues ​​in the semantic-intonation interaction compensation fusion representation vector of the user input speech syllable, To calculate each eigenvalue of the semantic-intonation interaction compensation fusion representation vector of the user input speech syllable, a power function with the inverse of the number of zero eigenvalues ​​as the exponent, A representation vector of the user input speech syllable semantic-intonation interaction compensation fusion information, represents the weight hyperparameter, inputting a speech syllable semantic-intonation interactive compensation fusion representation value for the second user, for The transposed vector of A speech syllable semantic-intonation interactive compensation fusion regression understanding matrix is ​​input for the user, represents matrix multiplication, The optimized user input speech syllable semantic-intonation interaction compensation fusion representation vector is provided.

[0054] Based on this, the present application realizes effective long-range modeling of the user input speech syllable semantic-intonation interaction compensation fusion representation vector under field linear complexity by modeling the vector field zero dimension of the user input speech syllable semantic-intonation interaction compensation fusion representation vector as a global field redundant dependency, so as to perform self-distillation of structured correlation between information representation and class understanding in a selective state space based on feature distribution in the feature regression process of the user input speech syllable semantic-intonation interaction compensation fusion representation vector, thereby maintaining the overall information complexity of the feature distribution of the user input speech syllable semantic-intonation interaction compensation fusion representation vector, while promoting the long-range dynamic regression understanding balance of its feature convergence, so as to improve the regression understanding consistency between the feature regression and feature extraction processes, and improve the signal quality of the answer speech signal obtained by inputting the user input speech syllable semantic-intonation interaction compensation fusion representation vector into the speech interaction model based on the end-to-end model. In this way, the emotional changes and intonation information in the speech can be captured more accurately, so that the system can better understand the user's intentions and emotions, thereby providing a more humane and natural response. At the same time, it can adapt to different language environments, reducing the need for a lot of manual tuning, so that the system can still maintain high recognition accuracy and generation quality when facing new vocabulary or background noise, thereby enhancing the system's adaptability and generalization capabilities.

[0055] In summary, an end-to-end speech large model interaction system 100 based on an embodiment of the present application is explained, which collects user input speech signals by sound sensors, and uses deep learning and end-to-end speech signal analysis and coding technology to perform speech signal segmentation and embedding fusion on the user input speech signals, and extracts intonation pattern features from the user input speech signals, thereby intelligently generating answering speech signals based on the fine-grained compensation fusion representation between the embedded fusion features of each user input speech syllable signal and the user input speech intonation features. In this way, the emotional changes and intonation information in the speech can be captured more accurately, so that the system can better understand the user's intentions and emotions, thereby providing a more humane and natural response. At the same time, it can adapt to different language environments, reducing the need for a large amount of manual tuning, so that the system can still maintain a high recognition accuracy and generation quality when facing new vocabulary or background noise, thereby enhancing the adaptability and generalization ability of the system.

[0056] As described above, the end-to-end speech large model interaction system 100 according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with an end-to-end speech intelligent interaction algorithm. In one possible implementation, the end-to-end speech large model interaction system 100 according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the end-to-end speech large model interaction system 100 can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the end-to-end speech large model interaction system 100 can also be one of the many hardware modules of the wireless terminal.

[0057] Alternatively, in another example, the end-to-end speech large model interaction system 100 and the wireless terminal may also be separate devices, and the end-to-end speech large model interaction system 100 may be connected to the wireless terminal via a wired and / or wireless network and transmit interaction information in accordance with an agreed data format.

[0058] The above descriptions of various implementations of the present disclosure are exemplary and not exhaustive. The disclosure is not limited to the disclosed implementations, and many modifications and variations are obvious to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The terms used herein are selected to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various practical methods disclosed herein.

Claims

1. An end-to-end speech large model interaction system, characterized in that: include: A voice signal acquisition module, used to acquire a user input voice signal collected by a sound sensor; A speech signal segmentation and embedding module, used for performing speech signal segmentation and embedding fusion on the user input speech signal to obtain the user input speech syllable signal embedding coding cascade feature; a speech intonation coding module, used for performing intonation pattern feature extraction on the user input speech signal to obtain the user input speech intonation coding feature; A speech feature compensation fusion module is used to perform speech feature fine-grained compensation fusion on the embedded coding cascade features of the user-input speech syllable signal and the user-input speech intonation coding features to obtain a user-input speech syllable semantic-intonation interactive compensation fusion representation, wherein the speech feature compensation fusion module includes: a speech common feature extraction unit, used to extract the common features between the embedded coding cascade features of the user-input speech syllable signal and the user-input speech intonation coding features to obtain a common feature representation between user-input speech features; a common feature guided fusion unit, used to perform common feature guided complementary fusion on the embedded coding cascade features of the user-input speech syllable signal and the user-input speech intonation coding features based on the common feature representation between the user-input speech features to obtain the user-input speech syllable semantic-intonation interactive compensation fusion representation; an answer speech signal generation module, used to obtain an answer speech signal based on the user-input speech syllable semantic-intonation interactive compensation fusion representation; Among them, the common feature guided fusion unit includes: a speech feature probabilistic subunit, which is used to input the user-input speech syllable signal embedding coding cascade vector, the user-input speech intonation coding feature vector and the common feature representation vector between the user-input speech features into a vector probabilistic unit based on the Sigmoid function to obtain the probabilistic user-input speech syllable signal embedding coding cascade vector, the probabilistic user-input speech intonation coding feature vector and the probabilistic user-input speech feature common feature representation vector; a speech syllable information compensation calculation subunit, which is used to calculate the user-input speech syllable information fine-grained compensation vector of the probabilistic user-input speech syllable signal embedding coding cascade vector relative to the probabilistic user-input speech feature common feature representation vector; a speech intonation information compensation calculation subunit, which is used to calculate the probabilistic user-input speech intonation coding feature vector relative to the probabilistic user-input speech feature common feature representation vector. A fine-grained compensation vector of user-input speech intonation information; a speech syllable feature compensation subunit, used to embed the fine-grained compensation vector of the user-input speech syllable information and the user-input speech syllable signal into a coding cascade vector input into a fine-grained compensation module to obtain a fine-grained feature compensation vector of the user-input speech syllable; a speech intonation feature compensation subunit, used to input the fine-grained compensation vector of the user-input speech intonation information and the user-input speech intonation coding feature vector into the fine-grained compensation module to obtain a fine-grained feature compensation vector of the user-input speech intonation; a speech syllable speech intonation compensation fusion subunit, used to cascade the user-input speech syllable fine-grained feature compensation vector, the user-input speech intonation fine-grained feature compensation vector and the common feature representation vector between the user-input speech features to obtain a user-input speech syllable semantic-intonation interaction compensation fusion representation vector as the user-input speech syllable semantic-intonation interaction compensation fusion representation.

2. The end-to-end speech large model interaction system according to claim 1, characterized in that: The speech signal segmentation and embedding module comprises: a speech signal segmentation unit, which is used to perform syllable-based speech signal segmentation on the user-input speech signal to obtain a sequence of user-input speech syllable signal segments; a speech signal embedding and coding unit, which is used to use a syllable signal embedding coding matrix to embed and code each user-input speech syllable signal segment in the sequence of user-input speech syllable signal segments to obtain a sequence of user-input speech syllable signal embedding coding vectors; and a speech signal cascading unit, which is used to cascade the sequence of user-input speech syllable signal embedding coding vectors to obtain a user-input speech syllable signal embedding coding cascade vector as the user-input speech syllable signal embedding coding cascade feature.

3. The end-to-end speech large model interaction system according to claim 2, characterized in that: The speech intonation encoding module is used to: input the user input speech signal into an intonation pattern feature extractor based on a dilated convolutional neural network model to obtain a user input speech intonation encoding feature vector as the user input speech intonation encoding feature.

4. The end-to-end speech large model interaction system according to claim 3, characterized in that: The speech common feature extraction unit is used to: embed the user input speech syllable signal into the encoding cascade vector and the user input speech intonation encoding feature vector into the common feature extraction network to obtain a common feature representation vector between user input speech features as the common feature representation between user input speech features.

5. The end-to-end speech large model interaction system according to claim 4, characterized in that: The speech commonality feature extraction unit is used to: embed the coding cascade vector of the user-input speech syllable signal and the coding feature vector of the user-input speech intonation, and perform position point addition to obtain the user-input speech syllable intonation point addition vector; multiply the user-input speech syllable intonation point addition vector with a weight matrix, and perform positional addition to the obtained user-input speech syllable intonation weight vector and a bias vector to obtain the user-input speech syllable intonation commonality vector; input the user-input speech syllable intonation commonality vector into a tanh function for processing to obtain the commonality feature representation vector between the user-input speech features.

6. The end-to-end speech large model interaction system according to claim 5, characterized in that: The speech syllable information compensation calculation subunit is used to: calculate the position point division of the probabilistic user input speech syllable signal embedding coding cascade vector and the common feature representation vector between the probabilistic user input speech features to obtain the user input speech syllable contribution vector; calculate the logarithmic function value with base two of the absolute value of each eigenvalue of the user input speech syllable contribution vector to obtain the user input speech syllable contribution logarithmic vector; calculate the position point multiplication of the probabilistic user input speech syllable signal embedding coding cascade vector and the user input speech syllable contribution logarithmic vector to obtain a point product vector, and calculate an exponential function with the natural constant e as the base, with each point product value in the point product vector as the exponent to obtain a user input speech syllable compensation vector; input the user input speech syllable compensation vector into the softmax function to obtain the user input speech syllable information fine-grained compensation vector.

7. The end-to-end speech large model interaction system according to claim 6, characterized in that: The speech syllable feature compensation subunit is used to calculate the point-by-point multiplication of the user-input speech syllable signal embedding coding cascade vector and the user-input speech syllable information fine-grained compensation vector to obtain the user-input speech syllable fine-grained feature compensation vector.

8. The end-to-end speech large model interaction system according to claim 7, characterized in that: The answering speech signal generating module is used to: input the semantic-intonation interaction compensation fusion representation vector of the user input speech syllable into a speech interaction model based on an end-to-end model to obtain the answering speech signal.

Images