Speech education robot based on children's speech modal characteristics perception

Abstract

The present application provides a speech education robot based on the perception of children's speech modal characteristics, which relates to the field of intelligent education. It uses AI and end-to-end speech analysis and processing technology to extract the time series of pitch and intensity from the speech input signal and perform a time series correlation pattern analysis on it. Then, the pitch time series correlation pattern characteristics and the intensity time series correlation pattern characteristics are compensated for feature commonality. At the same time, the speech input signal is segment embedded and encoded, so as to intelligently generate educational response speech through the compensated features and the semantic embedding features of each speech input signal segment. In this way, the robot can dynamically adjust the response content according to the real-time feedback and emotional changes of the children, and generate personalized educational response speech, which helps to enhance the interactive effect and improve children's enthusiasm for learning.

Description

Technical Field

[0001] The present application relates to the field of intelligent education, and more specifically, to a speech education robot based on the perception of children's speech modal characteristics. Background Art

[0002] With the continuous advancement of educational technology, personalized education has become a trend. Children show diverse interests, abilities and emotional states in the learning process. These individual differences require more flexible and targeted educational methods. Voice education robots can make children feel understood and cared for through natural and friendly interactions, thus stimulating their interest and enthusiasm in learning.

[0003] However, existing voice education robots usually use fixed reply templates, which lack flexibility and cannot be dynamically adjusted according to children's real-time feedback and emotional changes, thus affecting the interaction effect. For example, when a child shows confusion or disinterest, the robot cannot adjust the reply content in time to provide more targeted help. In addition, existing voice processing technology is insufficient in emotion recognition, and it mainly relies on text information, making it difficult to accurately capture and understand the changes in children's tone and emotional state. As a result, when robots interact with children, they cannot recognize and respond to children's emotional needs in time, affecting the naturalness and affinity of the interaction.

[0004] Therefore, an optimized voice education robot is desired. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a speech education robot based on the perception of children's speech modal characteristics.

[0006] According to one aspect of the present application, a speech education robot based on the perception of children's speech modal characteristics is provided, which includes: a microphone array for capturing a speech input signal of a target child object; a speech education processor for processing the speech input signal based on an end-to-end speech large model to generate an educational reply speech corresponding to the speech input signal; a speaker for playing the educational reply speech; wherein the speech education processor includes: a sound feature extraction module for extracting the sound features of the speech input signal to obtain a time queue of pitch and a time queue of sound intensity; a pitch and sound intensity modal perception module for the pitch The time queue of the sound intensity and the time queue of the sound intensity are used for speech modal perception to obtain pitch timing association pattern characteristics and sound intensity timing association pattern characteristics; a sound feature compensation interaction module is used to perform feature commonality compensation interaction on the pitch timing association pattern characteristics and the sound intensity timing association pattern characteristics to obtain a child speech modal characteristic representation; a speech signal embedding coding module is used to embed the speech input signal to obtain a sequence of semantic embedding features of speech input signal segments; a reply speech generation module is used to obtain the educational reply speech based on the child speech modal characteristic representation and the sequence of semantic embedding features of the speech input signal segments.

[0007] In the above-mentioned speech education robot based on children's speech modal characteristic perception, the pitch and intensity modal perception module is used to: input the time queue of the pitch and the time queue of the intensity into the speech modal sensor based on the forward LSTM model to obtain the pitch timing association pattern feature vector as the pitch timing association pattern feature and the intensity timing association pattern feature vector as the intensity timing association pattern feature.

[0008] In the above-mentioned speech education robot based on children's speech modal characteristic perception, the sound feature compensation interaction module includes: a pitch and intensity timing commonality extraction unit, which is used to input the pitch timing association pattern feature vector and the intensity timing association pattern feature vector into a commonality feature extraction network to obtain a commonality feature representation vector between pitch-intensity timing features; a pitch and intensity feature compensation unit, which is used to calculate the fine-grained compensation vectors of the pitch timing association pattern feature vector and the intensity timing association pattern feature vector respectively based on the commonality feature representation vector between pitch-intensity timing features to obtain a pitch timing fine-grained compensation vector and an intensity timing fine-grained compensation vector; a pitch and intensity feature compensation fusion unit, which is used to perform fine-grained feature compensation fusion on the pitch timing association pattern feature vector and the intensity timing association pattern feature vector based on the pitch timing fine-grained compensation vector and the intensity timing fine-grained compensation vector to obtain the children's speech modal characteristic representation.

[0009] In the above-mentioned speech education robot based on children's speech modal characteristic perception, the pitch-intensity timing commonality extraction unit is used to: add the pitch timing association pattern feature vector and the intensity timing association pattern feature vector according to the position point to obtain the pitch-intensity timing point addition vector; multiply the pitch-intensity timing point addition vector with the weight matrix, and add the obtained pitch-intensity timing weight vector and the bias vector according to the position to obtain the pitch-intensity timing commonality vector; input the pitch-intensity timing commonality vector into the tanh function for processing to obtain the commonality feature representation vector between the pitch-intensity timing features.

[0010] In the above-mentioned speech education robot based on children's speech modal characteristic perception, the pitch and intensity feature compensation unit includes: a feature vector probabilistic processing subunit, which is used to input the pitch timing association pattern feature vector, the intensity timing association pattern feature vector and the common feature representation vector between the pitch and intensity timing features into a vector probabilistic unit based on the Sigmoid function to obtain the probabilistic pitch timing association pattern feature vector, the probabilistic intensity timing association pattern feature vector and the probabilistic pitch and intensity timing feature common feature representation vector; a pitch feature timing compensation subunit, which is used to calculate the pitch timing fine-grained compensation vector of the probabilistic pitch timing association pattern feature vector relative to the probabilistic pitch and intensity timing feature common feature representation vector; and an intensity timing fine-grained compensation vector generation subunit, which is used to calculate the intensity timing fine-grained compensation vector of the probabilistic intensity timing association pattern feature vector relative to the probabilistic pitch and intensity timing feature common feature representation vector.

[0011] In the above-mentioned speech education robot based on children's speech modal characteristic perception, the pitch feature timing compensation subunit is used to: calculate the position point division of the common feature representation vector between the probabilistic pitch timing association pattern feature vector and the probabilistic pitch-intensity timing feature to obtain a pitch timing contribution vector; calculate the logarithmic function value with base two of the absolute value of each eigenvalue of the pitch timing contribution vector to obtain a pitch timing contribution logarithmic vector; calculate the position point multiplication of the probabilistic pitch timing association pattern feature vector and the pitch timing contribution logarithmic vector to obtain a point product vector, and calculate the exponential function value of each point product value in the point product vector with the natural constant e as the base to obtain a pitch timing compensation vector; input the pitch timing compensation vector into the softmax function to obtain the pitch timing fine-grained compensation vector.

[0012] In the above-mentioned speech education robot based on children's speech modal characteristic perception, the pitch and intensity feature compensation fusion unit includes: a pitch feature timing fine-grained compensation subunit, which is used to input the pitch timing fine-grained compensation vector and the pitch timing association pattern feature vector into the fine-grained compensation module to obtain the pitch timing fine-grained feature compensation vector; an intensity feature timing fine-grained compensation subunit, which is used to input the intensity timing fine-grained compensation vector and the intensity timing association pattern feature vector into the fine-grained compensation module to obtain the intensity timing fine-grained feature compensation vector; a children's speech modal characteristic representation generation subunit, which is used to cascade the pitch timing fine-grained feature compensation vector, the intensity timing fine-grained feature compensation vector and the common feature representation vector between the pitch-intensity timing features to obtain a children's speech modal characteristic representation vector as the children's speech modal characteristic representation.

[0013] In the above-mentioned speech education robot based on children's speech modal characteristic perception, the pitch feature timing fine-grained compensation subunit is used to: calculate the point-by-point multiplication of the pitch timing association pattern feature vector and the pitch timing fine-grained compensation vector to obtain the pitch timing fine-grained feature compensation vector.

[0014] In the above-mentioned speech education robot based on children's speech modal characteristic perception, the speech signal embedding coding module is used to: use the speech signal embedding matrix to embed the speech input signal to obtain a sequence of speech input signal segment semantic embedding coding vectors as a sequence of speech input signal segment semantic embedding features.

[0015] In the above-mentioned speech education robot based on children's speech modal characteristic perception, the reply speech generation module is used to: use the children's speech modal characteristic representation vector as prompt information, add the children's speech modal characteristic representation vector to the tail of the sequence of semantic embedding coding vectors of the speech input signal segment, and then input it into the end-to-end speech large model to obtain the educational reply speech.

[0016] Compared with the prior art, the speech education robot based on the perception of children's speech modal characteristics provided by the present application uses AI-based and end-to-end speech analysis and processing technology to extract the time series of pitch and intensity from the speech input signal and perform time series correlation pattern analysis on it. Then, the pitch time series correlation pattern features and the intensity time series correlation pattern features are compensated for feature commonality. At the same time, the speech input signal is segment embedded and encoded, so as to intelligently generate educational response speech through the compensated features and the semantic embedding features of each speech input signal segment. In this way, the robot can dynamically adjust the response content according to the real-time feedback and emotional changes of the children, and generate personalized educational response speech, which helps to enhance the interactive effect and improve children's enthusiasm for learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] In the attached picture: Figure 1 This is a block diagram of a speech education processor in a speech education robot based on the perception of children's speech modal characteristics according to an embodiment of the present application.

[0019] Figure 2 This is a data flow diagram of a speech education processor in a speech education robot based on perception of children's speech modal characteristics according to an embodiment of the present application.

[0020] Figure 3 It is a block diagram of a sound feature compensation interaction module in a speech education robot based on children's speech modal characteristic perception according to an embodiment of the present application.

[0021] Figure 4 This is a block diagram of a high-pitched and high-intensity feature compensation unit of a speech education robot based on the perception of children's speech modal characteristics according to an embodiment of the present application.

[0022] Figure 5 This is a block diagram of a high-pitched and high-intensity feature compensation fusion unit in a speech education robot based on the perception of children's speech modal characteristics according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0024] With the development of educational technology, personalized education is becoming more and more important. Children show different interests, abilities and emotional states in the process of learning and growing up, which requires that educational methods must be more flexible and personalized. Voice education robots can stimulate children's interest and enthusiasm in learning by simulating natural and friendly communication methods. However, most of the current voice education robots rely on fixed reply templates and lack the ability to dynamically adjust according to children's real-time feedback and emotional changes, which limits the effectiveness of interaction. For example, when children show confusion or disinterest, robots often cannot adjust their reply content in time to provide more targeted help. In addition, existing voice processing technology has limitations in emotion recognition, mainly relying on text information, and it is difficult to accurately capture and understand children's tone and emotional state changes. This results in the robot being unable to recognize and respond to children's emotional needs in a timely manner when interacting with children, affecting the naturalness and affinity of the interaction.

[0025] Based on this, the present application proposes a speech education robot based on the perception of children's speech modal characteristics, which includes: a microphone array for capturing the speech input signal of the target child object; a speech education processor for processing the speech input signal based on an end-to-end speech large model to generate an educational response speech corresponding to the speech input signal; and a speaker for playing the educational response speech.

[0026] Specifically, the microphone array is used to capture the voice input signal of the target child object. It should be understood that the voice input signal of the child object contains multiple aspects of information, such as the semantic information, tone, pitch and emotional information expressed by the child, which is crucial for the robot to understand the specific needs and interests of the child and generate personalized educational response voice. The microphone array is composed of multiple microphones, which can provide high-fidelity, multi-directional voice collection capabilities and support long-distance sound pickup and noise reduction processing.

[0027] Specifically, the speech education processor is used to process the speech input signal based on the end-to-end speech large model to generate an educational reply speech corresponding to the speech input signal. Accordingly, in the speech education processor, the technical concept of the present application is to use AI and end-to-end speech analysis and processing technology to extract the time series of pitch and intensity from the speech input signal and perform a timing association pattern analysis on it, then, perform feature commonality compensation on the pitch timing association pattern features and the intensity timing association pattern features, and at the same time, perform segment embedding coding on the speech input signal, so as to intelligently generate the educational reply speech through the compensated features and the semantic embedding features of each speech input signal segment. In this way, the robot can understand the voice features and emotional state of the child in real time and generate personalized educational reply speech. And it can dynamically adjust the reply content according to the child's real-time feedback and emotional changes, provide more targeted help, make the interaction more natural and friendly, thereby enhancing the child's emotional connection and learning motivation.

[0028] Figure 1 This is a block diagram of a speech education processor in a speech education robot based on the perception of children's speech modal characteristics according to an embodiment of the present application. Figure 2 Schematic diagram of data flow of a speech education processor in a speech education robot based on perception of children's speech modal characteristics according to an embodiment of the present application. Figure 1 and Figure 2 As shown, in the speech education processor 100, it includes: a sound feature extraction module 110, which is used to extract the sound characteristics of the speech input signal to obtain the time queue of pitch and the time queue of sound intensity; a pitch and sound intensity modal perception module 120, which is used to perform speech modal perception on the time queue of pitch and the time queue of sound intensity to obtain pitch timing association pattern characteristics and sound intensity timing association pattern characteristics; a sound feature compensation interaction module 130, which is used to perform feature commonality compensation interaction on the pitch timing association pattern characteristics and the sound intensity timing association pattern characteristics to obtain children's speech modal characteristic representation; a speech signal embedding coding module 140, which is used to embed the speech input signal to obtain a sequence of semantic embedding features of speech input signal segments; a reply speech generation module 150, which is used to obtain the educational reply speech based on the children's speech modal characteristic representation and the sequence of semantic embedding features of the speech input signal segments.

[0029] In an embodiment of the present application, the sound feature extraction module 110 is used to extract the sound characteristics of the voice input signal to obtain the time queue of pitch and the time queue of sound intensity. Accordingly, it is considered that the voice input signal contains multiple aspects of information, such as fundamental frequency (pitch), energy (sound intensity, loudness), spectral characteristics, etc. The change in pitch can reflect the emotional state of the speaker. For example, high notes are usually associated with excitement or tension, while low notes may be associated with sadness or calmness; changes in sound intensity can also reflect emotions. For example, speaking loudly may indicate anger or excitement, while whispering may indicate gentleness or shyness. Based on this, in order to better understand the pronunciation characteristics and emotional expression of children, in the technical solution of the present application, the voice input signal is subjected to sound feature extraction to obtain the time queue of pitch and the time queue of sound intensity. In particular, in a specific embodiment of the present application, the sound characteristics of the voice input signal are extracted to obtain the time queue of pitch and the time queue of sound intensity, including: a. Voice signal preprocessing: Use an appropriate audio reading function (such as the audioread function in MATLAB) to read the voice signal collected by the microphone, and the reading result usually includes an audio digital signal and a sampling frequency. And the read voice signal is framed according to a fixed length (such as 20-50 milliseconds), and the framing can divide the voice signal into smaller processing units to facilitate subsequent feature extraction; b. Pitch time series data extraction: First, determine the valid voice segment in the voice signal through an endpoint detection algorithm, and remove the silence and noise parts in the original voice signal. Then use an appropriate extraction algorithm (such as a short-time autocorrelation function method, a cepstrum analysis method, etc.) to calculate the pitch of each frame of the voice signal, and arrange the pitch values ​​of each frame of the voice signal in chronological order to construct a time queue of the pitch. c. Sound intensity time series data extraction: For each frame of speech signal, directly calculate its root mean square value (reflecting the average energy level of the signal) as the sound intensity value of the frame, and arrange the sound intensity values ​​of each frame of speech signal in chronological order to construct a time queue of sound intensity.

[0030] In an embodiment of the present application, the pitch and intensity modal perception module 120 is used to perform speech modal perception on the time queue of the pitch and the time queue of the intensity to obtain the pitch temporal association pattern characteristics and the intensity temporal association pattern characteristics. Specifically, in an embodiment of the present application, the pitch and intensity modal perception module is used to: input the time queue of the pitch and the time queue of the intensity into the speech modal perception based on the forward LSTM model to obtain the pitch temporal association pattern feature vector as the pitch temporal association pattern feature and the intensity temporal association pattern feature vector as the intensity temporal association pattern feature. It should be understood that considering that the time queue of the pitch and the time queue of the intensity both change with time, and there are temporal dynamic change characteristics and temporal association relationships between different time scales, the LSTM model controls the flow of information by introducing a gating mechanism (input gate, forget gate and output gate), which can remember the long-term dependencies between different positions in the input data and forget unnecessary information. Therefore, in order to capture and analyze the temporal dynamic change characteristics of pitch and intensity in speech signals and the temporal correlation between different time spans, in the technical solution of the present application, the time queue of the pitch and the time queue of the intensity are input into a speech modal sensor based on a forward LSTM model to obtain a pitch temporal correlation pattern feature vector and an intensity temporal correlation pattern feature vector.

[0031] In the embodiment of the present application, the sound feature compensation interaction module 130 is used to perform inter-feature commonality compensation interaction on the pitch temporal association pattern feature and the intensity temporal association pattern feature to obtain a children's speech modal characteristic representation. Specifically, Figure 3 FIG. 1 is a block diagram of a sound feature compensation interaction module in a speech education robot based on children's speech modal feature perception according to an embodiment of the present application. Figure 3 As shown, the sound feature compensation interaction module 130 includes: a pitch-intensity timing commonality extraction unit 131, which is used to input the pitch timing association pattern feature vector and the intensity timing association pattern feature vector into a commonality feature extraction network to obtain a commonality feature representation vector between pitch-intensity timing features; a pitch-intensity feature compensation unit 132, which is used to calculate the fine-grained compensation vectors of the pitch timing association pattern feature vector and the intensity timing association pattern feature vector respectively based on the commonality feature representation vector between pitch-intensity timing features to obtain a pitch timing fine-grained compensation vector and an intensity timing fine-grained compensation vector; a pitch-intensity feature compensation fusion unit 133, which is used to perform fine-grained feature compensation fusion on the pitch timing association pattern feature vector and the intensity timing association pattern feature vector based on the pitch timing fine-grained compensation vector and the intensity timing fine-grained compensation vector to obtain the children's speech modal characteristic representation.

[0032] It should be understood that, considering that pitch and intensity are two important features in speech signals, they each carry different information. The pitch timing association pattern feature mainly reflects the change of the tone of the speech over time, while the intensity timing association pattern feature mainly reflects the change trend of the loudness of the speech over time. Moreover, there are commonalities between the two in terms of timing, such as when a person is excited or angry, the pitch and intensity tend to increase at the same time. Therefore, in order to ensure that the common information in the two feature vectors is retained, while enhancing their respective unique information to generate a more comprehensive and rich feature representation, in the technical solution of the present application, the pitch timing association pattern feature and the intensity timing association pattern feature are subjected to inter-feature commonality compensation interaction to obtain a children's speech modal characteristic representation vector as a children's speech modal characteristic representation.

[0033] Specifically, first, the pitch temporal correlation pattern feature vector and the intensity temporal correlation pattern feature vector are input into a common feature extraction network to capture the temporal changes and mutual relationships of pitch and intensity. These common patterns may reflect the emotional expression and speech rhythm in the speech signal, and obtain the common feature representation vector between the pitch-intensity temporal features. In particular, the essence of the network is a neural network, for example, a fully connected layer is used to extract the common information between the two, thereby improving the accuracy of subsequent speech processing tasks.

[0034] Then, based on the common feature representation vector between the pitch-intensity timing features, the fine-grained compensation vectors of the pitch timing association pattern feature vector and the intensity timing association pattern feature vector are calculated respectively to obtain the pitch timing fine-grained compensation vector and the intensity timing fine-grained compensation vector. Specifically, first, in order to convert the feature vector into a probabilistic form so as to better adjust and fuse the feature weights, a vector probabilization unit based on the Sigmoid function is used to probabilize the pitch timing association pattern feature vector, the intensity timing association pattern feature vector and the common feature representation vector between the pitch-intensity timing features to map the feature values ​​to between 0 and 1, so that the comparison and fusion between different feature vectors are more consistent and stable. Afterwards, the compensation vectors of the probabilistic pitch timing association pattern feature vector and the probabilistic intensity timing association pattern feature vector relative to the common feature representation vector between the probabilistic pitch-intensity timing features are calculated respectively, so as to highlight the unique and key feature information in the pitch timing and intensity features. This information may not be included or not significant enough in the common features, but is crucial for the analysis and processing of the model, and the fine-grained pitch timing compensation vector and the fine-grained intensity timing compensation vector are obtained.

[0035] Next, based on the pitch timing fine-grained compensation vector and the intensity timing fine-grained compensation vector, the pitch timing association pattern feature vector and the intensity timing association pattern feature vector are subjected to fine-grained feature compensation fusion to obtain the children's speech modal characteristic representation. Specifically, in order to capture some subtle changes in pitch features, which may be ignored in the original pitch timing association pattern feature vector, the pitch timing fine-grained compensation vector and the pitch timing association pattern feature vector are input into the fine-grained compensation module, so that these subtle changes can be integrated into the final feature representation, making the features richer and more detailed. In particular, here the compensation vector is added to the pitch timing association pattern feature vector by calculating the positional dot product between the two vectors, so as to highlight the key information of the pitch timing association pattern feature. Similarly, the intensity timing fine-grained compensation vector and the intensity timing association pattern feature vector are input into the fine-grained compensation module to highlight the key features of the intensity by calculating the dot product between the two vectors, and the intensity timing fine-grained feature compensation vector is obtained. Finally, the pitch timing fine-grained feature compensation vector, the intensity timing fine-grained feature compensation vector and the common feature representation vector between pitch-intensity timing features are cascaded to obtain a comprehensive and comprehensive children's speech modal characteristic representation vector, so as to better capture and represent the modal characteristics of children's speech.

[0036] Specifically, in an embodiment of the present application, the pitch-intensity timing commonality extraction unit is used to: add the pitch timing association pattern feature vector and the intensity timing association pattern feature vector according to position points to obtain a pitch-intensity timing point addition vector; multiply the pitch-intensity timing point addition vector with a weight matrix, and add the obtained pitch-intensity timing weight vector and the bias vector according to position to obtain a pitch-intensity timing commonality vector; input the pitch-intensity timing commonality vector into a tanh function for processing to obtain a commonality feature representation vector between the pitch-intensity timing features.

[0037] Specifically, Figure 4 FIG. 1 is a block diagram of a high pitch and high intensity characteristic compensation unit of a speech education robot based on perception of children's speech modal characteristics according to an embodiment of the present application. Figure 4As shown, the pitch and intensity feature compensation unit 132 includes: a feature vector probabilistic processing subunit 1321, which is used to input the pitch timing association pattern feature vector, the intensity timing association pattern feature vector and the common feature representation vector between the pitch and intensity timing features into a vector probabilistic unit based on the Sigmoid function to obtain a probabilistic pitch timing association pattern feature vector, a probabilistic intensity timing association pattern feature vector and a probabilistic pitch and intensity timing feature common feature representation vector; a pitch feature timing compensation subunit 1322, which is used to calculate the pitch timing fine-grained compensation vector of the probabilistic pitch timing association pattern feature vector relative to the probabilistic pitch and intensity timing feature common feature representation vector; and an intensity timing fine-grained compensation vector generation subunit 1323, which is used to calculate the intensity timing fine-grained compensation vector of the probabilistic intensity timing association pattern feature vector relative to the probabilistic pitch and intensity timing feature common feature representation vector.

[0038] More specifically, in an embodiment of the present application, the pitch feature timing compensation subunit is used to: calculate the point-by-point division of the common feature representation vector between the probabilistic pitch timing association pattern feature vector and the probabilistic pitch-intensity timing feature to obtain a pitch timing contribution vector; calculate the logarithmic function value with base two of the absolute value of each eigenvalue of the pitch timing contribution vector to obtain a pitch timing contribution logarithmic vector; calculate the point-by-point multiplication of the probabilistic pitch timing association pattern feature vector and the pitch timing contribution logarithmic vector to obtain a dot product vector, and calculate the exponential function value of each dot product value in the dot product vector with the natural constant e as the base to obtain a pitch timing compensation vector; input the pitch timing compensation vector into a softmax function to obtain the pitch timing fine-grained compensation vector.

[0039] Specifically, Figure 5 : is a block diagram of a high pitch and high intensity feature compensation fusion unit in a speech education robot based on children's speech modal feature perception according to an embodiment of the present application. Figure 5As shown, the pitch and intensity feature compensation fusion unit 133 includes: a pitch feature timing fine-grained compensation subunit 1331, which is used to input the pitch timing fine-grained compensation vector and the pitch timing association pattern feature vector into the fine-grained compensation module to obtain the pitch timing fine-grained feature compensation vector; an intensity feature timing fine-grained compensation subunit 1332, which is used to input the intensity timing fine-grained compensation vector and the intensity timing association pattern feature vector into the fine-grained compensation module to obtain the intensity timing fine-grained feature compensation vector; a child speech modal characteristic representation generation subunit 1333, which is used to cascade the pitch timing fine-grained feature compensation vector, the intensity timing fine-grained feature compensation vector and the common feature representation vector between the pitch-intensity timing features to obtain a child speech modal characteristic representation vector as the child speech modal characteristic representation.

[0040] More specifically, in an embodiment of the present application, the pitch feature timing fine-grained compensation subunit is used to: calculate the point-by-point multiplication of the pitch timing association pattern feature vector and the pitch timing fine-grained compensation vector to obtain the pitch timing fine-grained feature compensation vector.

[0041] In the embodiment of the present application, specifically, the sound feature compensation interaction module is used to: perform feature commonality compensation interaction on the pitch timing association pattern feature vector and the intensity timing association pattern feature vector according to the following formula; wherein the formula is: .

[0042] in, and are respectively the pitch temporal association pattern feature vector and the intensity temporal association pattern feature vector, To add by location, and are the weight matrix and bias vector respectively, yes function, is the common feature representation vector between pitch-intensity temporal features, yes function, is the probabilistic pitch temporal correlation pattern feature vector, is the probabilistic sound intensity temporal correlation pattern feature vector, is the common feature representation vector between the probabilistic pitch-intensity temporal features, is the eigenvalue of each position in the probabilistic pitch timing association pattern eigenvector, is the eigenvalue of each position in the probabilistic sound intensity temporal association pattern eigenvector, is the feature value of each position in the common feature representation vector between the probabilistic pitch-intensity time series features, represents the logarithmic function value with base 2, represents the exponential function value with the natural constant e as the base, yes function, is the pitch timing fine-grained compensation vector, is the fine-grained compensation vector of the sound intensity timing, It is the point multiplication by position. is the pitch temporal fine-grained feature compensation vector, is the sound intensity temporal fine-grained feature compensation vector, It is a cascade operation on vectors. is the representation vector of the child's speech modal characteristics.

[0043] In the embodiment of the present application, the speech signal embedding coding module 140 is used to embed the speech input signal to obtain a sequence of semantic embedding features of the speech input signal segments. Specifically, in the embodiment of the present application, the speech signal embedding coding module is used to: embed the speech input signal using the speech signal embedding matrix to obtain a sequence of semantic embedding coding vectors of the speech input signal segments as the sequence of semantic embedding features of the speech input signal segments. It should be understood that the speech input signal also contains semantic information about the speech. Based on this, in order to convert the speech signal into a vector representation with semantic information, so as to better capture and represent the meaning of the speech content, in the technical solution of the present application, the speech input signal is embedded and coded to obtain a sequence of semantic embedding coding vectors of the speech input signal segments. It should be noted that the speech signal embedding matrix is ​​a mapping matrix in a high-dimensional space, which maps the original speech input signal (usually continuous audio data) to this high-dimensional space to obtain a sequence of semantic embedding coding vectors of the speech input signal segments containing rich semantic information. These vectors can capture the words and phrases in the speech signal and the contextual relationships between them, which helps the robot to understand the user's intentions more accurately, thereby providing real-time feedback.

[0044] In an embodiment of the present application, the reply speech generation module 150 is used to obtain the educational reply speech based on the child speech modal characteristic representation and the sequence of semantic embedding features of the speech input signal segment. Specifically, in an embodiment of the present application, the reply speech generation module is used to: use the child speech modal characteristic representation vector as prompt information, add the child speech modal characteristic representation vector to the end of the sequence of semantic embedding coding vectors of the speech input signal segment, and then input it into the end-to-end speech large model to obtain the educational reply speech. It should be understood that the child speech modal characteristic representation vector contains comprehensive information of pitch, sound intensity and their common characteristics, which can reflect the child's emotional state, tone and intention. The sequence of semantic embedding coding vectors of the speech input signal segment is the semantic information of the speech input signal at different time segments. Therefore, in order to generate a more personalized and context-related educational reply voice, in the technical solution of the present application, the child's voice modal characteristic representation vector is used as prompt information, and after the child's voice modal characteristic representation vector is added to the tail of the sequence of the semantic embedding coding vector of the voice input signal segment, it is input into the end-to-end voice large model to obtain the educational reply voice. In this way, by considering the voice modal characteristics of the child, the model can generate a reply that matches the child's emotional state, making the interaction more natural and cordial. In this way, the robot can understand the child's voice characteristics and emotional state in real time and generate personalized educational reply voice. And it can dynamically adjust the reply content according to the child's real-time feedback and emotional changes, provide more targeted help, make the interaction more natural and cordial, thereby enhancing the child's emotional connection and learning motivation.

[0045] It should be noted that the end-to-end speech model is a comprehensive technology that not only can perform speech recognition, but also integrates multiple links such as natural language processing, intent recognition, dialogue management, and speech synthesis. It can realize the complete interactive process from speech input to speech feedback, which is conducive to enriching the depth and breadth of human-computer interaction. In particular, in a specific embodiment of the present application, after adding the child's speech modal characteristic representation vector to the tail of the sequence of the semantic embedding coding vector of the speech input signal segment, it is input into the end-to-end speech model to obtain the educational reply speech, including: a. Model training: Collect a large amount of children's speech data, including speech input signals of children of different ages, different genders, and different speech characteristics, and collect the educational reply speech corresponding to these speech input signals as training data and test data. And use the preprocessed speech data and the corresponding educational reply speech to train the end-to-end speech model. b. Data preparation: Add the child's speech modal feature representation vector to the end of the sequence of semantic embedding coding vectors of the speech input signal segment to obtain the input data of the end-to-end speech model; c. Output and feedback: Input the concatenated vector into the trained end-to-end speech model, which will generate educational response speech corresponding to the child's speech input based on the pattern learned during training.

[0046] Preferably, when the children's speech modal characteristic representation vector represents the short-range and long-range bidirectional temporal correlation characteristics of the pitch and intensity of the speech input signal with a common feature pattern as a compensatory interaction feature masked, and the sequence of the speech input signal segment semantic embedding coding vectors represents the coding sequence characteristics of the speech input signal, the input feature vector input into the end-to-end speech model will also have the semantic feature representation of the speech input signal and the multi-dimensional expression diversity of the source semantics. Therefore, when it is input into the end-to-end speech model for generative regression, it is expected to improve its generative probabilistic reasoning regression effect.

[0047] Based on this, the present application optimizes the input feature vector obtained by adding the child speech modal characteristic representation vector to the tail of the sequence of the semantic embedding coding vector of the speech input signal segment, specifically including the steps of: determining the feature mean and feature variance of the input feature vector, and dividing the feature mean by the feature variance to obtain a probability feature value: performing point multiplication of the input feature vector and the inverse of the maximum feature value in the input feature vector to obtain a probability constraint vector: performing point addition of the probability constraint vector and the probability feature value, calculating the logarithm with base 2 to obtain an information interaction vector: performing point subtraction of the probability constraint vector with the probability feature value, calculating the inverse of each feature value to obtain a sequence constraint vector: performing point addition of the information interaction vector and the sequence constraint vector to obtain an optimized input feature vector. Then, the optimized input feature vector is input into the end-to-end speech large model to obtain the educational reply speech.

[0048] Here, the optimization of the input feature vector is expressed as: .

[0049] in, is the input feature vector, is the feature mean of the input feature vector, is the characteristic variance of the input feature vector, is the probability eigenvalue, represents the inverse of the largest eigenvalue in the input eigenvector, It means point multiplication by position. is the probability constraint vector, represents the logarithmic function with base 2, Indicates adding by position point. is the information interaction vector, It means to subtract by position point. is the sequence constraint vector, is the input feature vector for the optimization.

[0050] That is, considering that in the generative regression process, when the regression weight matrix acts on the input feature vector, the distribution diversity of the input feature vector will lead to the uncertainty of the weight stimulus parameters of the regression weight matrix, resulting in the lack of posterior probability density inference of the input feature vector through the weight matrix, affecting the probability reasoning regression effect of the generative regression.

[0051] In this case, the probability statistical characteristics of the input feature vector are used to simulate the mesoscale interaction structure under probability constraints between the eigenvalue scale and the eigenvector scale of the input feature vector, so as to construct a bidirectional latent variable motif based on the mesoscale short sequence relative to the probability statistical value to perform mesoscale bidirectional migration, and to perform a posteriori recovery based on the sequence constraints on the interactive information, thereby improving the convergence effect in the probability density domain and improving the voice quality of the educational reply speech.

[0052] Specifically, the speaker is used to play the educational reply voice. It should be understood that the speaker is an important component in the voice education robot, and its main function is to convert the generated educational reply voice into audible sound so that children can hear and understand the robot's reply, and through the high-fidelity speaker, the robot can express emotions more naturally, making the interaction more intimate and real.

[0053] In summary, the speech education robot 100 based on the perception of children's speech modal characteristics according to the embodiment of the present application is explained, which uses AI-based and end-to-end speech analysis and processing technology to extract the time series of pitch and intensity from the speech input signal and perform a time series correlation pattern analysis on it, and then compensates for the commonality of the pitch time series correlation pattern features and the intensity time series correlation pattern features, and at the same time, performs segment embedding coding on the speech input signal, so as to intelligently generate educational response speech through the compensated features and the semantic embedding features of each speech input signal segment. In this way, the robot can dynamically adjust the response content according to the real-time feedback and emotional changes of the child, and generate personalized educational response speech, which helps to enhance the interactive effect and improve children's enthusiasm for learning.

[0054] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details of the above application are only for the purpose of illustration and ease of understanding, rather than limitation, and the above details do not limit the present application to the use of the above specific details to be implemented.

Claims

1. A speech education robot based on the perception of children's speech modal characteristics, characterized in that: include: A microphone array for capturing a speech input signal of a target child subject; A speech education processor, for processing the speech input signal based on an end-to-end speech big model to generate an education reply speech corresponding to the speech input signal; A loudspeaker for playing the educational reply voice; wherein the voice education processor comprises: a sound feature extraction module for performing sound feature extraction on the voice input signal to obtain a time queue of pitch and a time queue of sound intensity; a pitch and sound intensity modal perception module for performing voice modal perception on the time queue of pitch and the time queue of sound intensity to obtain pitch timing association pattern features and sound intensity timing association pattern features; a sound feature compensation interaction module for performing feature commonality compensation interaction on the pitch timing association pattern features and the sound intensity timing association pattern features to obtain a children's voice modal feature representation; a voice signal embedding coding module for embedding coding the voice input signal to obtain a sequence of semantic embedding features of voice input signal segments; a reply voice generation module for obtaining the educational reply voice based on the children's voice modal feature representation and the sequence of semantic embedding features of the voice input signal segments; Among them, the sound feature compensation interaction module includes: a pitch and intensity timing commonality extraction unit, which is used to input the pitch timing association pattern feature vector and the intensity timing association pattern feature vector into a commonality feature extraction network to obtain a commonality feature representation vector between pitch-intensity timing features; a pitch and intensity feature compensation unit, which is used to calculate the fine-grained compensation vectors of the pitch timing association pattern feature vector and the intensity timing association pattern feature vector respectively based on the commonality feature representation vector between pitch-intensity timing features to obtain a pitch timing fine-grained compensation vector and an intensity timing fine-grained compensation vector; a pitch and intensity feature compensation fusion unit, which is used to perform fine-grained feature compensation fusion on the pitch timing association pattern feature vector and the intensity timing association pattern feature vector based on the pitch timing fine-grained compensation vector and the intensity timing fine-grained compensation vector to obtain the children's speech modal characteristic representation.

2. The speech education robot based on children's speech modal characteristic perception according to claim 1 is characterized in that: The pitch and intensity modal perception module is used to: input the time queue of the pitch and the time queue of the intensity into a speech modal perception device based on a forward LSTM model to obtain a pitch timing association pattern feature vector as the pitch timing association pattern feature and an intensity timing association pattern feature vector as the intensity timing association pattern feature.

3. The speech education robot based on children's speech modal characteristic perception according to claim 2 is characterized in that: The pitch-intensity timing commonality extraction unit is used to: add the pitch timing association pattern feature vector and the intensity timing association pattern feature vector according to position points to obtain a pitch-intensity timing point addition vector; multiply the pitch-intensity timing point addition vector with a weight matrix, and add the obtained pitch-intensity timing weight vector and bias vector according to position to obtain a pitch-intensity timing commonality vector; input the pitch-intensity timing commonality vector into a tanh function for processing to obtain a commonality feature representation vector between the pitch-intensity timing features.

4. The speech education robot based on children's speech modal characteristic perception according to claim 3 is characterized in that: The pitch and intensity feature compensation unit includes: a feature vector probabilistic processing subunit, which is used to input the pitch timing association pattern feature vector, the intensity timing association pattern feature vector and the common feature representation vector between the pitch and intensity timing features into a vector probabilistic unit based on the Sigmoid function to obtain a probabilistic pitch timing association pattern feature vector, a probabilistic intensity timing association pattern feature vector and a probabilistic pitch and intensity timing feature common feature representation vector; a pitch feature timing compensation subunit, which is used to calculate the pitch timing fine-grained compensation vector of the probabilistic pitch timing association pattern feature vector relative to the probabilistic pitch and intensity timing feature common feature representation vector; and an intensity timing fine-grained compensation vector generation subunit, which is used to calculate the intensity timing fine-grained compensation vector of the probabilistic intensity timing association pattern feature vector relative to the probabilistic pitch and intensity timing feature common feature representation vector.

5. The speech education robot based on children's speech modal characteristic perception according to claim 4 is characterized in that: The pitch feature timing compensation subunit is used to: calculate the point-by-point division of the common feature representation vector between the probabilistic pitch timing association pattern feature vector and the probabilistic pitch-intensity timing feature to obtain a pitch timing contribution vector; calculate the logarithmic function value with base two of the absolute value of each eigenvalue of the pitch timing contribution vector to obtain a pitch timing contribution logarithmic vector; calculate the point-by-point multiplication of the probabilistic pitch timing association pattern feature vector and the pitch timing contribution logarithmic vector to obtain a point product vector, and calculate the exponential function value of each point product value in the point product vector with base e as the natural constant to obtain a pitch timing compensation vector; input the pitch timing compensation vector into a softmax function to obtain the pitch timing fine-grained compensation vector.

6. The speech education robot based on children's speech modal characteristic perception according to claim 5, characterized in that: The pitch and intensity feature compensation fusion unit includes: a pitch feature timing fine-grained compensation subunit, which is used to input the pitch timing fine-grained compensation vector and the pitch timing association pattern feature vector into a fine-grained compensation module to obtain a pitch timing fine-grained feature compensation vector; an intensity feature timing fine-grained compensation subunit, which is used to input the intensity timing fine-grained compensation vector and the intensity timing association pattern feature vector into the fine-grained compensation module to obtain an intensity timing fine-grained feature compensation vector; and a children's speech modal characteristic representation generation subunit, which is used to cascade the pitch timing fine-grained feature compensation vector, the intensity timing fine-grained feature compensation vector and the common feature representation vector between the pitch-intensity timing features to obtain a children's speech modal characteristic representation vector as the children's speech modal characteristic representation.

7. The speech education robot based on children's speech modal characteristic perception according to claim 6, characterized in that: The pitch feature timing fine-grained compensation subunit is used to calculate the point-by-point multiplication of the pitch timing association pattern feature vector and the pitch timing fine-grained compensation vector to obtain the pitch timing fine-grained feature compensation vector.

8. The speech education robot based on children's speech modal characteristic perception according to claim 7, characterized in that: The speech signal embedding coding module is used to: use the speech signal embedding matrix to embed the speech input signal to obtain a sequence of speech input signal segment semantic embedding coding vectors as a sequence of speech input signal segment semantic embedding features.

9. The speech education robot based on children's speech modal characteristic perception according to claim 8, characterized in that: The reply speech generation module is used to: use the child speech modal characteristic representation vector as prompt information, add the child speech modal characteristic representation vector to the end of the sequence of semantic embedding coding vectors of the speech input signal segment, and then input it into the end-to-end speech large model to obtain the educational reply speech.

Images