Question tone recognition method and device, computer equipment and storage medium

Abstract

The invention relates to a question tone recognition method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring an original audio, and determining a phoneme sequence corresponding to the original audio; wherein the original audio comprises a plurality of frames, and the phoneme sequence comprises a plurality of phonemes; inputting the original audio and the phoneme sequence into a phoneme alignment model to obtain duration information of each phoneme; wherein the duration time information is used for indicating a corresponding relation between the phonemes and the frames; inputting the original audio into a pre-trained voice model to obtain frame feature vectors, and fusing the frame feature vectors into phoneme feature vectors according to the duration information; and performing question tone identification according to the phoneme feature vector to obtain an identification result. By adopting the method, the mood recognition accuracy in the voice information can be improved.

Description

Technical Field

[0001] The present application relates to the field of speech synthesis technology, and in particular to a method, apparatus, computer device, storage medium, and computer program product for identifying interrogative tone. Background Art

[0002] With the continuous development of text-to-speech (TTS) technology, current synthesized speech has reached a high level in terms of clarity and sound quality. However, significant deficiencies remain in terms of emotional expression and naturalness of voice. In intonation languages ​​like Chinese, the recognition and expression of interrogative tone in real conversations is particularly critical. If synthesized speech fails to accurately reflect interrogative tone, the realism and fluency of human-computer interaction will be significantly reduced. Therefore, accurately identifying and controlling interrogative tone in speech synthesis has become a key research topic in TTS.

[0003] In related technologies, because the text corresponding to audio data often lacks interrogative punctuation, acoustic models cannot effectively utilize the interrogative information contained in the original corpus during training. This results in the audio generated by the speech synthesis system lacking interrogative intonation. This omission can make the synthesized speech appear flat and unclear in questioning scenarios, seriously affecting the naturalness and expressiveness of the voice interaction system. To address this problem, related technologies identify interrogative intonation in speech.

[0004] However, in non-tonal languages ​​like Spanish, the expression of interrogative tone often lacks a clear pitch contour. Furthermore, native speakers vary greatly, and intonation habits vary significantly. For example, some speakers tend to raise their voice at the end of non-interrogative sentences, which doesn't necessarily indicate a question. Therefore, identifying tone based solely on intonation fluctuations can lead to mislabeling non-interrogative sentences as questions, creating training label noise and impacting model accuracy. Summary of the Invention

[0005] Based on this, it is necessary to provide a method, device, computer equipment, computer-readable storage medium and computer program product for interrogative tone recognition that can improve the accuracy of tone recognition in voice information in response to the above technical problems.

[0006] In a first aspect, the present application provides a method for identifying interrogative tone, comprising:

[0007] Acquire original audio, and determine a phoneme sequence corresponding to the original audio; wherein the original audio includes multiple frames, and the phoneme sequence includes multiple phonemes;

[0008] Inputting the original audio and the phoneme sequence into a phoneme alignment model to obtain duration information of each phoneme; wherein the duration information is used to indicate the corresponding relationship between the phoneme and the frame;

[0009] Inputting the original audio into a pre-trained speech model to obtain a frame feature vector, and fusing the frame feature vector into a phoneme feature vector according to the duration information;

[0010] Interrogative tone recognition is performed based on the phoneme feature vector to obtain a recognition result.

[0011] In one embodiment, fusing the frame feature vector into a phoneme feature vector according to the duration information includes:

[0012] Determining a frame set corresponding to each of the phonemes according to the duration information;

[0013] The frame feature vectors corresponding to the frames in the frame set are fused into the phoneme feature vector of the phoneme.

[0014] In one embodiment, fusing the frame feature vectors corresponding to the frames in the frame set into the phoneme feature vector of the phoneme includes:

[0015] An average value vector of the frame feature vectors corresponding to each frame in the frame set is calculated to obtain the phoneme feature vector of the phoneme.

[0016] In one embodiment, each of the frame feature vectors includes feature data of multiple dimensions; and fusing the frame feature vectors corresponding to each frame in the frame set into the phoneme feature vector of the phoneme includes:

[0017] For each dimension, finding the peak value of the feature data in the frame set to obtain target feature data;

[0018] The target feature data of each dimension are fused into a phoneme feature vector of the phoneme.

[0019] In one embodiment, fusing the frame feature vectors corresponding to the frames in the frame set into the phoneme feature vector of the phoneme includes:

[0020] Performing a linear transformation on the frame feature vector corresponding to each frame in the frame set to obtain an attention score of each frame feature vector;

[0021] Calculating a feature weight of each of the frame feature vectors according to the attention score;

[0022] According to the feature weights, weighted average calculation is performed on the feature vectors of each frame to obtain the phoneme feature vector of the phoneme.

[0023] In one embodiment, after performing interrogative tone recognition based on the phoneme feature vector and obtaining a recognition result, the method further includes:

[0024] Dividing the phoneme sequence into a plurality of word sequences; wherein the word sequence includes a plurality of phonemes;

[0025] When the recognition result indicates that the phoneme is in interrogative tone and the phoneme is the last phoneme in the corresponding word sequence, a interrogative tone mark is added to the phoneme.

[0026] In a second aspect, the present application further provides a device for identifying interrogative tone, comprising:

[0027] A data acquisition module, configured to acquire original audio and determine a phoneme sequence corresponding to the original audio; wherein the original audio includes a plurality of frames, and the phoneme sequence includes a plurality of phonemes;

[0028] a data processing module, configured to input the original audio and the phoneme sequence into a phoneme alignment model to obtain duration information of each phoneme; wherein the duration information is used to indicate a corresponding relationship between the phoneme and the frame;

[0029] a feature processing module, configured to input the original audio into a pre-trained speech model to obtain a frame feature vector, and fuse the frame feature vector into a phoneme feature vector according to the duration information;

[0030] The tone recognition module is used to perform question tone recognition based on the phoneme feature vector to obtain a recognition result.

[0031] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0032] Acquire original audio, and determine a phoneme sequence corresponding to the original audio; wherein the original audio includes multiple frames, and the phoneme sequence includes multiple phonemes;

[0033] Inputting the original audio and the phoneme sequence into a phoneme alignment model to obtain duration information of each phoneme; wherein the duration information is used to indicate the corresponding relationship between the phoneme and the frame;

[0034] Inputting the original audio into a pre-trained speech model to obtain a frame feature vector, and fusing the frame feature vector into a phoneme feature vector according to the duration information;

[0035] Interrogative tone recognition is performed based on the phoneme feature vector to obtain a recognition result.

[0036] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0037] Acquire original audio, and determine a phoneme sequence corresponding to the original audio; wherein the original audio includes multiple frames, and the phoneme sequence includes multiple phonemes;

[0038] Inputting the original audio and the phoneme sequence into a phoneme alignment model to obtain duration information of each phoneme; wherein the duration information is used to indicate the corresponding relationship between the phoneme and the frame;

[0039] Inputting the original audio into a pre-trained speech model to obtain a frame feature vector, and fusing the frame feature vector into a phoneme feature vector according to the duration information;

[0040] Interrogative tone recognition is performed based on the phoneme feature vector to obtain a recognition result.

[0041] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the following steps:

[0042] Acquire original audio, and determine a phoneme sequence corresponding to the original audio; wherein the original audio includes multiple frames, and the phoneme sequence includes multiple phonemes;

[0043] Inputting the original audio and the phoneme sequence into a phoneme alignment model to obtain duration information of each phoneme; wherein the duration information is used to indicate the corresponding relationship between the phoneme and the frame;

[0044] Inputting the original audio into a pre-trained speech model to obtain a frame feature vector, and fusing the frame feature vector into a phoneme feature vector according to the duration information;

[0045] Interrogative tone recognition is performed based on the phoneme feature vector to obtain a recognition result.

[0046] The above-mentioned interrogative tone recognition method, apparatus, computer device, storage medium and computer program product obtain original audio and determine the phoneme sequence corresponding to the original audio; wherein the original audio includes multiple frames, and the phoneme sequence includes multiple phonemes, and then the original audio and the phoneme sequence are input into a phoneme alignment model to obtain the duration information of each phoneme. The phoneme alignment model is used to obtain the duration information of each phoneme, which can clearly determine the start frame and end frame of each phoneme in the audio, thereby establishing a correspondence between the phoneme and the frame. The original audio is input into a pre-trained speech model to obtain a frame feature vector, and the frame feature vector is fused into a phoneme feature vector based on the duration information. The multiple frame feature vectors are fused into a phoneme-level feature vector corresponding to each phoneme, which can construct a feature representation at the phoneme granularity. Interrogative tone recognition is then performed based on the phoneme feature vector to obtain a recognition result. By performing the tone recognition task on the phoneme feature vector, the presence or absence of interrogative tone can be accurately judged at the phoneme granularity, avoiding the coarse-grained judgment error based only on text or sentence-end punctuation, and improving the accuracy of interrogative tone recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 A diagram showing an application environment of a method for identifying interrogative tone in one embodiment;

[0049] Figure 2 1 is a flow chart of a method for identifying interrogative tone in one embodiment;

[0050] Figure 3 1 is a flow chart of a method for identifying interrogative tone in another embodiment;

[0051] Figure 4 is a structural block diagram of a device for identifying interrogative tone in one embodiment;

[0052] Figure 5 is a diagram of the internal structure of a computer device in one embodiment;

[0053] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0055] The interrogative tone recognition method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network, and the terminal 102 can be used to obtain the original audio and send the obtained original audio to the server 104 for processing. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The data storage system can be used to store information such as the original audio, the extracted phoneme series, and the recognition results of the interrogative tone recognition. Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented as an independent server or a server cluster consisting of multiple servers.

[0056] In an exemplary embodiment, Figure 2 As shown, a question tone recognition method is provided, which is applied to Figure 1 The server 104 in the example is used as an example to illustrate the process, including the following steps S202 to S208.

[0057] Step S202: Acquire the original audio and determine the phoneme sequence corresponding to the original audio.

[0058] The original audio consists of multiple frames, and the phoneme sequence includes multiple phonemes. Original audio refers to the actual speech fragment uttered by the user, such as a WAV format audio file with a sampling rate of 16kHz. It consists of multiple consecutive acoustic frames, each representing the speech signal characteristics at a specific moment and can be 20 to 25 milliseconds. The phoneme sequence is the basic unit of pronunciation of the original text content in speech synthesis. This can be formed by converting the text into a phonetic sequence or an international phonetic alphabet sequence. For example, the phrase "how are you?" might have a phoneme sequence of [n i3 h a3 ma], where each phoneme represents a specific pronunciation action. A frame is the smallest processing unit for dividing a continuous audio signal along the time axis.

[0059] For example, server 104 may receive a raw audio input from terminal 102. This audio may be derived from user utterances in a natural speech conversation scenario, with complete speech flow, intonation, and pause information. Server 104 may divide the audio into frames based on a fixed window, with each frame recording the sound information within a time segment. For example, if one second of audio is divided into 25ms units, it can be divided into approximately 40 frames. After obtaining the framed audio data, server 104 may further extract the original sentence from the text and convert the text into corresponding phoneme sequences using a phoneme conversion tool.

[0060] Step S204: input the original audio and the phoneme sequence into a phoneme alignment model to obtain duration information of each phoneme.

[0061] Duration information is used to indicate the correspondence between phonemes and frames. A phoneme alignment model, which can be trained on an end-to-end speech recognition network, is used to assign explicit start and end frames to each phoneme in the temporal dimension, thereby forming phoneme temporal boundary information at the frame level.

[0062] For example, after obtaining the original audio and the corresponding phoneme sequence, server 104 may input the original audio and the phoneme sequence into a phoneme alignment model. The phoneme alignment model maps each phoneme in the phoneme sequence to each frame in the original audio, thereby obtaining phoneme-level duration information, that is, clearly defining the corresponding frame start position and frame end position of each phoneme in the audio. Specifically, the phoneme alignment model can adopt different alignment methods, such as the soft-alignment-based RAD (Recurrent Alignment with Dynamic Features) method, or the hard-alignment-based CTC (Connectionist Temporal Classification) method, or a combination of two or more alignment methods.

[0063] If the RAD mechanism is used for alignment, the phoneme alignment model can learn the probability distribution between phonemes and frames during the training phase. In this architecture, the server 104 can construct a two-dimensional alignment map between phonemes and frames based on the attention mechanism, where each element represents the probability score of a certain phoneme appearing in a certain frame. By introducing a monotonicity constraint, under the premise of ensuring the consistency of the order of phonemes, an optimal path is derived from the entire soft alignment probability map using a dynamic programming method, and several consecutive frames are attributed to a certain phoneme, thereby converting the soft alignment into a hard alignment time boundary. This method has a strong context modeling capability and is suitable for the recognition of fuzzy speech segments under complex speech flows, especially when the pronunciation boundaries are fuzzy and the speech rate changes frequently. It is more robust.

[0064] If the CTC architecture is used for alignment, the phoneme alignment model can calculate the probability distribution of the phoneme or blank symbol (blank) to which each frame belongs, and generate all possible alignment paths. During training or inference, the server 104 can calculate the total probability of all legal paths that satisfy the phoneme order through the forward-backward algorithm, and select the path with the highest probability as the final alignment result. The output result of CTC is frame-level annotation, that is, the phoneme to which each frame belongs. Therefore, the server 104 can directly identify the frame segments where each phoneme appears continuously from this annotation, so as to obtain the duration information of the phonemes. The CTC alignment method is suitable for audio scenarios with clear structure and low noise.

[0065] Through the above steps, the server 104 can establish a mapping relationship between phonemes and frames based on the alignment result. For example, for the phoneme sequence [n i3 h a3 m a] of the phrase "How are you", the server 104 can output the following structure: "n" corresponds to frames 3 to 11, "i3" corresponds to frames 12 to 22, "h" corresponds to frames 23 to 28... to obtain the duration information.

[0066] Step S206, input the original audio into the pre-trained speech model to obtain frame feature vectors, and fuse the frame feature vectors into phoneme feature vectors according to the duration information.

[0067] Among them, each frame feature vector includes feature data of multiple dimensions. The pre-trained speech model can be a large model trained based on self-supervised learning, and its structure has the ability of deep convolution or Transformer stacking, and can extract rich and structured frame-level features from the original audio.

[0068] Exemplarily, the server 104 can first perform preprocessing operations such as short-time Fourier transform and window function sliding on the original audio, divide the continuous audio stream into several equal-length frames, and each frame generally corresponds to 20 to 25 milliseconds of audio. Subsequently, the server 104 inputs these frames into the pre-trained speech model, and this model can extract hidden state features frame by frame through a multi-layer structure, and finally obtain a three-dimensional tensor [B, T, M], where B is the batch size, T is the number of frames, and M is the feature dimension corresponding to each frame (for example, 768 dimensions). These high-dimensional features not only capture the spectral details of each frame of audio, but also contain the general language representation ability that the model has learned from a large amount of speech data during the training stage, thus improving the basis for tone recognition in downstream tasks.

[0069] Furthermore, server 104 can obtain the frame-level boundary set of phonemes based on the duration information output by the previous phoneme alignment model, thereby constructing a matrix [N, L], where N is the length of the phoneme sequence, i.e., the number of phonemes actually pronounced, and L represents the number of frames corresponding to each phoneme (i.e., the frame length of the phoneme in the audio). Each row of this [N, L] matrix represents the timeline coverage of a phoneme. For example, the nth phoneme corresponds to the frame interval [tn_start, tn_end]. Server 104 can then extract the L frame feature vectors corresponding to the nth row to form a frame feature subset [L, M], where L is the number of frames spanned by the phoneme. This [N, L] structure not only reflects phoneme boundaries but also reflects individual differences in speaking rate and pronunciation duration. For example, some phonemes may occupy only three frames, while others may last for eight frames. Server 104 can dynamically slice and process each change in L. Subsequently, the server 104 may perform an aggregation operation on each [L, M] sub-matrix to fuse them into an [M]-dimensional phoneme feature vector, and arrange the features of all N phonemes in sequence to finally form an [N, M] phoneme feature vector.

[0070] Furthermore, the server 104 can determine the frame set corresponding to each phoneme based on the duration information; and fuse the frame feature vectors corresponding to each frame in the frame set into the phoneme feature vector of the phoneme. The server 104 can read the start frame and end frame number corresponding to each phoneme in the audio based on the duration information output by the phoneme alignment model. Taking a certain phoneme "a" as an example, if its corresponding frames in the original audio are the 25th to 32nd frames, the server 104 can classify all frame feature vectors with frame numbers between 25 and 32 as the frame set of the phoneme "a". In a speech segment composed of multiple phonemes, the server 104 can establish such frame sets one by one, and finally obtain a set sequence with a total length of the number of phonemes N, and each set contains a [L_n,M] sub-tensor composed of several frame features, where L_n is the number of frames of the nth phoneme and M is the frame feature dimension extracted by the pre-training model.

[0071] After completing the division of the frame sets, the server 104 can fuse the features of several frames in each set into a fixed-length phoneme feature vector [M] to form a phoneme-level input in a unified format. For example, the server 104 can select different fusion methods according to different tone modeling requirements. Taking average fusion as an example, the server 104 can calculate the arithmetic mean of the [M]-dimensional features corresponding to each frame in the set in the time dimension to obtain the fused phoneme features. On this basis, the server 104 can arrange the [M] vectors corresponding to each phoneme in sequence to finally generate a complete [N, M] phoneme feature sequence. Each [M] vector is derived from its corresponding frame set, fully retaining the actual duration, pitch change and rhythm characteristics of the phoneme during the pronunciation process, thereby making the phoneme representation more tone-sensitive and time-consistent, not only ensuring the structural standardization of the model input, but also ensuring that the feature expression of each phoneme accurately corresponds to the time period when it actually appears in the speech.

[0072] Furthermore, in different embodiments, server 104 may perform a fusion operation on the frame feature vectors in the set based on different strategies to construct a phoneme feature vector for each phoneme. Alternatively, server 104 may employ a combination of multiple strategies to construct a phoneme feature vector for each phoneme, such as performing a weighted average of the phoneme feature vectors obtained using multiple strategies to obtain a final phoneme feature vector. Different fusion strategies can capture different aspects of voice changes in audio, enhancing the system's adaptability in scenarios with complex intonation and diverse expressions.

[0073] Exemplarily, the server 104 can calculate the average vector of the frame feature vectors corresponding to each frame in the frame set to obtain the phoneme feature vector of the phoneme. The server 104 can generate a unified feature representation of the phoneme by calculating the dimension-by-dimension average of all frame feature vectors in the frame set. Exemplarily, for a number of frame features [L, M] corresponding to a certain phoneme, the server 104 can perform a mean operation on the time dimension L, add up all frame values ​​of each dimension and divide by the number of frames L, thereby obtaining an average feature vector of length [M]. This averaging strategy is suitable for scenarios where the tone changes are relatively smooth or the global stability requirements are high. It can smooth out the interference caused by occasional audio fluctuations and generate a phoneme representation that is more neutral to the overall pronunciation performance. When the pronunciation of a phoneme in the speech is relatively smooth and there is no obvious tone jump, such as the middle phoneme "de" in the declarative tone, this average fusion strategy can effectively retain its basic timbre and speech speed characteristics.

[0074] Exemplarily, the server 104 may also search for a peak value of feature data in the frame set for each dimension to obtain target feature data; and fuse the target feature data of each dimension into a phoneme feature vector of the phoneme.

[0075] Server 104 can perform a dimension-wise maximum operation on the frame set, searching for the peak value across all frame features in each dimension as the representative value for that dimension. Ultimately, server 104 can combine the peak data from each dimension to form a maximum-pooled phoneme feature vector for the phoneme. This strategy is suitable for situations with strong fluctuations in tone, rising intonation, or accelerated speech. It can enhance the most prominent expression points in the frame features and is particularly suitable for capturing intonation at the end of interrogative sentences.

[0076] For example, server 104 can also perform a linear transformation on the frame feature vectors corresponding to each frame in the frame set to obtain an attention score for each frame feature vector; calculate a feature weight for each frame feature vector based on the attention score; and perform a weighted average of the frame feature vectors based on the feature weights to obtain the phoneme feature vector for the phoneme. Server 104 can first perform a linear transformation on the [M] feature vector of each frame in the frame set through a linear layer to calculate [B, T, 1] to obtain its attention score; then normalize these scores using softmax to obtain a weight distribution for the corresponding frame. Next, server 104 can use these attention weights as weighting coefficients to perform a weighted average of all frame feature vectors to generate the final phoneme feature vector. This strategy has the advantage of dynamically emphasizing the frames most important for determining voice tone, rather than considering all frames equally. This allows for a stronger perception of semantically rich moments such as intonation fluctuations and emotional shifts. For example, if the phoneme "ma" has a pronounced upward pitch in its pronunciation, the attention mechanism can automatically assign weights to the corresponding frame segments, allowing the fused features to focus on this trend.

[0077] Through the above steps, the server 104 can flexibly select strategies such as averaging, maximum pooling or attention mechanism based on the phoneme frame set to generate phoneme feature vectors. Each strategy has its own advantages and can be flexibly combined according to the actual tone recognition task scenario to further improve the model's recognition accuracy and generalization ability for interrogative tone.

[0078] Step S208: performing question tone recognition based on the phoneme feature vector to obtain a recognition result.

[0079] For example, server 104 may input the aforementioned phoneme feature vectors into a pre-defined interrogative tone recognition model. This model may employ a lightweight phoneme classification network structure, or a deep neural network architecture that incorporates sequence context, such as a Transformer architecture with a bidirectional encoder, or a multi-task network that incorporates language modeling information. The model's function is to identify the temporal and semantic feature combinations of each phoneme, thereby determining whether the phoneme falls within the interrogative tone expression segment.

[0080] Furthermore, server 104 can assign a predicted interrogative tone label to each [M]-dimensional feature vector in the phoneme sequence, outputting an interrogative tone prediction result [N, 1] that matches the length of the phoneme sequence, where each value represents the probability of the corresponding phoneme being interrogative, or a binary classification result. For example, in a sentence, server 104 might output the first five phonemes as 0 (indicating a non-interrogative tone) and the last two phonemes as 1 (indicating an interrogative tone segment). This phoneme-level prediction granularity enables the model to not only classify but also detect interrogative tone boundaries.

[0081] During the training phase, server 104 can use data with phoneme-level interrogative tone annotations for supervised learning, such as question annotations generated by LLM, regular annotations combined with pitch changes, or manually proofread sentence-end tone markers, with all sentence-end pronunciations labeled "qs" as positive samples, and the remaining phonemes as negative samples. The model can optimize the binary classification loss function during training, thereby improving its stability and generalization ability in binary classification tasks. Next, server 104 can set a threshold based on the interrogative tone probability distribution output by the model to determine whether to output a "yes" or "no" label, and further use these labels to guide downstream systems to add interrogative intonation, change the speaking rate, or insert a rising tone curve in speech synthesis.

[0082] In the above-mentioned interrogative tone recognition method, original audio is obtained and a phoneme sequence corresponding to the original audio is determined; wherein, the original audio includes multiple frames, and the phoneme sequence includes multiple phonemes, and then the original audio and the phoneme sequence are input into a phoneme alignment model to obtain the duration information of each phoneme. The phoneme alignment model is used to obtain the duration information of each phoneme, which can clearly determine the start frame and end frame of each phoneme in the audio, thereby establishing a correspondence between the phoneme and the frame, and the original audio is input into a pre-trained speech model to obtain a frame feature vector, and the frame feature vector is fused into a phoneme feature vector based on the duration information. Multiple frame feature vectors are fused into a phoneme-level feature vector corresponding to each phoneme, which can construct a feature representation at the phoneme granularity, and then interrogative tone recognition is performed based on the phoneme feature vector to obtain a recognition result. By performing the tone recognition task on the phoneme feature vector, the presence or absence of interrogative tone can be accurately judged at the phoneme granularity, avoiding the coarse-grained judgment error based only on text or sentence-end punctuation, and improving the accuracy of interrogative tone recognition.

[0083] In an exemplary embodiment, Figure 3 As shown, the above method further includes steps S302 to S304. In which:

[0084] Step S302: Divide the phoneme sequence into several word sequences.

[0085] Among them, the word sequence includes several phonemes.

[0086] Exemplarily, the server 104 can perform word-level segmentation processing on the input phoneme sequence. This segmentation can be achieved based on text tokenization tools, dictionary comparison, language model inference, etc. For example, for the sentence "Have you eaten?", the server 104 can first tokenize the sentence into three phrases: "you", "have eaten", and "?", and then divide the corresponding phoneme sequence into three phoneme segments: [ni], [chifanle], and [ma]. Each word sequence contains several phonemes, and the server 104 can organize them into a [W, P] structure, where W is the number of words and P is the number of phonemes contained in each word, which may not be of equal length.

[0087] Step S304, in the case where the recognition result indicates that the phoneme is in an interrogative mood and the phoneme is the last phoneme in the corresponding word sequence, add an interrogative mood marker to the phoneme.

[0088] Exemplarily, the server 104 can traverse the previously recognized interrogative mood label results [N, 1], search for the phonemes determined to be in the "interrogative mood" (i.e., the value is 1), and judge their positions in the word structure. If a certain phoneme is both recognized as an interrogative mood phoneme and the last phoneme in its corresponding word sequence, the server 104 can use this phoneme as the boundary of the mood segment and perform an explicit marking operation for the interrogative mood. This marking operation can be achieved by adding a "qs" label to the phoneme structure, setting a binary classification flag bit, or guiding the intonation to rise in the form of meta-information in subsequent speech synthesis. For example, in the phoneme sequence [ma] of the word "?", if the server 104 recognizes "a" as an interrogative mood and confirms that it is in the last phoneme position in the word sequence, it can determine that this word has an interrogative mood, and thus add a "qs" label or other mood markers to the phoneme "a".

[0089] By only adding interrogative mood markers to the trailing phonemes of words, the server 104 can effectively avoid misidentifying the intonation changes in non-interrogative words as interrogative moods, thereby reducing false alarms in mood annotation, making the interrogative mood label have a clear structural boundary and semantic reference, and thus generating output speech with clearer semantics and more natural mood.

[0090] Finally, the server 104 can output the phoneme sequence with the completed markers as structured data, ensuring that each interrogative mood marker is attached to the natural breakpoints allowed by the language structure, not only retaining the accuracy of mood judgment but also enhancing the interpretability and controllability at the language level.

[0091] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0092] Based on the same inventive concept, embodiments of the present application also provide an interrogative tone recognition device for implementing the interrogative tone recognition method described above. The solution provided by this device is similar to the solution described in the method described above. Therefore, the specific limitations of one or more interrogative tone recognition device embodiments provided below can be found in the limitations of the interrogative tone recognition method described above and will not be further elaborated here.

[0093] In an exemplary embodiment, Figure 4 As shown, a question tone recognition device is provided, comprising: a data acquisition module 402, a data processing module 404, a feature processing module 406 and a tone recognition module 408, wherein:

[0094] The data acquisition module 402 is configured to acquire original audio and determine a phoneme sequence corresponding to the original audio; wherein the original audio includes a plurality of frames, and the phoneme sequence includes a plurality of phonemes;

[0095] The data processing module 404 is configured to input the original audio and the phoneme sequence into a phoneme alignment model to obtain duration information of each phoneme; wherein the duration information is used to indicate the correspondence between the phoneme and the frame;

[0096] A feature processing module 406 is configured to input the original audio into a pre-trained speech model to obtain a frame feature vector, and fuse the frame feature vector into a phoneme feature vector based on duration information;

[0097] The tone recognition module 408 is used to perform question tone recognition based on the phoneme feature vector to obtain a recognition result.

[0098] In one embodiment, the feature processing module 406 includes:

[0099] an information processing unit, configured to determine a frame set corresponding to each phoneme based on the duration information;

[0100] The feature fusion unit is used to fuse the frame feature vectors corresponding to each frame in the frame set into the phoneme feature vector of the phoneme.

[0101] In one embodiment, the feature fusion unit is specifically configured to calculate an average value vector of the frame feature vectors corresponding to each frame in the frame set to obtain a phoneme feature vector of the phoneme.

[0102] In one embodiment, each frame feature vector includes feature data of multiple dimensions; the feature fusion unit is specifically used to: for each dimension, find the peak value of the feature data in the frame set to obtain target feature data; and fuse the target feature data of each dimension into a phoneme feature vector of the phoneme.

[0103] In one embodiment, the feature fusion unit is specifically used to: perform linear transformation on the frame feature vector corresponding to each frame in the frame set to obtain the attention score of each frame feature vector; calculate the feature weight of each frame feature vector based on the attention score; and perform weighted average calculation on the feature vectors of each frame based on the feature weight to obtain the phoneme feature vector of the phoneme.

[0104] In one embodiment, the device further includes: a word boundary processing module for dividing a phoneme sequence into a plurality of word sequences; wherein the word sequence includes a plurality of phonemes; and adding a question tone mark to the phoneme when the recognition result indicates that the phoneme is an interrogative tone and the phoneme is the last phoneme in the corresponding word sequence.

[0105] Each module in the interrogative tone recognition device described above may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in hardware form, or may be stored in a computer device memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0106] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 5As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store information such as original audio, extracted phoneme series and recognition results of interrogative tone recognition. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for interrogative tone recognition is implemented.

[0107] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via wired or wireless means, and the wireless means can be achieved via Wi-Fi, mobile cellular networks, NFC (near-field communication), or other technologies. When executed by the processor, the computer program implements a method for identifying interrogative speech. The display unit of the computer device is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.

[0108] Those skilled in the art will understand that Figure 6The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0109] In an exemplary embodiment, a computer device is provided, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: obtaining original audio and determining a phoneme sequence corresponding to the original audio; wherein the original audio includes multiple frames and the phoneme sequence includes multiple phonemes; inputting the original audio and the phoneme sequence into a phoneme alignment model to obtain duration information of each phoneme; wherein the duration information is used to indicate the correspondence between the phoneme and the frame; inputting the original audio into a pre-trained speech model to obtain a frame feature vector, and fusing the frame feature vector into a phoneme feature vector based on the duration information; and performing interrogative tone recognition based on the phoneme feature vector to obtain a recognition result.

[0110] In one embodiment, when executing the computer program, the processor further implements the following steps: determining a frame set corresponding to each phoneme according to the duration information; and fusing frame feature vectors corresponding to each frame in the frame set into a phoneme feature vector of the phoneme.

[0111] In one embodiment, when the processor executes the computer program, the processor further implements the following steps: calculating an average value vector of the frame feature vectors corresponding to each frame in the frame set to obtain a phoneme feature vector of the phoneme.

[0112] In one embodiment, when executing the computer program, the processor further implements the following steps: for each dimension, searching for the peak value of the feature data in the frame set to obtain target feature data; and fusing the target feature data of each dimension into a phoneme feature vector of the phoneme.

[0113] In one embodiment, when the processor executes the computer program, it also implements the following steps: performing a linear transformation on the frame feature vector corresponding to each frame in the frame set to obtain an attention score of each frame feature vector; calculating the feature weight of each frame feature vector based on the attention score; and performing a weighted average calculation on the feature vectors of each frame based on the feature weight to obtain the phoneme feature vector of the phoneme.

[0114] In one embodiment, when the processor executes the computer program, it also implements the following steps: dividing the phoneme sequence into several word sequences; wherein the word sequence includes several phonemes; when the recognition result indicates that the phoneme is an interrogative tone and the phoneme is the last phoneme in the corresponding word sequence, adding an interrogative tone mark to the phoneme.

[0115] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: obtaining original audio and determining a phoneme sequence corresponding to the original audio; wherein the original audio includes multiple frames and the phoneme sequence includes multiple phonemes; inputting the original audio and the phoneme sequence into a phoneme alignment model to obtain duration information of each phoneme; wherein the duration information is used to indicate the correspondence between the phoneme and the frame; inputting the original audio into a pre-trained speech model to obtain a frame feature vector, and fusing the frame feature vector into a phoneme feature vector based on the duration information; and performing interrogative tone recognition based on the phoneme feature vector to obtain a recognition result.

[0116] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: determining a frame set corresponding to each phoneme according to the duration information; and fusing the frame feature vectors corresponding to each frame in the frame set into a phoneme feature vector of the phoneme.

[0117] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: calculating an average value vector of the frame feature vectors corresponding to each frame in the frame set to obtain a phoneme feature vector of the phoneme.

[0118] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: for each dimension, finding the peak value of the feature data in the frame set to obtain target feature data; and fusing the target feature data of each dimension into a phoneme feature vector of the phoneme.

[0119] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: linearly transforming the frame feature vector corresponding to each frame in the frame set to obtain the attention score of each frame feature vector; calculating the feature weight of each frame feature vector based on the attention score; and performing weighted averaging calculation on the feature vectors of each frame based on the feature weight to obtain the phoneme feature vector of the phoneme.

[0120] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: dividing a phoneme sequence into several word sequences; wherein the word sequence includes several phonemes; and when the recognition result indicates that the phoneme is an interrogative tone and the phoneme is the last phoneme in the corresponding word sequence, adding an interrogative tone mark to the phoneme.

[0121] In one embodiment, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the following steps: obtaining original audio and determining a phoneme sequence corresponding to the original audio; wherein the original audio includes multiple frames and the phoneme sequence includes multiple phonemes; inputting the original audio and the phoneme sequence into a phoneme alignment model to obtain duration information of each phoneme; wherein the duration information is used to indicate the correspondence between the phoneme and the frame; inputting the original audio into a pre-trained speech model to obtain a frame feature vector, and fusing the frame feature vector into a phoneme feature vector based on the duration information; and performing interrogative tone recognition based on the phoneme feature vector to obtain a recognition result.

[0122] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: determining a frame set corresponding to each phoneme according to the duration information; and fusing the frame feature vectors corresponding to each frame in the frame set into a phoneme feature vector of the phoneme.

[0123] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: calculating an average value vector of the frame feature vectors corresponding to each frame in the frame set to obtain a phoneme feature vector of the phoneme.

[0124] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: for each dimension, finding the peak value of the feature data in the frame set to obtain target feature data; and fusing the target feature data of each dimension into a phoneme feature vector of the phoneme.

[0125] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: linearly transforming the frame feature vector corresponding to each frame in the frame set to obtain the attention score of each frame feature vector; calculating the feature weight of each frame feature vector based on the attention score; and performing weighted averaging calculation on the feature vectors of each frame based on the feature weight to obtain the phoneme feature vector of the phoneme.

[0126] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: dividing a phoneme sequence into several word sequences; wherein the word sequence includes several phonemes; and when the recognition result indicates that the phoneme is an interrogative tone and the phoneme is the last phoneme in the corresponding word sequence, adding an interrogative tone mark to the phoneme.

[0127] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0128] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0129] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0130] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for identifying interrogative tone, characterized in that: The method comprises: Acquire original audio, and determine a phoneme sequence corresponding to the original audio; wherein the original audio includes multiple frames, and the phoneme sequence includes multiple phonemes; Inputting the original audio and the phoneme sequence into a phoneme alignment model to obtain duration information of each phoneme; wherein the duration information is used to indicate the corresponding relationship between the phoneme and the frame; Inputting the original audio into a pre-trained speech model to obtain a frame feature vector, and fusing the frame feature vector into a phoneme feature vector according to the duration information; Interrogative tone recognition is performed based on the phoneme feature vector to obtain a recognition result.

2. The method according to claim 1, characterized in that The fusing the frame feature vectors into a phoneme feature vector according to the duration information includes: Determining a frame set corresponding to each of the phonemes according to the duration information; The frame feature vectors corresponding to the frames in the frame set are fused into the phoneme feature vector of the phoneme.

3. The method according to claim 2, characterized in that The fusing the frame feature vectors corresponding to the frames in the frame set into the phoneme feature vector of the phoneme includes: An average value vector of the frame feature vectors corresponding to each frame in the frame set is calculated to obtain the phoneme feature vector of the phoneme.

4. The method according to claim 2, characterized in that Each of the frame feature vectors includes feature data of multiple dimensions; fusing the frame feature vectors corresponding to the frames in the frame set into the phoneme feature vector of the phoneme includes: For each dimension, finding the peak value of the feature data in the frame set to obtain target feature data; The target feature data of each dimension are fused into a phoneme feature vector of the phoneme.

5. The method according to claim 2, characterized in that The fusing the frame feature vectors corresponding to the frames in the frame set into the phoneme feature vector of the phoneme includes: Performing a linear transformation on the frame feature vector corresponding to each frame in the frame set to obtain an attention score of each frame feature vector; Calculating a feature weight of each of the frame feature vectors according to the attention score; According to the feature weights, weighted average calculation is performed on the feature vectors of each frame to obtain the phoneme feature vector of the phoneme.

6. The method according to any one of claims 1 to 5, characterized in that After performing interrogative tone recognition according to the phoneme feature vector and obtaining a recognition result, the method further includes: Dividing the phoneme sequence into a plurality of word sequences; wherein the word sequence includes a plurality of phonemes; When the recognition result indicates that the phoneme is in interrogative tone and the phoneme is the last phoneme in the corresponding word sequence, a interrogative tone mark is added to the phoneme.

7. A device for identifying interrogative tone, characterized in that: The device comprises: A data acquisition module, configured to acquire original audio and determine a phoneme sequence corresponding to the original audio; wherein the original audio includes a plurality of frames, and the phoneme sequence includes a plurality of phonemes; a data processing module, configured to input the original audio and the phoneme sequence into a phoneme alignment model to obtain duration information of each phoneme; wherein the duration information is used to indicate a corresponding relationship between the phoneme and the frame; a feature processing module, configured to input the original audio into a pre-trained speech model to obtain a frame feature vector, and fuse the frame feature vector into a phoneme feature vector according to the duration information; The tone recognition module is used to perform question tone recognition based on the phoneme feature vector to obtain a recognition result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images