Audio intonation recognition method and device, computer equipment and readable storage medium

Abstract

The invention relates to an audio intonation recognition method and device, computer equipment and a readable storage medium. The method comprises the following steps: acquiring a to-be-processed audio signal, and performing fundamental frequency extraction on the to-be-processed audio signal to obtain a fundamental frequency track; performing intonation analysis on a first fundamental frequency sequence corresponding to a mute leading segment in the fundamental frequency track to obtain an intonation type of the mute leading segment; performing intonation analysis on the second fundamental frequency sequences corresponding to the plurality of phonemes in the fundamental frequency track to obtain intonation types of the plurality of phonemes; and determining the tone type according to the intonation type of the mute leading segment, the intonation types of the plurality of phonemes, the short-time energy of the plurality of audio frames in the audio signal to be processed and the pronunciation durations of the plurality of phonemes. By adopting the method, the audio intonation recognition accuracy can be improved, and the naturalness and expressive force of voice interaction can be improved.

Description

Technical Field

[0001] This application relates to the field of audio recognition technology, and in particular to an audio intonation recognition method, apparatus, computer device, and readable storage medium. Background Technology

[0002] In fields such as speech synthesis, speech recognition, dialogue systems, and affective computing, accurately and automatically identifying intonation changes, emotional states, and special speech events (such as laughter, sighs, and hesitation) is a key technological bottleneck for achieving highly human-like, natural, and fluent human-computer interaction. Traditional mainstream solutions primarily employ intonation analysis based on statistical models, such as using HMM (Hidden Markov Model) and GMM (Gaussian Mixture Model) to model the fundamental frequency profile. However, these models have poor generalization ability and are insufficiently adaptable to individual differences and complex intonation patterns. Rule-driven acoustic event detection, such as setting threshold rules for specific events (e.g., laughter) based on energy and zero-crossing rate, is also used. However, rule design is cumbersome and struggles to cover diverse expressions. Therefore, traditional audio intonation recognition methods suffer from low accuracy, limiting the improvement of naturalness and expressiveness in speech interaction. Summary of the Invention

[0003] Therefore, it is necessary to provide an audio intonation recognition method, device, computer equipment, computer-readable storage medium, and computer program product to address the aforementioned technical problems, which can improve the accuracy of audio intonation recognition and thus enhance the naturalness and expressiveness of voice interaction.

[0004] Firstly, this application provides an audio intonation recognition method, including:

[0005] The audio signal to be processed is acquired, and the fundamental frequency is extracted from the audio signal to be processed to obtain the fundamental frequency trajectory;

[0006] The intonation analysis of the first fundamental frequency sequence corresponding to the silence preamble in the fundamental frequency trajectory is performed to obtain the intonation type of the silence preamble;

[0007] Tone analysis is performed on the second fundamental frequency sequence corresponding to each of the multiple phonemes in the fundamental frequency trajectory to obtain the tone type of each phoneme.

[0008] The tone type is determined based on the intonation type of the silent lead-in, the intonation types of multiple phonemes, the short-time energy of multiple audio frames in the audio signal to be processed, and the pronunciation duration of multiple phonemes.

[0009] In one embodiment, intonation analysis is performed on the first fundamental frequency sequence corresponding to the silence leader segment in the fundamental frequency trajectory to obtain the intonation type of the silence leader segment, including:

[0010] Obtain the difference between adjacent frames in the first fundamental frequency sequence corresponding to the silent preamble in the fundamental frequency trajectory;

[0011] The mean of the differences between multiple adjacent frames in the first fundamental frequency sequence is taken as the mean of the differences corresponding to the first fundamental frequency sequence.

[0012] The intonation type of the silent lead segment is determined based on the difference mean value corresponding to the first fundamental frequency sequence and the preset difference threshold.

[0013] In one embodiment, the intonation type of the silence leading segment is determined based on the difference mean corresponding to the first fundamental frequency sequence and a preset difference threshold, including:

[0014] If the mean difference value corresponding to the first fundamental frequency sequence is greater than the preset difference threshold, the intonation type of the silent leading segment is determined to be the intonation rising type.

[0015] If the mean difference value corresponding to the first fundamental frequency sequence is equal to the preset difference threshold, the intonation type of the silence leading segment is determined to be the intonation smooth type.

[0016] If the mean difference value corresponding to the first fundamental frequency sequence is less than the preset difference threshold, the intonation type of the silent leading segment is determined to be the intonation falling type.

[0017] In one embodiment, the audio intonation recognition method further includes:

[0018] If the mean difference value corresponding to the first fundamental frequency sequence is less than a preset difference threshold and greater than a preset leniency threshold, the scenario type of the silent prelude is determined.

[0019] In the case where the scenario type is cross-silence rising type and the mean difference value of the first fundamental frequency segment in the fundamental frequency trajectory that is later than the silence leading segment is greater than the mean difference value of the second fundamental frequency segment in the fundamental frequency trajectory that is earlier than the silence leading segment, the intonation type of the silence leading segment will be corrected from intonation falling type to intonation rising type.

[0020] In the case of a scenario with a leading silence interference, the silent leading segment is re-extracted from the audio signal to be processed at a position earlier than the preset duration of the silent leading segment, resulting in an updated silent leading segment. If the mean difference value corresponding to the updated silent leading segment is greater than the preset difference threshold, the tone type of the silent leading segment is corrected from a falling tone type to a rising tone type.

[0021] In one embodiment, determining the scenario type of the silence prelude includes:

[0022] Intonation analysis is performed on the first fundamental frequency segment in the fundamental frequency trajectory that is later than the silent preamble to obtain the first intonation type. Intonation analysis is performed on the second fundamental frequency segment in the fundamental frequency trajectory that is earlier than the silent preamble to obtain the second intonation type.

[0023] When both the first and second intonation types are intonation rising, the scenario type of the silent leading segment is determined to be the cross-silence rising type;

[0024] If, in the audio signal to be processed, there is a reference audio segment whose average energy is less than a preset energy threshold in an audio segment earlier than the silent prelude, determine the prelude silent interference type of the scene type of the silent prelude.

[0025] In one embodiment, intonation analysis is performed on the second fundamental frequency sequences corresponding to each of the multiple phonemes in the fundamental frequency trajectory to obtain the intonation type of each of the multiple phonemes, including:

[0026] For any phoneme of the audio signal to be processed, the second fundamental frequency sequence corresponding to the current phoneme is divided into a first fundamental frequency subsequence and a second fundamental frequency subsequence;

[0027] The intonation type of the current phoneme is determined based on the sum of the differences between the first and second fundamental frequency subsequences.

[0028] In one embodiment, the audio intonation recognition method further includes:

[0029] The intonation level of the silent preamble is determined based on the mean difference value corresponding to the silent preamble and the threshold ranges of multiple intonation levels.

[0030] The intonation level of each phoneme in the audio signal to be processed is determined based on the mean difference of each phoneme and the threshold range of each intonation level.

[0031] The intonation level of the silent lead-in, the intonation level of each of the multiple phonemes, the short-time energy of multiple audio frames in the audio signal to be processed, and the pronunciation duration of each of the multiple phonemes are converted into feature vectors.

[0032] Based on the feature vectors, sentiment classification is performed using a pre-trained sentiment classification model to obtain the sentiment classification results.

[0033] Secondly, this application also provides an audio intonation recognition device, comprising:

[0034] The acquisition module is used to acquire the audio signal to be processed, extract the fundamental frequency of the audio signal to be processed, and obtain the fundamental frequency trajectory.

[0035] The first analysis module is used to perform intonation analysis on the first fundamental frequency sequence corresponding to the silence preamble in the fundamental frequency trajectory to obtain the intonation type of the silence preamble.

[0036] The second analysis module is used to perform intonation analysis on the second fundamental frequency sequence corresponding to each of the multiple phonemes in the fundamental frequency trajectory, and to obtain the intonation type of each of the multiple phonemes.

[0037] The determination module is used to determine the tone type based on the tone type of the silent lead-in, the tone types of multiple phonemes, the short-time energy of multiple audio frames in the audio signal to be processed, and the pronunciation duration of multiple phonemes.

[0038] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0039] The audio signal to be processed is acquired, and the fundamental frequency is extracted from the audio signal to be processed to obtain the fundamental frequency trajectory;

[0040] The intonation analysis of the first fundamental frequency sequence corresponding to the silence preamble in the fundamental frequency trajectory is performed to obtain the intonation type of the silence preamble;

[0041] Tone analysis is performed on the second fundamental frequency sequence corresponding to each of the multiple phonemes in the fundamental frequency trajectory to obtain the tone type of each phoneme.

[0042] The tone type is determined based on the intonation type of the silent lead-in, the intonation types of multiple phonemes, the short-time energy of multiple audio frames in the audio signal to be processed, and the pronunciation duration of multiple phonemes.

[0043] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0044] The audio signal to be processed is acquired, and the fundamental frequency is extracted from the audio signal to be processed to obtain the fundamental frequency trajectory;

[0045] The intonation analysis of the first fundamental frequency sequence corresponding to the silence preamble in the fundamental frequency trajectory is performed to obtain the intonation type of the silence preamble;

[0046] Tone analysis is performed on the second fundamental frequency sequence corresponding to each of the multiple phonemes in the fundamental frequency trajectory to obtain the tone type of each phoneme.

[0047] The tone type is determined based on the intonation type of the silent lead-in, the intonation types of multiple phonemes, the short-time energy of multiple audio frames in the audio signal to be processed, and the pronunciation duration of multiple phonemes.

[0048] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0049] The audio signal to be processed is acquired, and the fundamental frequency is extracted from the audio signal to be processed to obtain the fundamental frequency trajectory;

[0050] The intonation analysis of the first fundamental frequency sequence corresponding to the silence preamble in the fundamental frequency trajectory is performed to obtain the intonation type of the silence preamble;

[0051] Tone analysis is performed on the second fundamental frequency sequence corresponding to each of the multiple phonemes in the fundamental frequency trajectory to obtain the tone type of each phoneme.

[0052] The tone type is determined based on the intonation type of the silent lead-in, the intonation types of multiple phonemes, the short-time energy of multiple audio frames in the audio signal to be processed, and the pronunciation duration of multiple phonemes.

[0053] The aforementioned audio intonation recognition method, apparatus, computer equipment, computer-readable storage medium, and computer program product acquire the audio signal to be processed, extract the fundamental frequency from the audio signal to obtain the fundamental frequency trajectory, perform intonation analysis on the first fundamental frequency sequence corresponding to the silence leading segment in the fundamental frequency trajectory to obtain the intonation type of the silence leading segment, and can accurately identify subtle intonation features before silence; perform intonation analysis on the second fundamental frequency sequences corresponding to multiple phonemes in the fundamental frequency trajectory to obtain the intonation type of each phoneme, and can achieve more refined intonation recognition; this method, through silence leading segment... The dual-granularity intonation analysis, combining lead-segment analysis and phoneme-level analysis, avoids the loss of local intonation details caused by analyzing entire speech segments, thus improving the accuracy of audio intonation recognition. The tone type is determined based on the intonation type of the silence lead-segment, the intonation types of multiple phonemes, the short-time energy of multiple audio frames in the audio signal to be processed, and the pronunciation duration of each phoneme. This method, which uses multi-dimensional features to jointly determine the tone type, avoids misjudgment based on a single feature, improves the accuracy of tone determination, and is suitable for scenarios with high intonation recognition requirements, thereby enhancing the naturalness and expressiveness of voice interaction. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is an application environment diagram of the audio intonation recognition method in one embodiment;

[0056] Figure 2 This is a flowchart illustrating an audio intonation recognition method in one embodiment;

[0057] Figure 3 This is a structural block diagram of an audio intonation recognition device in one embodiment;

[0058] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0060] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0061] In fields such as speech synthesis, speech recognition, dialogue systems, and affective computing, traditional mainstream technical solutions include: statistical model-based intonation analysis, such as using HMM and GMM to model the fundamental frequency profile, but the model has poor generalization ability and insufficient adaptability to individual differences and complex intonation patterns; deep learning-based end-to-end affective recognition, such as using CNN, RNN and other networks to directly extract features from spectrograms for classification, but the interpretability is poor and it is difficult to accurately control the details of intonation generation; and rule-driven acoustic event detection, which sets threshold rules such as energy and zero-crossing rate for specific events (such as laughter) for detection, but the rule design is cumbersome and it is difficult to cover diverse expressions.

[0062] Traditional audio intonation recognition methods suffer from the following significant problems: Coarse-grained intonation analysis: Traditional methods often focus on the overall intonation of a sentence or keyword stress, lacking fine-grained intonation trend analysis at the level of silent preludes and phoneme levels. They fail to accurately capture subtle intonation changes such as rising intonation at the end of interrogative sentences and falling intonation at the end of declarative sentences, resulting in mechanical and unnatural synthesized speech. Limited and unfused features: Most systems rely solely on fundamental frequency for intonation judgment, failing to effectively combine acoustic features such as energy and duration. They cannot recognize complex tones like emphasis and exclamation, and struggle to support rich emotional label recognition. Misjudgment of intonation at silence points: Traditional methods often treat silent segments simply as silent intervals, ignoring the intonation progression of the speech segment before the silence begins. While traditional methods play a crucial role in semantic understanding (such as distinguishing between questions and statements), they are prone to misjudgment due to discontinuous F0 sequences or interference from leading silences. Emotion and event labeling rely on text or manual intervention: traditional emotion recognition is largely based on text content or requires extensive manual annotation of speech data, failing to automatically and finely identify rich paralinguistic events and emotional nuances such as [Laughs], [sarcastic], and [excited] directly from acoustic signals. Furthermore, the system modules are fragmented, lacking a unified analytical framework: intonation analysis, emotion recognition, and event detection are typically implemented by independent modules, with no feature sharing and no collaborative decision-making, resulting in low overall recognition efficiency and limited effectiveness. Therefore, traditional methods fail to systematically address issues such as multi-level intonation analysis, multi-feature fusion, and fine-grained intonation determination at silent locations, leading to low accuracy in audio intonation recognition and limited improvement in the naturalness and expressiveness of voice interaction.

[0063] Based on this, the audio intonation recognition method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated on server 104 or placed on the cloud or other network servers. This embodiment uses the method applied to terminal 102 as an example for illustration. It is understood that this method can also be applied to server 104, and can also be applied to systems including terminals and servers, and implemented through the interaction between the terminal and the server. Terminal 102 acquires the audio signal to be processed, extracts the fundamental frequency of the audio signal to be processed, and obtains the fundamental frequency trajectory; performs intonation analysis on the first fundamental frequency sequence corresponding to the silence prelude in the fundamental frequency trajectory to obtain the intonation type of the silence prelude; performs intonation analysis on the second fundamental frequency sequences corresponding to each of the multiple phonemes in the fundamental frequency trajectory to obtain the intonation type of each of the multiple phonemes; and determines the tone type based on the intonation type of the silence prelude, the intonation types of each of the multiple phonemes, the short-time energy of multiple audio frames in the audio signal to be processed, and the pronunciation duration of each of the multiple phonemes. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, and projection equipment. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0064] In one exemplary embodiment, such as Figure 2 As shown, an audio intonation recognition method is provided, which can be applied to... Figure 1 Taking terminal 102 as an example, the explanation includes the following steps 202 to 208. Wherein:

[0065] Step 202: Obtain the audio signal to be processed, extract the fundamental frequency of the audio signal to be processed, and obtain the fundamental frequency trajectory.

[0066] The audio signal to be processed refers to the audio signal that needs to be analyzed for intonation. It can be the raw audio signal received in various scenarios, including voice interaction scenarios and voice recognition scenarios. The audio signal to be processed can be the audio signal received in real time, or it can be the audio signal obtained from a pre-stored audio library.

[0067] Fundamental frequency extraction refers to extracting the fundamental frequency value (i.e., F0 value) from the audio signal to be processed. F0 prediction models such as CREPE (Convolutional Representation for Pitch Estimation) and PYIN (Probabilistic YIN) can be used. After segmenting the audio signal into frames, the fundamental frequency value is calculated frame by frame. Invalid F0 values ​​are filtered out, and only valid F0 values ​​are retained, resulting in multiple fundamental frequency values ​​ordered by time, i.e., the fundamental frequency trajectory. The fundamental frequency trajectory reflects the continuous change trend of the pitch of the audio signal over time.

[0068] Step 204: Perform intonation analysis on the first fundamental frequency sequence corresponding to the silence prelude in the fundamental frequency trajectory to obtain the intonation type of the silence prelude.

[0069] The silence prelude refers to the audio segment immediately preceding the silence segment in the audio signal to be processed; that is, the audio segment about to enter a pause or silence. The terminal can use an audio signal earlier than the silence start frame in the audio signal to be processed as the silence prelude. The silence prelude is used to determine the true intonation at the end of a sentence or before a pause, such as the rising intonation at the end of an interrogative sentence or the falling intonation at the end of a declarative sentence.

[0070] The terminal extracts a segment of fundamental frequency sequence containing multiple fundamental frequency values ​​corresponding to the silence prelude from the fundamental frequency trajectory, obtaining the first fundamental frequency sequence. The terminal performs intonation analysis on the first fundamental frequency sequence, for example, by calculating statistical data such as the mean, difference mean, and slope of the multiple fundamental frequency values ​​in the first fundamental frequency sequence, and determines the intonation type based on at least one statistical data point. Intonation types include rising intonation, falling intonation, and stable intonation. Through intonation analysis of the first fundamental frequency sequence, the intonation type of the silence prelude is obtained.

[0071] Step 206: Perform intonation analysis on the second fundamental frequency sequence corresponding to each of the multiple phonemes in the fundamental frequency trajectory to obtain the intonation type of each of the multiple phonemes.

[0072] Here, a phoneme refers to the smallest unit of articulation in the audio signal to be processed, such as an initial consonant or a final vowel. The terminal obtains the phoneme-level time boundary information through forced alignment, including the start and end times of each of the multiple phonemes. Based on the time boundary information of each of the multiple phonemes, the terminal extracts the second fundamental frequency sequence corresponding to each of the multiple phonemes from the fundamental frequency trajectory.

[0073] The terminal performs intonation analysis on the second fundamental frequency sequence corresponding to each phoneme to obtain the intonation type of a single phoneme. The intonation type of a single phoneme can include intonation rising, intonation falling, and intonation remaining stable. If the pronunciation duration of any phoneme exceeds a preset duration, the intonation type can also include intonation rising continuously, intonation falling continuously, intonation rising then falling, and intonation falling then rising.

[0074] Step 208: Determine the tone type based on the tone type of the silent lead-in segment, the tone types of each of the multiple phonemes, the short-time energy of multiple audio frames in the audio signal to be processed, and the pronunciation duration of each of the multiple phonemes.

[0075] Short-time energy refers to the energy level of a single audio frame, reflecting the intensity of the sound within that frame. Higher short-time energy indicates a louder, more intense sound.

[0076] Pronunciation duration refers to the duration of a single phoneme. For emphasis, hesitation, exclamation, and other emotional expressions, the pronunciation duration of a phoneme may be longer.

[0077] The tone type refers to the tone of the audio signal to be processed, which can include emphatic tone, as well as interrogative tone, affirmative tone, exclamatory tone, declarative tone, etc.

[0078] By utilizing the energy characteristics, intonation characteristics, and pronunciation duration of the audio signal to be processed, the tone type can be determined with high accuracy. In scenarios such as voice interaction, it can adapt to individual differences and complex intonation patterns, thereby improving the naturalness and expressiveness of voice interaction.

[0079] In some embodiments, when the intonation type of the silent lead segment is an up-pitched type, there is an up-pitched type in the intonation types of multiple phonemes, the short-time energy of multiple speech frames shows an increasing trend, and the pronunciation duration of multiple phonemes shows a lengthening trend, the tone type is determined to be an emphatic tone type.

[0080] In some embodiments, the energy peaks in the short-time energy of multiple speech frames are determined, the ratio of the energy peaks to the pronunciation duration of each of the multiple phonemes is calculated, the ratios are normalized and weighted with fundamental frequency feature values ​​to obtain a weighted result, and the tone type is determined based on the weighted result and a preset interval. The weights can be configured according to different languages ​​and scenarios, and the fundamental frequency feature values ​​can be differential mean, fundamental frequency value, etc.

[0081] In the aforementioned audio intonation recognition method, the fundamental frequency of the audio signal to be processed is extracted to obtain the fundamental frequency trajectory. Intonation analysis is then performed on the first fundamental frequency sequence corresponding to the silence prelude in the fundamental frequency trajectory to obtain the intonation type of the silence prelude, enabling accurate identification of subtle intonation features before silence. Intonation analysis is also performed on the second fundamental frequency sequences corresponding to multiple phonemes in the fundamental frequency trajectory to obtain the intonation type of each phoneme, achieving more refined intonation recognition. This dual-granularity intonation analysis, combining silence prelude analysis and phoneme-level analysis, avoids the loss of local intonation details caused by analyzing the entire speech segment, thus improving the accuracy of audio intonation recognition. The tone type is determined based on the intonation type of the silence prelude, the intonation types of multiple phonemes, the short-time energy of multiple audio frames in the audio signal to be processed, and the pronunciation duration of each phoneme. This method, which uses multi-dimensional features to jointly determine the tone type, avoids misjudgment based on a single feature, improves the accuracy of tone determination, and is suitable for scenarios with high intonation recognition requirements, thus enhancing the naturalness and expressiveness of voice interaction.

[0082] In an exemplary embodiment, pitch analysis is performed on the first fundamental frequency sequence corresponding to the silence prelude in the fundamental frequency trajectory to obtain the pitch type of the silence prelude, including the following steps: obtaining the difference between adjacent frames in the first fundamental frequency sequence corresponding to the silence prelude in the fundamental frequency trajectory; taking the mean of the differences between multiple adjacent frames in the first fundamental frequency sequence as the mean difference value corresponding to the first fundamental frequency sequence; and determining the pitch type of the silence prelude based on the mean difference value corresponding to the first fundamental frequency sequence and a preset difference threshold.

[0083] The first fundamental frequency sequence includes the fundamental frequency values ​​of multiple audio frames. The difference between adjacent frames refers to the difference in the fundamental frequency values ​​of adjacent audio frames in the first fundamental frequency sequence, which is used to characterize the pitch change between adjacent frames.

[0084] The difference mean is calculated by summing the differences between all adjacent frames in the first fundamental frequency sequence and then dividing by the number of differences. The difference mean is used to characterize the average pitch variation trend of a silence leading section.

[0085] The preset difference threshold is a manually set threshold, which can be a single value or a range of values. The terminal determines the tone type of the silence leading segment by comparing the mean difference value corresponding to the first fundamental frequency sequence with the preset difference threshold.

[0086] In this embodiment, by using the mean difference as the standard for intonation recognition, it is possible to filter out minor tremors, reflect the overall pitch trend of the silent leading segment, and objectively and accurately identify the subtle intonation changes of the silent leading segment.

[0087] In an exemplary embodiment, determining the tone type of the silence prelude based on the mean difference value corresponding to the first fundamental frequency sequence and a preset difference threshold includes: determining the tone type of the silence prelude to be an rising tone type when the mean difference value corresponding to the first fundamental frequency sequence is greater than the preset difference threshold; determining the tone type of the silence prelude to be a stable tone type when the mean difference value corresponding to the first fundamental frequency sequence is equal to the preset difference threshold; and determining the tone type of the silence prelude to be a falling tone type when the mean difference value corresponding to the first fundamental frequency sequence is less than the preset difference threshold.

[0088] If the mean difference is greater than the preset difference threshold, it indicates that the pitch of the first fundamental frequency sequence has an overall upward trend, and the intonation type is determined to be an rising tone. If the mean difference is less than the preset difference threshold, it indicates that the pitch of the first fundamental frequency sequence has an overall downward trend, and the intonation type is determined to be a falling tone. If the mean difference is equal to the preset difference threshold, it indicates that the pitch of the first fundamental frequency sequence has an overall stable trend, and the intonation type is determined to be a stable tone.

[0089] In some embodiments, the preset difference threshold can be a numerical range consisting of a first threshold and a second threshold, where the first threshold is less than the second threshold. Accordingly, when the mean difference is greater than the second threshold, the intonation type is determined to be an rising intonation type; when the mean difference is within the numerical range of the first and second thresholds, the intonation type is determined to be a stable intonation type; and when the mean difference is less than the first threshold, the intonation type is determined to be a falling intonation type.

[0090] In this embodiment, by comparing the mean difference value corresponding to the first fundamental frequency sequence with a preset difference threshold, subjective judgment bias can be avoided, and the quantitative indicator can be converted into a type indicator, which can be adapted to different pronunciation individuals and scenarios, thereby improving the accuracy of intonation recognition.

[0091] In an exemplary embodiment, the audio tone recognition method further includes: determining the scene type of the silence prelude when the mean difference value corresponding to the first fundamental frequency sequence is less than a preset difference threshold and greater than a preset leniency threshold; correcting the tone type of the silence prelude from a tone falling type to a tone rising type when the scene type is a cross-silence rising type and the mean difference value corresponding to the first fundamental frequency segment later than the silence prelude in the fundamental frequency trajectory is greater than the mean difference value corresponding to the second fundamental frequency segment earlier than the silence prelude in the fundamental frequency trajectory; re-extracting the silence prelude from a position in the audio signal to be processed earlier than a preset duration of the silence prelude to obtain an updated silence prelude; and correcting the tone type of the silence prelude from a tone falling type to a tone rising type when the mean difference value corresponding to the updated silence prelude is greater than a preset difference threshold.

[0092] In this embodiment, if the mean difference value corresponding to the first fundamental frequency sequence is less than a preset difference threshold, the terminal determines the tone type as a falling tone type. However, if the mean difference value is greater than a preset leniency threshold, for example, in cases involving a cross-silence rising tone type or a leading silence interference type, the actual tone type may be a rising tone type due to silence pauses or leading silence interference. Therefore, this application proposes to further judge scenarios where the mean difference value is less than the preset difference threshold but greater than the preset leniency threshold, in order to improve the accuracy of audio tone recognition.

[0093] The terminal determines whether the scenario type of the silence leading segment is a cross-silence rising type or a leading silence interference type. If it is a cross-silence rising type, it further determines whether the mean difference value corresponding to the first fundamental frequency segment is greater than the mean difference value corresponding to the second fundamental frequency segment. The first fundamental frequency segment is the fundamental frequency sequence in the fundamental frequency trajectory that is later than the silence leading segment, and the second fundamental frequency segment is the fundamental frequency sequence in the fundamental frequency trajectory that is earlier than the silence leading segment. If the scenario type of the silence leading segment is true, and the mean difference value corresponding to the first fundamental frequency segment is greater than the mean difference value corresponding to the second fundamental frequency segment, it indicates that both the beginning and end of the silence leading segment have an rising trend, and the overall pitch trend in the latter segment is higher than in the former segment. The terminal then corrects the intonation type from a falling intonation type to a rising intonation type. For example,

[0094] If the interference is a leading silence, it indicates that there is an actual silent zone before the silent leading segment. This silent zone causes the mean difference value corresponding to the first fundamental frequency sequence to be too small. In this case, the silent leading segment is re-trunculated at a position earlier than the preset duration of the silent leading segment to skip the silent zone. If the recalculated mean difference value is greater than the preset leniency threshold, the tone type is corrected from a falling tone type to a rising tone type. For example, K points (e.g., 240ms data) are intercepted starting from a point further away (e.g., 540ms) from the silent starting frame. This audio segment is used as the updated silent starting frame, and the tone type is re-analyzed.

[0095] In this embodiment, by identifying scenarios involving both silent rising and leading silence interference, and employing different correction methods in the two scenarios, the system adapts to misjudgment issues in different scenarios such as silent interruption and leading silence interference, thereby ensuring the accuracy of intonation recognition.

[0096] In an exemplary embodiment, determining the scenario type of the silence prelude includes: performing pitch analysis on a first fundamental frequency segment in the fundamental frequency trajectory that is later than the silence prelude to obtain a first pitch type; performing pitch analysis on a second fundamental frequency segment in the fundamental frequency trajectory that is earlier than the silence prelude to obtain a second pitch type; if both the first pitch type and the second pitch type are pitch-rising types, determining the scenario type of the silence prelude as a cross-silence-rising type; and if, in the audio segment of the audio signal to be processed that is earlier than the silence prelude, there is a reference audio segment with an average energy value less than a preset energy threshold, determining the prelude silence interference type of the scenario type of the silence prelude.

[0097] Specifically, the intonation analysis of the first and second fundamental frequency segments can be performed using the intonation analysis methods described in steps 204 or 206 to obtain the intonation type of the first and second fundamental frequency segments. When both the first and second intonation types are rising intonation types (i.e., the preceding and following sections of the silent leading segment show an upward trend, interrupted at the silence), the mean difference value corresponding to the silent leading segment is less than the preset difference threshold, leading to a misjudgment as a falling intonation type. This scenario is thus identified as a cross-silence rising intonation type.

[0098] The average energy value refers to the average short-time energy of multiple audio frames in a certain audio segment (such as a reference audio segment). It is used to determine whether the audio segment is valid speech. If the average energy value is lower than the preset energy threshold, it means that the audio segment is a silent zone.

[0099] The reference audio segment is the silent area in the audio to be processed that precedes the silent prelude. The existence of this silent area causes the mean difference value corresponding to the silent prelude to be less than the preset difference threshold, thus misjudging it as a tone drop type. This scenario is identified as a prelude silence interference type.

[0100] In this embodiment, by clearly defining the judgment criteria for two scenarios—across silence rising and leading silence interference—it is beneficial to quickly and accurately identify the two scenario types, and to correct scenarios with falling intonation, thus ensuring the accuracy of intonation recognition.

[0101] In an exemplary embodiment, intonation analysis is performed on the second fundamental frequency sequences corresponding to multiple phonemes in the fundamental frequency trajectory to obtain the intonation type of each phoneme, including: for any phoneme of the audio signal to be processed, dividing the second fundamental frequency sequence corresponding to the current phoneme into a first fundamental frequency subsequence and a second fundamental frequency subsequence; and determining the intonation type of the current phoneme based on the difference sum of the first fundamental frequency subsequence and the second fundamental frequency subsequence.

[0102] In this process, the terminal divides the second fundamental frequency sequence corresponding to any phoneme into a first fundamental frequency subsequence and a second fundamental frequency subsequence in chronological order.

[0103] Taking the example that the first fundamental frequency subsequence is earlier than the second fundamental frequency subsequence in the second fundamental frequency sequence, the first fundamental frequency subsequence is the fundamental frequency sequence corresponding to the first half of the phoneme, and the second fundamental frequency subsequence is the fundamental frequency sequence corresponding to the second half of the phoneme.

[0104] The terminal calculates the difference between adjacent frames in the first fundamental frequency subsequence, sums the differences between multiple adjacent frames in the first fundamental frequency subsequence to obtain the sum of differences of the first fundamental frequency subsequence, sum_front, which is used to characterize the overall pitch change amplitude of the first fundamental frequency subsequence. Similarly, the terminal sums the differences between multiple adjacent frames in the second fundamental frequency subsequence to obtain the sum of differences of the second fundamental frequency subsequence, sum_rear, which is used to characterize the overall pitch change amplitude of the second fundamental frequency subsequence.

[0105] For example, if sum_front > preset value (e.g., the preset value can be 0) and sum_rear > preset value, the intonation type of the current phoneme is determined to be a continuous rising type; if sum_front < preset value and sum_rear < preset value, the intonation type of the current phoneme is determined to be a continuous falling type; if sum_front > preset value and sum_rear < preset value, the intonation type of the current phoneme is determined to be a rising-then-falling type; if sum_front < preset value and sum_rear > preset value, the intonation type of the current phoneme is determined to be a falling-then-rising type; and if sum_front = preset value and sum_rear = preset value, the intonation type of the current phoneme is determined to be a steady type.

[0106] In some embodiments, if the pronunciation duration of any phoneme exceeds a preset duration (e.g., a long phoneme exceeding L milliseconds), the second fundamental frequency sequence corresponding to the current phoneme is divided into a first fundamental frequency subsequence and a second fundamental frequency subsequence; the intonation type of the current phoneme is determined based on the sum of the differences between the first fundamental frequency subsequence and the second fundamental frequency subsequence.

[0107] In this embodiment, by splitting a single phoneme into two fundamental frequency sequences, the details of intonation changes within the phoneme can be accurately captured based on the pitch changes of the two sequences, thereby improving the precision of phoneme intonation determination.

[0108] In an exemplary embodiment, the audio intonation recognition method further includes: determining the intonation level of a silence prelude based on the mean difference value corresponding to the silence prelude and the threshold intervals for each of the multiple intonation levels; determining the intonation level of each of the multiple phonemes in the audio signal to be processed based on the mean difference value of each of the multiple phonemes and the threshold intervals for each of the multiple intonation levels; converting the intonation level of the silence prelude, the intonation level of each of the multiple phonemes, the short-time energy of multiple audio frames in the audio signal to be processed, and the pronunciation duration of each of the multiple phonemes into feature vectors; and performing emotion classification based on the feature vectors using a pre-trained emotion classification model to obtain the emotion classification result.

[0109] Intonation level refers to the strength of intonation changes. Different intonation levels correspond to different threshold ranges. If the mean difference value of the preceding silent segment falls within the threshold range corresponding to any intonation level, then that intonation level is determined as the intonation level of the preceding silent segment. Similarly, if the mean difference value of the current phoneme falls within the threshold range corresponding to any intonation level, then that intonation level is determined as the intonation level of the current phoneme. Intonation levels can include severe, moderate, and mild. The mean difference value of the current phoneme can be determined using the second fundamental frequency sequence corresponding to the current phoneme.

[0110] The terminal converts intonation, short-term energy, and pronunciation duration into computer-recognizable numerical vectors, obtaining feature vectors. These feature vectors are then input into a pre-trained sentiment classification model for sentiment classification, yielding the sentiment classification result output by the pre-trained model. The sentiment classification model is pre-trained using a large amount of speech signals and labeled sentiment tags to create an initial classification model. This initial classification model can include SVM (Support Vector Machine), random forest, or neural network models, among others.

[0111] The emotion classification results indicate the emotion type of the audio signal to be processed, such as [curious], [excited], [laughs], etc.

[0112] In this embodiment, by using multiple threshold ranges for different intonation levels, the difference mean is converted into a quantifiable intonation level, which can capture subtle intonation differences. Furthermore, the feature vectors converted from intonation level, short-term energy, and pronunciation duration meet the input requirements of the model, and can support rich emotion type recognition for complex tones such as emphasis and exclamation.

[0113] To illustrate the audio intonation recognition method and its effects in this solution in detail, the following is a detailed implementation example:

[0114] 1. Audio preprocessing and feature extraction.

[0115] Audio input: Receives the raw speech signal as the audio signal to be processed.

[0116] Fundamental frequency extraction: Using F0 prediction models such as CREPE and PYIN, the fundamental frequency trajectory is extracted from the audio signal to be processed.

[0117] Time alignment: Using a forced alignment model (such as an HMM-based or end-to-end model), phoneme-level time boundary information is obtained, and the starting frame position of the silence segment is identified.

[0118] Energy and duration calculation: Calculate the short-time energy of speech frames, as well as the pronunciation duration of each phoneme, syllable, or word.

[0119] 2. Analysis of the degree of intonation rise before silence, used to resolve the issue of intonation at the silence point.

[0120] This step aims to analyze the intonation of the last short segment of speech before the silence begins, in order to determine whether the silence is a "pause after an upward inflection of a question" or a "silence after a downward inflection of a statement."

[0121] Data extraction: Taking the silence start frame as the reference point, backtrack N milliseconds and extract the corresponding base frequency sequence to obtain the first base frequency sequence F0_silence.

[0122] Data cleaning and normalization: Remove zero or unreliable values ​​from the first fundamental frequency sequence F0_silence, calculate the mean μ of the effective F0 values, and perform speaker relative normalization to eliminate speaker pitch differences, resulting in the normalized first fundamental frequency sequence: F0_norm=F0_silence / μ.

[0123] Trend classification: Calculate the first-order difference of the normalized first fundamental frequency sequence, diff = F0_norm[i] - F0_norm[i-1], and calculate its average value to obtain the mean difference value avg_diff corresponding to the first fundamental frequency sequence.

[0124] Based on the comparison between avg_diff and the preset difference threshold Th_flat, a primary classification is performed: if avg_diff > Th_flat, the initial determination of the intonation type is an rising intonation type. If avg_diff < - Th_flat, the initial determination of the intonation type is a falling intonation type. Otherwise, the initial determination of the intonation type is a stable intonation type.

[0125] Correction of misjudgments related to pitch drop.

[0126] If the initial determination of the intonation type is a falling intonation type, but avg_diff is greater than a preset lenient threshold, there may be two scenarios of misjudgment:

[0127] Scenario A (Cross-Silence Rising Pitch Type): The pitch is actually rising before and after the silence, but the F0 is interrupted at the silence, causing a large negative value in the difference mean calculation. Correction method: Take the last M F0 points before the silence (e.g., the last 240ms of data) and recalculate avg_diff. If the new avg_diff > Th_flat, then correct the pitch type to rising pitch.

[0128] Scenario B (Leading Silence Interference Type): A silent zone already exists before the silence marker, and its F0 drop is meaningless. Correction Method: Starting from a point further forward from the silence start point (e.g., 540ms), extract K points (e.g., 240ms of data), use this audio segment as the updated silence start frame, and reanalyze the intonation type.

[0129] Intonation level quantification: For sequences that are ultimately determined to be of the rising tone type or the first depression followed by a rise, the intonation level is divided according to its avg_diff value, such as multiple intonation level levels such as r_0, r_1, r_2, etc.

[0130] Threshold setting logic: By performing statistical analysis on a large-scale speech database (containing interrogative sentences, declarative sentences, etc. with different intonations), the threshold range corresponding to different intonation levels is determined.

[0131] 3. Analysis of intonation trends and degree at the phoneme level.

[0132] Obtaining the second fundamental frequency sequence corresponding to the phoneme: Based on the alignment result, extract the F0 subsequence corresponding to each phoneme as the second fundamental frequency sequence.

[0133] Intraphone trend analysis: For long phonemes with a pronunciation duration exceeding L milliseconds, their second fundamental frequency sequence is divided into two equal sub-intervals: front and back. The sums of differences between the two sub-intervals are calculated: sum_front and sum_rear.

[0134] A fine classification is performed based on the symbols of both: For example, if sum_front > preset value (e.g., the preset value can be 0) and sum_rear > preset value, the intonation type of the current phoneme is determined to be a continuous rising type; if sum_front < preset value and sum_rear < preset value, the intonation type of the current phoneme is determined to be a continuous falling type; if sum_front > preset value and sum_rear < preset value, the intonation type of the current phoneme is determined to be a rising-then-falling type; if sum_front < preset value and sum_rear > preset value, the intonation type of the current phoneme is determined to be a falling-then-rising type; and if sum_front = preset value and sum_rear = preset value, the intonation type of the current phoneme is determined to be a steady type.

[0135] Intonation level calculation: For each intonation type, the intonation level of each phoneme is calculated by combining its overall avg_diff value and referring to the intonation level quantification method in Part 2.

[0136] 4. Use energy and duration features to identify emphasis. When the three features of rising fundamental frequency, increased energy, and prolonged duration appear simultaneously and significantly, the tone type at that point is determined to be emphasis.

[0137] 5. Multi-feature fusion classification of emotion and event tags.

[0138] Feature concatenation: The intonation level of the silent introductory segment, the intonation level of each of the multiple phonemes, energy and duration features, and possible spectral features (such as MFCC) are concatenated into a comprehensive feature vector.

[0139] Classifier: Input the feature vector into a pre-trained sentiment classification model (such as SVM, random forest or neural network).

[0140] Label output: The model output is one or more emotion / event labels corresponding to the audio signal to be processed, such as [curious], [excited], [Laughs], etc.

[0141] The aforementioned audio intonation recognition method acquires the audio signal to be processed, extracts the fundamental frequency to obtain the fundamental frequency trajectory, and performs intonation analysis on the first fundamental frequency sequence corresponding to the silence prelude in the fundamental frequency trajectory to obtain the intonation type of the silence prelude, which can accurately identify subtle intonation features before silence. It then performs intonation analysis on the second fundamental frequency sequences corresponding to multiple phonemes in the fundamental frequency trajectory to obtain the intonation type of each phoneme, achieving more refined intonation recognition. This dual-granularity intonation analysis, combining silence prelude analysis and phoneme-level analysis, avoids the loss of local intonation details caused by analyzing the entire speech segment, thus improving the accuracy of audio intonation recognition. Based on the intonation type of the silence prelude, the intonation types of multiple phonemes, the short-time energy of multiple audio frames in the audio signal to be processed, and the pronunciation duration of multiple phonemes, the tone type is determined. This method, which uses multi-dimensional features to jointly determine the tone type, avoids misjudgment based on a single feature, improves the accuracy of tone determination, and is suitable for scenarios with high intonation recognition requirements, thus improving the naturalness and expressiveness of voice interaction.

[0142] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0143] Based on the same inventive concept, this application also provides an audio intonation recognition device for implementing the audio intonation recognition method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more audio intonation recognition device embodiments provided below can be found in the limitations of the audio intonation recognition method described above, and will not be repeated here.

[0144] In one exemplary embodiment, such as Figure 3 As shown, an audio intonation recognition device 300 is provided, including: an acquisition module 320, a first analysis module 340, a second analysis module 360, and a determination module 380.

[0145] The acquisition module 320 is used to acquire the audio signal to be processed, extract the fundamental frequency of the audio signal to be processed, and obtain the fundamental frequency trajectory.

[0146] The first analysis module 340 is used to perform intonation analysis on the first fundamental frequency sequence corresponding to the silence leading segment in the fundamental frequency trajectory to obtain the intonation type of the silence leading segment.

[0147] The second analysis module 360 ​​is used to perform intonation analysis on the second fundamental frequency sequence corresponding to each of the multiple phonemes in the fundamental frequency trajectory, and to obtain the intonation type of each of the multiple phonemes.

[0148] The determination module 380 is used to determine the tone type based on the tone type of the silent prelude, the tone types of multiple phonemes, the short-time energy of multiple audio frames in the audio signal to be processed, and the pronunciation duration of multiple phonemes.

[0149] The aforementioned audio intonation recognition device acquires the audio signal to be processed, extracts the fundamental frequency to obtain the fundamental frequency trajectory, and performs intonation analysis on the first fundamental frequency sequence corresponding to the silence prelude in the fundamental frequency trajectory to obtain the intonation type of the silence prelude, enabling accurate identification of subtle intonation features before silence. It then performs intonation analysis on the second fundamental frequency sequences corresponding to multiple phonemes in the fundamental frequency trajectory to obtain the intonation type of each phoneme, achieving more refined intonation recognition. This dual-granularity intonation analysis, combining silence prelude analysis and phoneme-level analysis, avoids the loss of local intonation details caused by analyzing the entire speech segment, improving the accuracy of audio intonation recognition. Based on the intonation type of the silence prelude, the intonation types of multiple phonemes, the short-time energy of multiple audio frames in the audio signal to be processed, and the pronunciation duration of each phoneme, the tone type is determined. This method of determining tone type through multi-dimensional features avoids misjudgment based on a single feature, improves the accuracy of tone determination, and is suitable for scenarios with high intonation recognition requirements, thus enhancing the naturalness and expressiveness of voice interaction.

[0150] In one embodiment, the tone analysis is performed on the first fundamental frequency sequence corresponding to the silence prelude in the fundamental frequency trajectory to obtain the tone type of the silence prelude. The first analysis module 340 is further configured to: obtain the difference between adjacent frames in the first fundamental frequency sequence corresponding to the silence prelude in the fundamental frequency trajectory; take the mean of the differences between multiple adjacent frames in the first fundamental frequency sequence as the mean difference value corresponding to the first fundamental frequency sequence; and determine the tone type of the silence prelude based on the mean difference value corresponding to the first fundamental frequency sequence and a preset difference threshold.

[0151] In one embodiment, the tone type of the silence prelude is determined based on the mean difference value corresponding to the first fundamental frequency sequence and a preset difference threshold. The first analysis module 340 is further configured to: determine the tone type of the silence prelude as an rising tone type when the mean difference value corresponding to the first fundamental frequency sequence is greater than the preset difference threshold; determine the tone type of the silence prelude as a stable tone type when the mean difference value corresponding to the first fundamental frequency sequence is equal to the preset difference threshold; and determine the tone type of the silence prelude as a falling tone type when the mean difference value corresponding to the first fundamental frequency sequence is less than the preset difference threshold.

[0152] In one embodiment, the first analysis module 340 is further configured to: determine the scenario type of the silence prelude when the mean difference value corresponding to the first fundamental frequency sequence is less than a preset difference threshold and greater than a preset leniency threshold; when the scenario type is a cross-silence rising type and the mean difference value corresponding to the first fundamental frequency segment later than the silence prelude in the fundamental frequency trajectory is greater than the mean difference value corresponding to the second fundamental frequency segment earlier than the silence prelude in the fundamental frequency trajectory, correct the tone type of the silence prelude from a falling tone type to a rising tone type; when the scenario type is a leading silence interference type, re-extract the silence prelude at a position in the audio signal to be processed earlier than the preset duration of the silence prelude to obtain an updated silence prelude; when the mean difference value corresponding to the updated silence prelude is greater than the preset difference threshold, correct the tone type of the silence prelude from a falling tone type to a rising tone type.

[0153] In one embodiment, to determine the scene type of the silence prelude, the first analysis module 340 is further configured to: perform pitch analysis on the first fundamental frequency segment in the fundamental frequency trajectory that is later than the silence prelude to obtain a first pitch type; perform pitch analysis on the second fundamental frequency segment in the fundamental frequency trajectory that is earlier than the silence prelude to obtain a second pitch type; if both the first pitch type and the second pitch type are pitch rising types, determine that the scene type of the silence prelude is a cross-silence rising type; if there is a reference audio segment in the audio segment of the audio signal to be processed that is earlier than the silence prelude and has an average energy value less than a preset energy threshold, determine the prelude silence interference type of the scene type of the silence prelude.

[0154] In one embodiment, intonation analysis is performed on the second fundamental frequency sequence corresponding to each of the multiple phonemes in the fundamental frequency trajectory to obtain the intonation type of each of the multiple phonemes. The second analysis module 360 ​​is further configured to: for any phoneme of the audio signal to be processed, divide the second fundamental frequency sequence corresponding to the current phoneme into a first fundamental frequency subsequence and a second fundamental frequency subsequence; and determine the intonation type of the current phoneme based on the difference sum of the first fundamental frequency subsequence and the second fundamental frequency subsequence.

[0155] In one embodiment, the audio intonation recognition device 300 further includes an emotion classification module, which is used to: determine the intonation level of a silence prelude based on the mean difference value corresponding to the silence prelude and the threshold intervals of multiple intonation levels; determine the intonation level of multiple phonemes in the audio signal to be processed based on the mean difference value of multiple phonemes and the threshold intervals of multiple intonation levels; convert the intonation level of the silence prelude, the intonation level of multiple phonemes, the short-time energy of multiple audio frames in the audio signal to be processed, and the pronunciation duration of multiple phonemes into feature vectors; and perform emotion classification using a pre-trained emotion classification model based on the feature vectors to obtain the emotion classification result.

[0156] Each module in the aforementioned audio intonation recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0157] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements an audio tone recognition method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0158] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0159] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0160] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0161] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0162] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0163] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0164] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0165] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An audio intonation recognition method, characterized in that, The method includes: Acquire the audio signal to be processed, extract the fundamental frequency from the audio signal to obtain the fundamental frequency trajectory; The intonation analysis is performed on the first fundamental frequency sequence corresponding to the silence preamble in the fundamental frequency trajectory to obtain the intonation type of the silence preamble; Tone analysis is performed on the second fundamental frequency sequence corresponding to each of the multiple phonemes in the fundamental frequency trajectory to obtain the tone type of each of the multiple phonemes; The tone type is determined based on the intonation type of the silent prelude, the intonation types of each of the multiple phonemes, the short-time energy of multiple audio frames in the audio signal to be processed, and the pronunciation duration of each of the multiple phonemes.

2. The method according to claim 1, characterized in that, The step of performing intonation analysis on the first fundamental frequency sequence corresponding to the silence leader segment in the fundamental frequency trajectory to obtain the intonation type of the silence leader segment includes: Obtain the difference between adjacent frames in the first fundamental frequency sequence corresponding to the silence preamble in the fundamental frequency trajectory; The mean of the differences between multiple adjacent frames in the first baseband sequence is taken as the mean of the differences corresponding to the first baseband sequence. The intonation type of the silence prelude is determined based on the difference mean value corresponding to the first fundamental frequency sequence and the preset difference threshold.

3. The method according to claim 2, characterized in that, The step of determining the intonation type of the silence leading segment based on the difference mean and preset difference threshold corresponding to the first fundamental frequency sequence includes: If the mean difference value corresponding to the first fundamental frequency sequence is greater than a preset difference threshold, the tone type of the silent leading segment is determined to be an upward tone type. If the mean difference value corresponding to the first fundamental frequency sequence is equal to a preset difference threshold, the intonation type of the silence leading segment is determined to be a stable intonation type. If the mean difference value corresponding to the first fundamental frequency sequence is less than a preset difference threshold, the tone type of the silence leading segment is determined to be a tone falling type.

4. The method according to claim 2, characterized in that, The method further includes: If the mean difference value corresponding to the first fundamental frequency sequence is less than a preset difference threshold and greater than a preset leniency threshold, the scenario type of the silence preamble is determined. If the scenario type is a cross-silence rising type and the mean difference value corresponding to the first fundamental frequency segment in the fundamental frequency trajectory that is later than the silence leading segment is greater than the mean difference value corresponding to the second fundamental frequency segment in the fundamental frequency trajectory that is earlier than the silence leading segment, the tone type of the silence leading segment will be corrected from tone falling type to tone rising type. In the case where the scenario type is a leading silence interference type, the silent leading segment is re-extracted from the audio signal to be processed at a position earlier than the preset duration of the silent leading segment, resulting in an updated silent leading segment; if the mean difference value corresponding to the updated silent leading segment is greater than the preset difference threshold, the tone type of the silent leading segment is corrected from the tone falling type to the tone rising type.

5. The method according to claim 4, characterized in that, Determining the scenario type of the silent lead-in includes: The first fundamental frequency segment in the fundamental frequency trajectory that is later than the silence preamble is analyzed to obtain a first tone type, and the second fundamental frequency segment in the fundamental frequency trajectory that is earlier than the silence preamble is analyzed to obtain a second tone type. When both the first intonation type and the second intonation type are intonation rising types, the scenario type of the silence leading segment is determined to be the cross-silence rising type; If, in the audio signal to be processed, there is a reference audio segment whose average energy is less than a preset energy threshold in an audio segment earlier than the silent prelude, the prelude silence interference type of the scene type of the silent prelude is determined.

6. The method according to claim 1, characterized in that, The step of performing intonation analysis on the second fundamental frequency sequences corresponding to each of the multiple phonemes in the fundamental frequency trajectory to obtain the intonation type of each of the multiple phonemes includes: For any phoneme of the audio signal to be processed, the second fundamental frequency sequence corresponding to the current phoneme is divided into a first fundamental frequency subsequence and a second fundamental frequency subsequence; The intonation type of the current phoneme is determined based on the sum of the differences between the first fundamental frequency subsequence and the second fundamental frequency subsequence.

7. The method according to claim 1, characterized in that, The method further includes: The intonation level of the silent preamble is determined based on the mean difference value corresponding to the silent preamble and the threshold ranges of multiple intonation levels. The intonation level of each phoneme in the audio signal to be processed is determined based on the mean difference of each phoneme and the threshold range of each intonation level in the audio signal to be processed. The intonation level of the silence prelude, the intonation level of each of the multiple phonemes, the short-time energy of multiple audio frames in the audio signal to be processed, and the pronunciation duration of each of the multiple phonemes are converted into feature vectors. Based on the feature vector, sentiment classification is performed using a pre-trained sentiment classification model to obtain the sentiment classification result.

8. An audio intonation recognition device, characterized in that, The device includes: The acquisition module is used to acquire the audio signal to be processed, extract the fundamental frequency from the audio signal to be processed, and obtain the fundamental frequency trajectory. The first analysis module is used to perform intonation analysis on the first fundamental frequency sequence corresponding to the silence preamble in the fundamental frequency trajectory to obtain the intonation type of the silence preamble. The second analysis module is used to perform intonation analysis on the second fundamental frequency sequence corresponding to each of the multiple phonemes in the fundamental frequency trajectory, and to obtain the intonation type of each of the multiple phonemes. The determination module is used to determine the tone type based on the tone type of the silence prelude, the tone types of each of the multiple phonemes, the short-time energy of multiple audio frames in the audio signal to be processed, and the pronunciation duration of each of the multiple phonemes.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images