Fine-grained audio sentiment description generation method and system

Abstract

The application discloses a fine-grained audio emotion description generation method and system, and relates to the field of speech understanding. The method comprises the following steps: adopting at least two audio understanding models to respectively label each original audio and generate initial text descriptions; taking each original audio and the corresponding initial text description as a labeling sample; calculating the semantic similarity between the initial text descriptions of the same original audio to divide the labeling samples into high-consistency samples and low-consistency samples; constructing positive samples, soft negative samples and hard negative samples based on the high-consistency samples to train a discriminative judgment model; generating multiple candidate text descriptions for the low-consistency samples; inputting each candidate text description and the corresponding original audio into the discriminative judgment model to obtain a matching score; performing acoustic consistency verification based on the matching score to determine a candidate set; and determining a target text description from the candidate set. The automatically generated audio emotion description of the application takes into account the emotional richness and acoustic authenticity.

Description

Technical Field

[0001] This invention relates to the field of speech understanding technology, specifically to a method and system for generating fine-grained audio emotion descriptions. Background Technology

[0002] Existing audio emotion description technologies are mainly divided into two categories: manual annotation and automatic generation. While manual annotation yields reliable results, it is costly, time-consuming, and highly subjective, making it difficult to support large-scale model training. Among automatic generation solutions, coarse-grained emotion tag expansion and large-scale text model expansion are simple to implement, but the former is limited by insufficient tag information, and the latter relies on linguistic priors for rationalization. Neither can accurately describe the complex information in real speech, such as tone of voice, psychological state, and paralinguistic changes, and they are prone to producing acoustic illusions that are inconsistent with the acoustic performance of the original audio.

[0003] Meanwhile, while directly using audio language models to automatically generate descriptions can handle large-scale audio, these models generally exhibit a neutral bias, tending to output conservative descriptions such as calm tone and natural voice, ignoring subtle emotions and paralinguistic details in the audio. When the model integrates multimodal information such as visual or text transcription, it is also prone to favoring high-information-density text content while ignoring acoustic variations in the audio itself, such as speech rate, pitch, and energy. Furthermore, existing screening schemes based on text similarity or text evaluators can only assess description quality in the text space and cannot verify whether candidate descriptions truly match the original audio signal, leading to a disconnect between evaluation results and acoustic reality. In other words, the core contradiction faced by existing technologies lies in the difficulty of simultaneously achieving rich emotional descriptions and acoustic realism. Summary of the Invention

[0004] This invention aims to address, to a certain extent, one of the technical problems in related technologies. To this end, this invention provides a fine-grained audio emotion description generation method and system. Without requiring large-scale manual annotation, it automatically generates fine-grained descriptions that are emotionally rich and consistent with the acoustic performance of the original audio, while reducing neutral bias and acoustic illusions. This solves the problem that automatically generated audio emotion descriptions struggle to simultaneously achieve both emotional richness and acoustic realism.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A fine-grained audio emotion description generation method, the method comprising:

[0007] The original audio is obtained, and each original audio is labeled using at least two audio understanding models to generate at least two initial text descriptions. Each original audio and its corresponding initial text description are used together as a labeled sample.

[0008] Calculate the semantic similarity between the initial text descriptions of the same original audio, and divide the labeled samples into high consistency samples and low consistency samples based on the semantic similarity and a preset similarity threshold;

[0009] Based on the high consistency samples, positive samples, soft negative samples, and hard negative samples are constructed. A discriminative evaluation model is trained using the positive samples, the soft negative samples, and the hard negative samples. For the low consistency samples, multiple candidate text descriptions are generated based on at least one initial text description corresponding to them.

[0010] Each candidate text description and its corresponding original audio are input into the discriminative evaluation model to obtain a matching score. Based on the matching score and a preset acceptance threshold, the candidate text descriptions are subjected to acoustic consistency verification. A candidate set is determined according to the verification results, and the target text description is determined from the candidate set according to a preset priority.

[0011] Optionally, constructing positive samples, soft negative samples, and hard negative samples based on the highly consistent samples includes:

[0012] One text description is selected from at least two initial text descriptions corresponding to the highly consistent sample as the positive sample;

[0013] Based on the positive samples, the soft negative samples and the hard negative samples are generated using a text generation model or rule template;

[0014] The soft negative sample is a text description that has the same sentence template as the positive sample, but whose emotional polarity or overall psychological state is opposite to that of the positive sample.

[0015] The hard negative sample is a text description that has the same overall semantic and emotional intensity as the positive sample, but is inconsistent with the audio in subtle emotions, tone, psychological state, or expression.

[0016] By precisely controlling variables with minimal textual alterations, the evaluation model can systematically learn the acoustic consistency discrimination boundary during training from two levels: coarse-grained emotion errors and fine-grained paralinguistic biases, thereby improving the model's ability to recognize subtle emotional differences and its generalization.

[0017] Optionally, training the discriminative evaluation model using the positive samples, the soft negative samples, and the hard negative samples includes:

[0018] Using the positive samples as positive examples and the soft negative samples and hard negative samples as negative examples, construct training triples;

[0019] The discriminative evaluation model is trained based on the training triplet using a binary classification cross-entropy loss function, so that the output of the positive sample approaches the first label value, and the output of the soft negative sample and the hard negative sample approaches the second label value.

[0020] Wherein, the first tag value is a matching tag, and the second tag value is a non-matching tag.

[0021] By using a triplet training structure consisting of positive examples and soft / hard negative examples, combined with a binary cross-entropy loss function, the evaluation model learns acoustic consistency discrimination ability collaboratively from two levels: coarse-grained sentiment error and fine-grained paralinguistic bias, thereby achieving precise calibration of the decision boundary.

[0022] Optionally, the candidate text descriptions include semantic drift candidates, conservative refinement candidates, and original description candidates; the step of generating multiple candidate text descriptions based on at least one corresponding initial text description for the low consistency sample includes:

[0023] For the low-consistency sample, based on its corresponding initial text description, semantic drift is performed along multiple sentiment dimensions to generate multiple semantic drift candidates; and / or

[0024] Perform conservative refinement on the initial text description to generate conservative refinement candidates; and / or

[0025] The initial text description itself is used as a candidate for the original description.

[0026] By constructing a three-tiered progressive candidate description spectrum of semantic drift candidates, conservative refinement candidates, and original description candidates, the emotional expression space is systematically expanded while ensuring acoustic authenticity, taking into account both descriptive richness and output security.

[0027] Optionally, determining the target text description from the candidate set according to a preset priority includes:

[0028] If the semantic drift candidate exists in the candidate set, the one with the highest matching score or the highest sentiment value among the semantic drift candidates is selected as the target text description, wherein the sentiment value is used to measure the emotional richness of the text description;

[0029] If the semantic drift candidate does not exist in the candidate set, but the conservative refinement candidate does exist, the conservative refinement candidate shall be used as the target text description.

[0030] If only the original description candidate exists in the candidate set, then the original description candidate is used as the target text description.

[0031] By employing a three-tiered progressive selection strategy, while ensuring that each output description passes acoustic consistency verification, the candidate with the richest emotional expression is selected first, thus achieving a dynamic balance between acoustic realism and emotional richness.

[0032] Optionally, the acoustic consistency verification of the candidate text description based on the matching score and the preset acceptance threshold includes:

[0033] Candidate text descriptions that match scores that reach or exceed the preset acceptance threshold are added to the candidate set of the corresponding original audio.

[0034] If the matching scores of all candidate text descriptions corresponding to the original audio are lower than the preset acceptance threshold, then the original audio and all its corresponding text descriptions are discarded.

[0035] To ensure that every description entering the candidate set has quantifiable acoustic authenticity, the quality of the final dataset is guaranteed by discarding all substandard samples, thus preventing the introduction of acoustic illusion data.

[0036] Optionally, calculating the semantic similarity between initial text descriptions of the same original audio includes:

[0037] Each of the initial text descriptions is input into the text embedding model to obtain the corresponding text vector;

[0038] Calculate the cosine similarity between the text vectors corresponding to the same original audio, and use the cosine similarity as the semantic similarity between the initial text descriptions.

[0039] It achieves an objective, efficient, and batch-computable quantitative evaluation of the semantic consistency among multiple model outputs, providing a standardized and manual basis for subsequent sample partitioning.

[0040] Furthermore, the present invention also provides a fine-grained audio emotion description generation system, the system comprising:

[0041] The annotation module is used to acquire the original audio, use at least two audio understanding models to annotate each original audio, generate at least two initial text descriptions, and combine each original audio and its corresponding initial text description as a single annotation sample.

[0042] The segmentation module is used to calculate the semantic similarity between the initial text descriptions of the same original audio, and to divide the labeled samples into high consistency samples and low consistency samples based on the semantic similarity and a preset similarity threshold.

[0043] The model training module is used to construct positive samples, soft negative samples, and hard negative samples based on the high consistency samples, train a discriminative evaluation model through the positive samples, the soft negative samples, and the hard negative samples, and generate multiple candidate text descriptions based on the low consistency samples and at least one initial text description corresponding to them.

[0044] The description determination module is used to input each candidate text description and its corresponding original audio into the discriminative evaluation model to obtain a matching score, perform acoustic consistency verification on the candidate text description based on the matching score and a preset acceptance threshold, determine a candidate set according to the verification result, and determine the target text description from the candidate set according to a preset priority.

[0045] The fine-grained audio emotion description generation system provided by this invention is used to implement the aforementioned fine-grained audio emotion description generation method. Its specific implementation process can be referred to the aforementioned method embodiments, and will not be repeated here.

[0046] Furthermore, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the fine-grained audio emotion description generation method described above.

[0047] Meanwhile, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the fine-grained audio emotion description generation method described above.

[0048] These features and advantages of the present invention will be disclosed in detail in the following specific embodiments and accompanying drawings. The preferred embodiments or means of the present invention will be shown in detail in conjunction with the accompanying drawings, but are not intended to limit the technical solutions of the present invention. In addition, each of these features, elements and components appearing in the following text and drawings is a plurality of, and different symbols or numbers are used for convenience of representation, but all represent parts with the same or similar construction or function. Attached Figure Description

[0049] The present invention will be further described below with reference to the accompanying drawings:

[0050] Figure 1 This is a flowchart of a fine-grained audio emotion description generation method according to an embodiment of this application;

[0051] Figure 2 This is a flowchart of a discrimination model training process according to an embodiment of this application;

[0052] Figure 3 This is an audio description filtering flowchart according to an embodiment of this application. Detailed Implementation

[0053] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described are intended to explain the present invention and should not be construed as limiting the invention.

[0054] The terms "an embodiment," "example," or "trademark" used in this specification refer to a particular feature, structure, or characteristic described in connection with the embodiment itself that may be included in at least one embodiment disclosed in this patent. The phrase "in an embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment.

[0055] Definitions of abbreviations and key terms:

[0056] Large Audio Language Model (LALM) refers to a multimodal language model that can process audio signals and text information and perform tasks such as audio understanding, audio description generation, speech question answering, and speech emotion understanding.

[0057] Audio Caption: Refers to a natural language description generated for an audio clip. The description can include information such as the speaker's tone, emotion, speaking speed, pitch, energy, scene, and event.

[0058] Fine-grained emotion description refers to natural language description that not only provides coarse-grained categories such as "happy, sad, angry, neutral", but also further describes the intensity of emotion, emotional tendency, psychological state, tone changes, and paralinguistic features in speech.

[0059] Neutral bias refers to the phenomenon where audio language models or audio description data tend to generate safe, generalized, and neutral descriptions, such as "calm tone," "natural voice," and "clear speech," while ignoring the more subtle emotions, tone, and psychological states in the actual audio.

[0060] Acoustic hallucination refers to a situation where the text description generated by a model appears semantically reasonable but is inconsistent with the actual acoustic performance in the original audio signal. For example, if there is no obvious anxiety in the audio, a description such as "rapid and tense tone" may be generated.

[0061] Semantic drift refers to generating multiple candidate emotional descriptions along dimensions such as emotional polarity, arousal, tone strength, and psychological state while keeping the basic semantic framework or descriptive object of the original text unchanged, so that the candidate text covers a richer range of potential emotional states.

[0062] A discriminative judge model is a model that takes raw audio and candidate text descriptions as input and judges whether the candidate text matches the acoustic performance of the audio. Its output can be a "match / not match" label, or a match probability or match score.

[0063] Hard negative samples refer to negative samples that are similar to positive sample text in overall semantics or sentence structure, but inconsistent with the audio in subtle aspects such as emotion, tone intensity, and psychological state. Their purpose is to train models to identify fine-grained paralinguistic differences.

[0064] Soft negative samples refer to negative samples that are similar to positive sample texts in sentence structure but have significantly opposite emotional polarity or overall sentiment direction. Their purpose is to train the model to eliminate obviously erroneous emotional descriptions.

[0065] Acoustic Fidelity refers to the consistency between the acoustic performance of the text description and the original audio, including consistency in emotional tone, tone, speech rate, energy, pitch variation, and expressive state.

[0066] Example:

[0067] This embodiment provides a method for generating fine-grained audio emotion descriptions. Figure 1 This is a flowchart of a fine-grained audio emotion description generation method according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps:

[0068] Step S101: Obtain the original audio, use at least two audio understanding models to annotate each original audio, generate at least two initial text descriptions, and use each original audio and its corresponding initial text description as a single annotation sample.

[0069] The input audio set is denoted as [A={a i} N i=1 ], of which (a i Let represent the (i)th audio file. Audio files can come from voice dialogues, audiobooks, podcasts, customer service recordings, film and television dialogues, emotional speech libraries, or other audio data containing human voice expressions. Each audio file undergoes uniform format processing to meet the input requirements of subsequent audio understanding and evaluation models.

[0070] Obtaining the original audio specifically includes: collecting the original audio data and assigning a unique number to each audio file; standardizing the audio format, including standardizing the sampling rate, channel, audio trimming, silence processing, and loudness normalization; filtering out abnormal audio, such as audio that is too short, too noisy, missing speech content, or audio that cannot be processed normally by the model; and inputting the preprocessed audio into the subsequent audio understanding model.

[0071] For example, the audio understanding model includes a first annotation model (M1) and a second annotation model (M2). For each audio clip (a i Two initial text descriptions (c) are generated using the first annotation model (M1) and the second annotation model (M2), respectively. i (1) ) and (c i (2) The prompts require the model to describe not only the speech content, but also the speaker's tone, emotion, psychological state, and paralinguistic features.

[0072] The two labeled models can be audio language models, multimodal language models, or other speech models with audio understanding capabilities. Using two or more models can reduce the impact of subjective bias from a single model on subsequent data construction. It should be noted that the number of audio understanding models is not limited to two; it can be extended to three or more.

[0073] Step S102: Calculate the semantic similarity between the initial text descriptions of the same original audio, and divide the labeled samples into high consistency samples and low consistency samples according to the semantic similarity and the preset similarity threshold.

[0074] Optionally, calculating the semantic similarity between initial text descriptions of the same original audio includes:

[0075] Each initial text description is input into the text embedding model to obtain the corresponding text vector.

[0076] Calculate the cosine similarity between the text vectors corresponding to the same original audio, and use the cosine similarity as the semantic similarity between the initial text descriptions.

[0077] Using the text embedding model (E) to (c i (1) ) and (c i (2) Encode them separately to obtain text vectors (e) i (1) ) and (e i (2) ):

[0078] .

[0079] Calculation (e)i (1) ) and (e i (2) The cosine similarity s i :

[0080] .

[0081] If (s) i If ≥τ), then the sample is classified as a highly consistent sample; if (s i If the similarity threshold is less than τ, then the sample is classified as a low-consistency sample. Here, (τ) is a preset similarity threshold, such as 0.8. High-consistency samples indicate that different models describe the audio in a relatively consistent manner and can be used as a relatively reliable source of positive samples; low-consistency samples indicate that there may be more complex or difficult-to-determine emotional expressions in the audio, which are suitable for subsequent semantic drift and other expansions.

[0082] It should be noted that the text embedding model can be replaced by any semantic representation model. Similarity metrics can be cosine similarity, Euclidean distance, dot product similarity, cross-encoder scoring, or large language models. The threshold (τ) can be fixed or dynamically set according to the dataset distribution, for example, by selecting a certain quantile of the similarity distribution.

[0083] Continue to refer to Figure 1 After dividing the labeled samples into high consistency samples and low consistency samples, step S103 is executed.

[0084] Step S103: Construct positive samples, soft negative samples, and hard negative samples based on high consistency samples. Train a discriminative evaluation model using positive samples, soft negative samples, and hard negative samples. For low consistency samples, generate multiple candidate text descriptions based on at least one initial text description corresponding to them.

[0085] Optionally, constructing positive samples, soft negative samples, and hard negative samples based on highly consistent samples includes:

[0086] Select one text description from at least two initial text descriptions corresponding to a highly consistent sample as a positive sample;

[0087] Based on positive samples, soft negative samples and hard negative samples are generated using text generation models or rule templates.

[0088] Among them, soft negative samples are text descriptions that maintain the same sentence template as positive samples, but whose emotional polarity or overall psychological state is opposite to that of positive samples;

[0089] Hard negative samples are text descriptions that have the same overall semantic and emotional intensity as positive samples, but are inconsistent with the audio in subtle emotions, tone, psychological state, or expression.

[0090] For highly consistent samples, one of the corresponding multiple initial text descriptions is selected as the positive sample (c i + Since the multiple descriptions of a highly consistent sample are semantically very similar, any one can be selected. Then, using a text generation model or rule generation method, two classes of negative samples are constructed around this positive sample.

[0091] The first category is soft negative samples (c i soft Soft negative samples maintain the same sentence template as positive samples, but replace the emotional polarity or overall psychological state with a clearly opposite direction. "Same sentence template" means that the two texts are basically identical in the order of sentence components, subject-verb-object structure, and modification relationships; the difference lies only in the replacement of emotional keywords. For example, a positive sample might be "calm tone with a sense of nostalgia," while a soft negative sample would be rewritten as "urgent tone with a sense of fear"—the sentence structure is completely identical ("tone [emotion A] and with [psychological state B]"), but the emotional polarity (calm → urgent) and psychological state (nostalgia → fear) are both replaced with opposite directions. The purpose of soft negative samples is to train the evaluation model to have a coarse-grained emotion discrimination ability, capable of eliminating obviously erroneous emotional descriptions.

[0092] The second category is hard negative samples (c i hard Hard negative samples are close to positive samples in overall semantics and emotional intensity, but inconsistent with the audio in subtle emotions, tone, psychological state, or expression. "Same overall semantics and emotional intensity" means that the emotional category (e.g., "pleasant," "sad," "neutral") and its intensity level (e.g., "slightly" "very") of the text description are consistent with the positive sample; however, there are deviations in "subtle emotions, tone, psychological state, or expression"—these deviations may be due to different emotional subtypes (e.g., writing "excited pleasure" instead of "relaxed pleasure"), different tone (e.g., writing "mild narration" instead of "flat narration"), different psychological states (e.g., writing "satisfied" instead of "reassuring"), or different paralinguistic features (e.g., writing "smooth speech rate" instead of "slightly fast speech rate"). For example, if the positive sample is "calm and pleasant," the hard negative sample can be rewritten as "flat and polite, but lacking genuine pleasure." The overall emotional intensity is basically the same as the positive sample, but the psychological state shifts from "calm" to "flat," and the emotion shifts from "pleasant" to "polite." These subtle differences are difficult to detect with coarse-grained judgment, but it is necessary to evaluate whether the model has the ability to identify fine-grained paralinguistic biases.

[0093] Soft and hard negative samples can be generated automatically by text generation models, by rule templates, or by sentiment dictionaries, sentiment dimension controllers, and minor manual corrections. Hard negative samples may focus on errors in tone intensity, slight errors in emotional polarity, errors in arousal, errors in mental state, or errors in paralinguistic features. Soft negative samples may focus on obviously opposite emotions or obviously unreasonable tone changes.

[0094] Two types of negative samples form a complementary training strategy of "coarse-grained + fine-grained". Soft negative samples force the evaluation model to learn to exclude candidates with obviously incorrect emotional polarity and establish basic emotional classification boundaries; hard negative samples, on the other hand, force the evaluation model to further refine the decision boundaries and learn to identify candidates that are correct in overall emotional category but whose fine-grained expression is inconsistent with the audio. The collaborative training of the two allows the evaluation model to calibrate its judgment criteria step by step from coarse to fine, avoiding both the model's excessive rejection of valid candidates with rich emotions and its acceptance of illusory descriptions with acoustic inconsistencies, thus improving the accuracy and robustness of acoustic consistency verification.

[0095] In an alternative approach, in addition to hard and soft negative samples, polarity reversal negative samples, intensity perturbation negative samples, paralinguistic contradiction negative samples, nearest neighbor difficulty negative samples, text over-expansion negative samples, and semantic over-shift negative samples can be further constructed. By using negative samples at more levels, a discriminative evaluation model can be trained to identify erroneous candidates at different granularities.

[0096] Optionally, training the discriminative evaluation model using positive samples, soft negative samples, and hard negative samples includes:

[0097] Use positive samples as positive examples and soft negative samples and hard negative samples as negative examples to construct training triples.

[0098] A binary classification cross-entropy loss function is adopted, and a discriminative evaluation model is trained based on training triples, so that the output of positive samples approaches the first label value, and the output of soft negative samples and hard negative samples approaches the second label value.

[0099] The first tag value is a matching tag, and the second tag value is a non-matching tag.

[0100] Each highly consistent audio segment can form a training triple:

[0101] ;

[0102] In this context, label 1 indicates that the text matches the audio, and label 0 indicates that the text does not match the audio.

[0103] Construct a discriminative evaluation model (J). The model takes the original audio (a) as input. i Given candidate text (c) and candidate text (c), output the matching probability (p). i ):

[0104] ;

[0105] The training objective is to make the output of positive samples close to 1, and the output of soft and hard negative samples close to 0. A binary classification cross-entropy loss function can be used.

[0106] ;

[0107] Here, (y) represents the sample label. By using both soft and hard negative samples simultaneously, the evaluation model can learn to exclude obviously erroneous sentiment polarities while also learning to identify subtle paralinguistic biases. Compared to using only one type of negative sample, this training strategy can better calibrate the decision boundary of the evaluation model, avoiding the model from excessively rejecting valid candidates and also avoiding accepting acoustically inconsistent candidates.

[0108] By using a binary cross-entropy loss function to drive the model's output to approximate matching / non-matching labels, the evaluation model calibrates the decision boundary step by step from coarse to fine, effectively solving the problem of decision boundaries being too wide (falsely accepting non-matching candidates) or too narrow (falsely rejecting valid candidates) caused by training with a single negative sample. Simultaneously, the highly controllable differences in sentence templates and semantic strength between positive and negative examples allow the model to focus on learning the intrinsic correspondence between acoustic performance and text description, rather than superficial textual statistical patterns, thus improving the accuracy and generalization ability of verifying candidate descriptions based on the original audio signal.

[0109] It should be noted that the discriminative evaluation model can output not only matching scores, but also reasons for mismatches, error types, or suspicious segments in the candidate text. Reasons for mismatches can include inconsistencies in emotional polarity, arousal levels, tone intensity, paralinguistic features, excessive text expansion, and excessive semantic shift. This output can be used for subsequent adjustments to candidate generation strategies or updates to the negative sample library.

[0110] Figure 2 This is a flowchart illustrating the training process of a discriminative model according to an embodiment of this application. In this embodiment, the judging model can be a dual-tower model fusing an audio encoder and a text encoder, or it can be an audio language model, a multimodal large model, a cross-attention model, or any model capable of taking audio and text as input and outputting a matching score. The model output can be a Yes / No label, matching probability, ranking score, or multi-level confidence level.

[0111] Optionally, candidate text descriptions include semantic drift candidates, conservative refinement candidates, and original description candidates; for low consistency samples, multiple candidate text descriptions are generated based on at least one corresponding initial text description, including:

[0112] For low consistency samples, based on at least one corresponding initial text description, semantic drift is performed along multiple sentiment dimensions to generate multiple semantic drift candidates.

[0113] The initial text description is conservatively refined to generate conservative refinement candidates.

[0114] The initial text description itself is used as a candidate for the original description.

[0115] This embodiment designs three types of candidate text description generation strategies for low-consistency samples (i.e., audio with significant discrepancies in multi-model annotation results) to cover the sentiment expression space from conservative to aggressive. The first type is semantic drift candidates: based on at least one initial text description corresponding to the low-consistency sample, a systematic semantic shift is performed along multiple sentiment dimensions such as emotional polarity, arousal, tone intensity, psychological state, and paralinguistic features. This actively explores the potential emotional interpretations that the original audio may contain, generating multiple description variants with richer and more diverse emotional expressions. The second type is conservative refinement candidates: these make slight word adjustments, grammatical optimizations, or detail additions to the initial text description without changing its core sentiment judgment and acoustic facts, aiming to improve the naturalness and readability of the text. The third type is original description candidates: these directly use the initial text description itself as one of the candidates, serving as the most conservative backup option.

[0116] For semantic drift candidates, one of the initial descriptions of the low-consistency samples is selected as the baseline description (c 0 i Then, the candidate description generation model (G) is used to perform semantic drift across multiple dimensions, resulting in a semantic drift candidate set:

[0117] ;

[0118] Among them, (d1,d2,…,d K () represents different drift dimensions. Semantic drift dimensions include, but are not limited to, emotional polarity, arousal, tone intensity, psychological state, speaking style, and paralinguistic features.

[0119] Emotional polarity dimension: for example, shifting from neutral to slightly positive, slightly negative, significantly positive, or significantly negative.

[0120] Arousal dimension: for example, shifting from calm to excitement, agitation, depression, fatigue, or restraint.

[0121] The intensity of tone: for example, shifting from ordinary narration to strong questioning, gentle reassurance, suppressed patience, or slight sarcasm.

[0122] Psychological state dimension: such as nostalgia, hesitation, resentment, vigilance, relaxation, indifference, surprise, etc.

[0123] Paralinguistic features include: such as speech rate, pitch, energy, pauses, trembling, laughter, and crying.

[0124] Semantic drift is not simply a paraphrase; rather, it actively explores the emotional interpretation space that the original audio may contain while maintaining the core semantic framework. This allows it to break through the limitations of the original neutral description and form a richer set of candidate descriptions.

[0125] Semantic shifting can be achieved using large language model prompts, or through emotion-dimensional control words, emotion tag combinations, template replacement, text editing models, reinforcement learning generators, or retrieval enhancement generators. The number of semantic shift candidates can be 3, 5, 10, 20, or dynamically determined based on audio complexity.

[0126] It should be noted that candidate text descriptions can be generated all at once or iteratively. In iterative generation, coarse-grained candidates are first generated, and then a second round of revised candidates is generated based on the rejection reasons or low-scoring dimensions of the evaluation model, until the candidates pass the verification or the maximum number of iterations is reached.

[0127] Continue to refer to Figure 1 After generating candidate text descriptions, step S104 is executed.

[0128] Step S104: Match each candidate text description with the corresponding original audio input discriminative evaluation model to obtain a matching score. Based on the matching score and a preset acceptance threshold, perform acoustic consistency verification on the candidate text descriptions. Determine the candidate set according to the verification results and select the target text description from the candidate set according to a preset priority.

[0129] Optionally, acoustic consistency verification of candidate text descriptions based on matching scores and preset acceptance thresholds includes:

[0130] Candidate text descriptions that match scores that reach or exceed a preset acceptance threshold are added to the candidate set of the corresponding original audio.

[0131] If the matching scores of all candidate text descriptions corresponding to the original audio are lower than the preset acceptance threshold, then the original audio and all its corresponding text descriptions are discarded.

[0132] Candidate set (C) i In the text, each candidate text description (c∈C) is used to describe the candidate text. i ) and the original audio (a i Input the discriminative evaluation model (J) to obtain the matching score (r). i (c)

[0133] ;

[0134] If (r) i(c)≥γ), then the candidate is considered to have passed the acoustic consistency verification; if (r)≥γ), then the candidate is considered to have passed the acoustic consistency verification; i If (c) < γ), then the candidate is considered to have acoustic hallucinations or emotional biases and is rejected. Here, (γ) is a preset acceptance threshold, which can be dynamically adjusted based on the performance of the validation set or the candidate type.

[0135] Optionally, the target text description is determined from the candidate set according to a preset priority, including:

[0136] If semantically drifting candidates exist in the candidate set, the one with the highest matching score or the highest sentiment value among the semantically drifting candidates is selected as the target text description. The sentiment value is used to measure the emotional richness of the text description.

[0137] If there are no semantically drifting candidates in the candidate set, but there are conservative refinement candidates, the conservative refinement candidates will be used as the target text description.

[0138] If only the original description candidate exists in the candidate set, then the original description candidate will be used as the target text description.

[0139] For the verified candidate set, the following priority is used for selection:

[0140] ;

[0141] Wherein, Drift represents semantic drift candidates, Refine represents conservative refinement candidates, and Origin represents original description candidates. If there are validated semantic drift candidates, the candidate with the highest evaluation score or the richest sentiment information is selected as the final description; if none of the semantic drift candidates pass validation, the validated conservative refinement candidate is selected; if the refinement candidate also fails validation, the original description is retained; if the original description also fails validation, the sample is discarded.

[0142] When the evaluation model has high confidence in the drift candidate, it outputs a fine-grained description rich in emotion; when the drift candidate is deemed unreliable, the system automatically downgrades without crashing; when even the original description fails verification, the system discards the sample to ensure the overall quality of the data. This mechanism ensures that the system pursues emotional richness while always maintaining acoustic realism as a hard constraint, without sacrificing accuracy for richness, and without outputting a large number of low-information neutral descriptions due to excessive conservatism, thus achieving a dynamic optimal balance between the two.

[0143] It should be noted that the priority order Drift → Refine → Origin can be replaced with a ranking method based on a comprehensive score. For example, it can comprehensively consider factors such as the evaluation model score, sentiment information content, text diversity, description length, and semantic distance from the original description. Priorities can also be set for different application scenarios; for example, Origin or Refine can be prioritized in scenarios with high security requirements, while Drift can be prioritized in scenarios with high requirements for generative expressiveness.

[0144] Figure 3 This is a flowchart of an audio description filtering process according to an embodiment of this application. The final dataset (D) is obtained according to this process:

[0145] ;

[0146] Among them, (c * i (N') represents the final description obtained after semantic drift, acoustic verification, and reordering, and (N') represents the number of samples retained. This dataset can be used to train or fine-tune audio language models to generate more granular audio descriptions that are more consistent with acoustic facts, and can also be used for tasks such as speech synthesis, emotion recognition, and voice dialogue.

[0147] It should be noted that for highly consistent samples, any initial text description can be chosen as the target text description.

[0148] The high-quality descriptive data generated above is used to supervise the fine-tuning of the audio language model, enabling it to output richer and more accurate emotional and paralinguistic descriptions when inputting audio. These descriptions can also be used as emotion-controlled text for speech synthesis systems, enhancing the expressiveness of audiobooks, dialogue systems, virtual humans, or emotional speech generation systems. The specific process includes:

[0149] Construct training samples, each sample including the original audio input and the final fine-grained emotion description output.

[0150] The training samples described above are used to perform supervised fine-tuning of a large audio language model, enabling the model to learn the mapping relationship from audio to fine-grained emotional descriptions.

[0151] After training, the model is able to generate more nuanced emotional descriptions than the original neutral descriptions, such as "the tone is slightly restrained but with a sense of grievance", "the voice is relaxed and has a sense of reminiscence", and "the speaking speed is slightly fast and the expression has a sense of urgency".

[0152] This embodiment can process the entire audio segment, or it can first divide the long audio into sentence-level, phrase-level, or time-segment-level units, generate local audio descriptions for each, and then merge them into a complete audio description through global consistency constraints. For audiobooks, long dialogues, or long conference audio, this method can avoid contradictions in the emotional descriptions between different parts of the audio.

[0153] Steps S101-S104 above propose an end-to-end, fine-grained automatic generation scheme for audio sentiment descriptions. This scheme systematically addresses core pain points in existing technologies, such as the difficulty in simultaneously achieving rich emotional content and acoustic realism, severe neutrality bias, and frequent acoustic hallucinations, without requiring large-scale manual annotation. The scheme achieves automatic sample division and control through multi-model collaborative annotation and semantic similarity partitioning. High-consistency samples are used to construct positive, soft-negative, and hard-negative samples to train a discriminative evaluation model, enabling the model to collaboratively learn acoustic consistency discrimination capabilities from both coarse-grained sentiment errors and fine-grained paralinguistic biases. Low-consistency samples undergo semantic drift along multiple dimensions, including emotional polarity, arousal, tone intensity, psychological state, and paralinguistic features, generating a candidate description set covering a rich array of potential emotional states. Simultaneously, conservative refinement and original baseline construction are used to build candidate descriptions. Ultimately, by using a hard threshold screening based on acoustic consistency verification of the original audio signal and a three-level progressive sorting strategy of "prioritizing semantic drift candidates, followed by conservative refinement candidates, and guaranteeing original description candidates as a backup," the candidate with the richest emotional expression is selected first, ensuring that each output description passes acoustic verification. This achieves a dynamic optimal balance between acoustic authenticity and emotional richness.

[0154] The entire solution combines multiple technical benefits, including reducing manual annotation costs, overcoming neutral bias, reducing acoustic illusions, improving fine-grained expressive capabilities, enhancing the accuracy of the evaluation model, ensuring the lower limit of dataset quality, and strong scalability. It can be widely applied to downstream tasks such as fine-tuning audio language models, speech emotion recognition, emotional speech synthesis, and intelligent customer service emotion analysis, providing a complete and feasible technical path for the large-scale automated construction of fine-grained audio emotion descriptions.

[0155] This embodiment can be used not only for audio description data construction, but also for scenarios such as speech emotion recognition, speech synthesis emotion control, dialogue system emotion understanding, audiobook production, intelligent customer service, educational speech analysis, film and television dialogue analysis, and psychological state auxiliary assessment.

[0156] Used for building audio emotion description datasets:

[0157] First, several audio segments are collected, each containing the tone and emotional expression of a real speaker. Then, two audio understanding models are used to annotate the same audio segment, resulting in two natural language descriptions. A text embedding model is used to calculate the semantic similarity between the two descriptions, and samples with similarity above a threshold are designated as high-consistency samples, while those below the threshold are designated as low-consistency samples. For high-consistency samples, a text generation model is used to construct positive, soft-negative, and hard-negative samples to train a discriminative evaluation model. For low-consistency samples, a semantic drift model is used to generate multiple candidate emotion descriptions, which are then validated by the discriminative evaluation model based on the original audio. Finally, the final description for each audio segment is output according to the priority order (Drift→Refine→Origin).

[0158] For example, given a user's voice message "I'm fine, you don't need to worry about me," the first annotation model generates "the speaker's tone is calm," while the second annotation model generates "the speaker's tone is low and suppressed." The system calculates the semantic similarity between the two texts. If the similarity is below a preset threshold, the audio is classified as a low-consistency sample. The system then uses one of these descriptions as a baseline to generate multiple candidate descriptions, such as "restrained tone with a hint of grievance," "cold tone with a sense of alienation," and "slightly fast speech with a sense of anxiety." The discriminative evaluation model takes the original audio and each candidate description as input and outputs a matching score. If "restrained tone with a hint of grievance" has the highest matching score and exceeds the acceptance threshold, it is adopted as the final description; otherwise, if all drifting candidates fail verification, the system reverts to either a conservative, refined description or the original description.

[0159] For fine-tuning large audio language models:

[0160] Fine-grained audio sentiment description data, constructed from an audio sentiment description dataset, is used as supervised fine-tuning data. The input is audio, and the output is natural language sentiment description. Compared to models trained using raw neutral descriptions, models trained with the data in this embodiment can more accurately describe subtle emotions in audio, such as "slightly restrained but with a hint of grievance," "relaxed voice with a sense of reminiscence," and "slightly fast speech with a sense of urgency," thereby reducing the proportion of generalized descriptions such as "calm tone" and "natural voice" output by the model.

[0161] Used for audiobooks or emotional speech synthesis:

[0162] In audiobook scenarios, the text content often contains complex character emotions and narrative tone. The audio emotion descriptions generated by this invention can provide more granular emotional references for speech synthesis systems, enabling synthesized speech to have more natural expressive variations in scenarios such as dialogue, narration, recollection, tension, sadness, and irony.

[0163] Used for customer service recordings and conversation sentiment analysis:

[0164] In customer service or human-computer interaction scenarios, the same sentence may convey different intentions depending on the tone of voice. The fine-grained audio descriptions generated in this embodiment can help the system identify whether a user is in a state of dissatisfaction, anxiety, hesitation, repression, anger, or calm, thereby assisting in emotion analysis, service quality assessment, and intelligent response strategy selection.

[0165] Furthermore, this embodiment also provides a fine-grained audio emotion description generation system. This system is used to implement the embodiments and preferred implementations of the above-described fine-grained audio emotion description generation method, and details already described will not be repeated. The fine-grained audio emotion description generation system includes:

[0166] The annotation module is used to acquire the original audio. It uses at least two audio understanding models to annotate each original audio file and generates at least two initial text descriptions. Each original audio file and its corresponding initial text description are used together as an annotation sample.

[0167] The segmentation module is used to calculate the semantic similarity between the initial text descriptions of the same original audio, and to divide the labeled samples into high consistency samples and low consistency samples based on the semantic similarity and a preset similarity threshold.

[0168] The model training module is used to construct positive samples, soft negative samples, and hard negative samples based on highly consistent samples. It trains a discriminative evaluation model using positive samples, soft negative samples, and hard negative samples. For low-consistency samples, it generates multiple candidate text descriptions based on at least one initial text description corresponding to them.

[0169] The description determination module is used to match each candidate text description with the corresponding original audio input discriminant evaluation model to obtain a matching score, perform acoustic consistency verification on the candidate text descriptions based on the matching score and a preset acceptance threshold, determine the candidate set according to the verification results, and determine the target text description from the candidate set according to a preset priority.

[0170] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0171] Meanwhile, this embodiment also provides a computer device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the fine-grained audio emotion description generation method described above. The steps of the fine-grained audio emotion description generation method here can be the steps in the fine-grained audio emotion description generation methods of the various embodiments described above.

[0172] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. Accordingly, the computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can implement the methods of any of the above embodiments. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0173] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Those skilled in the art should understand that the present invention includes, but is not limited to, the contents described in the accompanying drawings and the specific embodiments above. Any modifications that do not depart from the functional and structural principles of the present invention will be included within the scope of the claims.

Claims

1. A method for generating fine-grained audio emotion descriptions, characterized in that, The method includes: The original audio is obtained, and each original audio is labeled using at least two audio understanding models to generate at least two initial text descriptions. Each original audio and its corresponding initial text description are used together as a labeled sample. Calculate the semantic similarity between the initial text descriptions of the same original audio, and divide the labeled samples into high consistency samples and low consistency samples based on the semantic similarity and a preset similarity threshold; Based on the high consistency samples, positive samples, soft negative samples, and hard negative samples are constructed. A discriminative evaluation model is trained using the positive samples, the soft negative samples, and the hard negative samples. For the low consistency samples, multiple candidate text descriptions are generated based on at least one initial text description corresponding to them. Each candidate text description and its corresponding original audio are input into the discriminative evaluation model to obtain a matching score. Based on the matching score and a preset acceptance threshold, the candidate text descriptions are subjected to acoustic consistency verification. A candidate set is determined according to the verification results, and the target text description is determined from the candidate set according to a preset priority.

2. The fine-grained audio emotion description generation method according to claim 1, characterized in that, The construction of positive samples, soft negative samples, and hard negative samples based on the highly consistent samples includes: One text description is selected from at least two initial text descriptions corresponding to the highly consistent sample as the positive sample; Based on the positive samples, the soft negative samples and the hard negative samples are generated using a text generation model or rule template; The soft negative sample is a text description that has the same sentence template as the positive sample, but whose emotional polarity or overall psychological state is opposite to that of the positive sample. The hard negative sample is a text description that has the same overall semantic and emotional intensity as the positive sample, but is inconsistent with the audio in subtle emotions, tone, psychological state, or expression.

3. The fine-grained audio emotion description generation method according to claim 1, characterized in that, The training of the discriminative evaluation model using the positive samples, the soft negative samples, and the hard negative samples includes: Using the positive samples as positive examples and the soft negative samples and hard negative samples as negative examples, construct training triples; The discriminative evaluation model is trained based on the training triplet using a binary classification cross-entropy loss function, so that the output of the positive sample approaches the first label value, and the output of the soft negative sample and the hard negative sample approaches the second label value. Wherein, the first tag value is a matching tag, and the second tag value is a non-matching tag.

4. The fine-grained audio emotion description generation method according to claim 1, characterized in that, The candidate text descriptions include semantic drift candidates, conservative refinement candidates, and original description candidates; For the low-consistency sample, generating multiple candidate text descriptions based on at least one corresponding initial text description includes: For the low consistency sample, based on its corresponding initial text description, semantic drift is performed along multiple sentiment dimensions to generate multiple semantic drift candidates. and / or Perform conservative refinement on the initial text description to generate conservative refinement candidates; and / or The initial text description itself is used as a candidate for the original description.

5. The fine-grained audio emotion description generation method according to claim 4, characterized in that, The step of determining the target text description from the candidate set according to a preset priority includes: If the semantic drift candidate exists in the candidate set, the one with the highest matching score or the highest sentiment value among the semantic drift candidates is selected as the target text description, wherein the sentiment value is used to measure the emotional richness of the text description; If the semantic drift candidate does not exist in the candidate set, but the conservative refinement candidate does exist, the conservative refinement candidate shall be used as the target text description. If only the original description candidate exists in the candidate set, then the original description candidate is used as the target text description.

6. The fine-grained audio emotion description generation method according to claim 1, characterized in that, The acoustic consistency verification of the candidate text description based on the matching score and the preset acceptance threshold includes: Candidate text descriptions that match scores that reach or exceed the preset acceptance threshold are added to the candidate set of the corresponding original audio. If the matching scores of all candidate text descriptions corresponding to the original audio are lower than the preset acceptance threshold, then the original audio and all its corresponding text descriptions are discarded.

7. The fine-grained audio emotion description generation method according to claim 1, characterized in that, The calculation of semantic similarity between initial text descriptions of the same original audio includes: Each of the initial text descriptions is input into the text embedding model to obtain the corresponding text vector; Calculate the cosine similarity between the text vectors corresponding to the same original audio, and use the cosine similarity as the semantic similarity between the initial text descriptions.

8. A fine-grained audio emotion description generation system, characterized in that, The system includes: The annotation module is used to acquire the original audio, use at least two audio understanding models to annotate each original audio, generate at least two initial text descriptions, and combine each original audio and its corresponding initial text description as a single annotation sample. The segmentation module is used to calculate the semantic similarity between the initial text descriptions of the same original audio, and to divide the labeled samples into high consistency samples and low consistency samples based on the semantic similarity and a preset similarity threshold. The model training module is used to construct positive samples, soft negative samples, and hard negative samples based on the high consistency samples, train a discriminative evaluation model through the positive samples, the soft negative samples, and the hard negative samples, and generate multiple candidate text descriptions based on the low consistency samples and at least one initial text description corresponding to them. The description determination module is used to input each candidate text description and its corresponding original audio into the discriminative evaluation model to obtain a matching score, perform acoustic consistency verification on the candidate text description based on the matching score and a preset acceptance threshold, determine a candidate set according to the verification result, and determine the target text description from the candidate set according to a preset priority.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the fine-grained audio emotion description generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the fine-grained audio emotion description generation method according to any one of claims 1 to 7.

Images