Audio data annotation method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence and the field of financial science and technology, and discloses an audio data labeling method, which comprises the following steps: acquiring audio data, determining gender attributes by using a gender classifier, determining accent attributes by using an accent classifier, and analyzing audio quality to obtain a signal-to-noise ratio and an early and late reflection ratio; the pitch features are analyzed to obtain the average pitch and the pitch standard deviation of the speaker, and the ratio of the number of phonemes to the audio length is calculated to determine the speaking speed attribute. And generating a corresponding keyword for each attribute, and generating annotation information of the audio data based on the keywords. According to the method, the corresponding keywords are generated by analyzing various attributes of the audio data, so that the efficiency and the accuracy of annotation of the audio data are remarkably improved. According to discretization processing, continuous attributes (such as speaking speed, signal-to-noise ratio and the like) are converted into simple category names, processing of complex data is simplified, and the system is more efficient in the aspects of classification, retrieval and expansibility.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to an audio data annotation method, device, equipment and storage medium. Background Art

[0002] With the continuous development of financial technology, voice technology is increasingly used in financial services, including intelligent customer service, voice identity authentication, automated trading instructions, etc. Existing speech recognition and synthesis technologies can handle basic voice tasks, but have obvious shortcomings when dealing with diverse audio data.

[0003] Although existing speech recognition and synthesis systems can recognize the pronunciation of standard languages, their accuracy and naturalness are significantly reduced when dealing with accents of users from different regions. As accents have both discrete and continuous characteristics, traditional discrete labeling methods are difficult to fully and accurately reflect the characteristics of different accents, resulting in large differences in speech recognition results. Especially in financial scenarios, differences in accents of different users may affect the quality of service.

[0004] At the same time, existing technologies have difficulty in effectively labeling various features in audio data, such as pitch, speech speed, etc. These features have an important impact on speech recognition and synthesis, but the existing system lacks the ability to fine-tune these detailed features, resulting in unsatisfactory speech processing results in some complex scenarios, and cannot meet the needs of the financial field for high-precision speech processing. Summary of the invention

[0005] The main purpose of the present invention is to provide an audio data annotation method, device, equipment and storage medium, aiming to solve the technical problem that the prior art is difficult to accurately annotate and distinguish various key features such as accent, pitch, speaking speed in audio data.

[0006] To achieve the above object, the present invention provides an audio data tagging method, comprising:

[0007] Get audio data;

[0008] Based on the audio data, determining a gender attribute using a gender classifier;

[0009] determining an accent attribute using an accent classifier based on the audio data;

[0010] Analyze the audio quality of the audio data to obtain a signal-to-noise ratio attribute and an early-late reflection ratio value attribute;

[0011] Analyze the pitch characteristics of the audio data to obtain the average pitch attribute and the pitch standard deviation attribute of the speaker;

[0012] Analyzing the ratio of the number of phonemes in the audio data to the length of the audio data to determine a speaking speed attribute;

[0013] Using the category name of the gender attribute as a keyword of the gender attribute, and using the category name of the accent attribute as a keyword of the accent attribute;

[0014] Mapping the speaking speed attribute, the signal-to-noise ratio attribute, the early-late reflection ratio attribute, the speaker average pitch attribute, and the pitch standard deviation attribute to corresponding preset discrete category names, and using the preset discrete category names as keywords for the corresponding attributes respectively;

[0015] The tag information of the audio data is generated based on the keyword of each attribute.

[0016] Furthermore, to achieve the above-mentioned purpose, the present invention provides an audio data tagging device, comprising:

[0017] A data acquisition module, used for acquiring audio data;

[0018] A gender recognition module, configured to determine a gender attribute based on the audio data using a gender classifier;

[0019] an accent recognition module for determining accent attributes using an accent classifier based on the audio data;

[0020] An audio quality analysis module, used to analyze the audio quality of the audio data and obtain a signal-to-noise ratio attribute and an early-late reflection ratio value attribute;

[0021] A pitch feature analysis module, used to analyze the pitch features of the audio data and obtain the average pitch attribute and the pitch standard deviation attribute of the speaker;

[0022] A speaking speed analysis module, used to analyze the ratio of the number of phonemes in the audio data to the audio length, and determine the speaking speed attribute;

[0023] a gender and accent classification module, used to use the category name of the gender attribute as a keyword of the gender attribute, and use the category name of the accent attribute as a keyword of the accent attribute;

[0024] An attribute mapping module, used to map a speaking speed attribute, a signal-to-noise ratio attribute, an early-late reflection ratio attribute, a speaker average pitch attribute, and a pitch standard deviation attribute to corresponding preset discrete category names, and use the preset discrete category names as keywords for the corresponding attributes respectively;

[0025] The annotation generation module is used to generate annotation information of the audio data based on the keyword of each attribute.

[0026] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and an audio data annotation program stored in the memory and executable on the processor, and the audio data annotation program, when executed by the processor, implements the steps of the audio data annotation method as described above.

[0027] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which an audio data annotation program is stored, and when the audio data annotation program is executed by a processor, the steps of the audio data annotation method described above are implemented.

[0028] Beneficial effects: The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses an audio data annotation method, which obtains audio data, determines gender attributes using a gender classifier, determines accent attributes using an accent classifier, analyzes audio quality to obtain signal-to-noise ratio and early-late reflection ratio, analyzes pitch features to obtain the average pitch and pitch standard deviation of the speaker, and calculates the ratio of the number of phonemes to the length of the audio to determine the speaking speed attribute. Corresponding keywords are generated for each attribute, and annotation information for audio data is generated based on these keywords. The present invention significantly improves the efficiency and accuracy of audio data annotation by automatically analyzing multiple attributes of audio data (such as gender, accent, pitch, etc.) and generating corresponding keywords. Discretization processing converts continuous attributes (such as speaking speed, signal-to-noise ratio, etc.) into simple category names, simplifies the processing of complex data, and makes the system more efficient in classification, retrieval, and scalability. At the same time, the automated process reduces manual operations, ensures the consistency and accuracy of data annotation, and is particularly suitable for scenarios that require large-scale processing of audio data. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0030] Figure 1 A schematic diagram of an application environment of an audio data annotation method according to an embodiment of the present invention;

[0031] Figure 2 A schematic diagram of a flow chart of an embodiment of an audio data annotation method of the present invention;

[0032] Figure 3 A schematic diagram of functional modules of a preferred embodiment of the audio data tagging device of the present invention;

[0033] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0034] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0035] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0036] The audio data annotation method provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, the user end communicates with the server end through a network. The server end can obtain audio data through the user end, determine the gender attribute using a gender classifier, determine the accent attribute using an accent classifier, analyze the audio quality to obtain the signal-to-noise ratio and the early-late reflection ratio, analyze the pitch feature to obtain the speaker's average pitch and pitch standard deviation, and calculate the ratio of the number of phonemes to the audio length to determine the speaking speed attribute. Generate corresponding keywords for each attribute, and generate annotation information for the audio data based on these keywords. By analyzing and annotating multiple attributes in the audio data, the present invention can accurately annotate multiple key features in the audio data, ensure the comprehensive expression of each attribute, and improve the processing efficiency and accuracy of the voice data. Among them, the user end can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server end can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0037] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of an audio data tagging method provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0038] like Figure 2 As shown, the audio data annotation method proposed by the present invention includes the following steps:

[0039] S10, obtaining audio data;

[0040] In this embodiment, obtaining audio data refers to collecting voice signals or audio samples from a specified data source or device. This can be achieved through multiple channels, such as directly collecting through a recording device, extracting from an audio database, or downloading existing audio data from a cloud service platform. Audio can be collected using a microphone or other recording device, or audio files can be retrieved from a cloud service through an API interface. The collected audio data can be saved in formats such as WAV, MP3, etc., and basic preprocessing such as noise reduction and background sound removal is required to ensure that the quality of the audio data is suitable for subsequent analysis. Audio files can also be extracted from existing audio databases or public data sets (such as LibriVox).

[0041] In one specific implementation, the transaction conversation between the customer and the customer service is collected through a dedicated telephone line, and the conversation content is automatically recorded using the built-in recording function of the device. The system automatically identifies key financial transaction terms in the conversation, such as "transfer" and "remittance", and automatically marks these parts during collection for subsequent processing.

[0042] Example description: In the identity verification process of a financial institution, the system uses voice recognition and analysis of the customer's voice data to ensure the legitimacy of the transaction. By obtaining the customer's voice data, the system can use the gender classifier to determine the customer's gender attribute to ensure that it is consistent with their registration information; use the accent classifier to identify the customer's accent to determine whether the customer's voice is abnormal; obtain features such as signal-to-noise ratio, pitch, and speaking speed to ensure the clarity and naturalness of the voice.

[0043] Through the above steps, it can be ensured that the audio data obtained by the system comes from a wide range of sources and has controllable quality, which can meet the needs of different application scenarios. It not only improves the availability of audio data, but also ensures the accuracy and efficiency of subsequent processing (such as audio feature analysis and annotation). In the financial field, it can realize real-time collection of customer voice data or quickly obtain customer voice information from historical data, thereby improving the quality of voice interaction services.

[0044] S20, determining a gender attribute using a gender classifier based on the audio data;

[0045] In the present embodiment, by processing audio data, a specific gender classifier is used to determine the gender attribute of the speaker. Gender classifier is a model based on machine learning or deep learning, which is specifically used to analyze features (such as frequency, pitch, timbre, etc.) in audio and predict the gender of the speaker based on these features. First, audio features that can distinguish gender are extracted from audio data, mainly including pitch, spectral features and volatility of voice, etc. A pre-trained gender classifier model is used, which can classify the gender of the speaker based on audio features. According to the result of the gender classifier, the speaker is marked as "male" or "female", and a corresponding gender attribute label is generated.

[0046] By preprocessing the audio data, the system extracts relevant frequency information and pitch features, especially the dominant frequency and formant of the audio. Based on these features, the gender classifier can identify gender differences in the speaker. Using a trained gender classification model, the extracted audio features are input into the model. Common gender classification models include models based on convolutional neural networks (CNN) or models based on naive Bayes classifiers. The classifier processes the input audio features and outputs a label of "male" or "female", which is annotated as a gender attribute.

[0047] In one specific implementation, frequency components in the audio, especially pitch and formant, are extracted through spectrum analysis technology, and gender classification is performed using a CNN model. For audio data with different accents, the classifier can adapt to pronunciation differences and ensure classification accuracy, especially stability under different tones and speaking speeds.

[0048] Example description: In the intelligent financial assistant application, the system interacts with the customer through voice, and after determining the customer's gender, provides customized financial advice and financial product recommendations. For example, the system can recommend suitable financial products based on the preferences of customers of different genders. The customer communicates with the intelligent financial assistant system through voice, and the system collects voice data in real time. The gender classifier is used to analyze the customer's voice and identify the customer's gender. Based on the identified gender attribute, the system automatically provides personalized financial advice and financial product recommendations to the customer.

[0049] Through the above steps, the system can quickly and accurately identify the gender attribute of the speaker and provide accurate basic data for subsequent speech processing.

[0050] S30, determining an accent attribute using an accent classifier based on the audio data;

[0051] In the present embodiment, an accent classifier is used to determine the accent attribute of the speaker through audio data. The accent classifier can identify dialects or accents in different regions, such as Mandarin, Cantonese, Sichuanese, Minnan dialect, etc. By extracting audio features related to the accent, the classifier can effectively distinguish the pronunciation differences in different places and determine the accent attribute of the speaker. The system extracts features that can distinguish different accents from the audio data, including pitch, tone, speech speed, formant frequency, rhythm, etc. These features have significant differences in pronunciation in different places. Using a specially trained accent classifier model, the model is trained with various dialects and accents and can identify common accents such as Mandarin, Cantonese, Minnan dialect, Northeastern dialect, Sichuan dialect, etc. According to the recognition result of the accent classifier, a corresponding accent attribute label is generated to identify the specific accent features of the speaker.

[0052] In one specific implementation, the system extracts pitch and spectral features from audio data and uses an LSTM (Long Short-Term Memory)-based accent classifier model to identify the difference between Mandarin and local dialects (such as Cantonese and Minnan dialect). An assessment of the standardization of Mandarin is added to the classifier model to determine whether Mandarin has a local accent, such as "Mandarin with a Cantonese accent" or "Mandarin with a Sichuan accent."

[0053] By using the accent classifier, the system can accurately distinguish the accent attributes of the speaker and has a high degree of accuracy in the recognition of Mandarin and dialects. It can help financial institutions better understand the language characteristics of customers and improve the performance of the speech recognition system in multi-regional and multi-accent scenarios.

[0054] S40, analyzing the audio quality of the audio data to obtain a signal-to-noise ratio attribute and an early-late reflection ratio value attribute;

[0055] In this embodiment, the signal-to-noise ratio (SNR) and the early-late reflection ratio are obtained by analyzing the audio quality attributes in the audio data. These two attributes directly affect the clarity and recognizability of the audio. The signal-to-noise ratio is used to measure the clarity of the audio signal relative to the background noise, while the early-late reflection ratio is used to describe the reverberation characteristics in the audio, especially the ratio of early reflection to late reflection, which plays an important role in determining the audibility and spatial sense of the audio.

[0056] The signal-to-noise ratio (SNR) is the ratio of the signal strength to the background noise strength, reflecting the clarity of the speech signal relative to the noise. By separating the speech and background noise components, the system calculates the signal-to-noise ratio.

[0057] The ratio of early and late reflections reflects the reverberation characteristics of the audio. Early reflections are short reflections after direct sound, and late reflections are reverberation reflections with a longer delay. By measuring the energy of different reflected sounds in the audio, the spatial sense and reverberation degree of the audio are determined.

[0058] Use spectrum analysis or waveform separation technology to separate the effective speech part and the background noise part in the audio data. By calculating the power of the effective speech signal and the power of the background noise and taking the ratio between the two, the signal-to-noise ratio attribute is obtained. According to the predetermined signal-to-noise ratio threshold range, the audio data is classified into different quality levels (such as "high quality", "medium quality" and "low quality").

[0059] Through time-frequency analysis technology, the time points and energy of early reflections and late reflections in the audio are identified. The energy of early reflections and late reflections is measured respectively, and the energy ratio of the two is used as the early-late reflection ratio attribute. According to the preset ratio range, the audio is classified into different reverberation levels (such as "strong reverberation", "moderate reverberation", "weak reverberation").

[0060] By analyzing the signal-to-noise ratio and early-to-late reflection ratio of audio data, the system can accurately judge the clarity and spatial sense of the audio, thereby improving the processing quality and recognition accuracy of audio data.

[0061] S50, analyzing the pitch features of the audio data to obtain the average pitch attribute and the pitch standard deviation attribute of the speaker;

[0062] In this embodiment, by analyzing the pitch features in the audio data, the average pitch attribute of the speaker and the volatility of the pitch (standard deviation attribute) are obtained. Pitch is an important feature of speech, and is particularly important for the emotion and identity recognition of the speaker. The average pitch is used to reflect the basic intonation of the speaker, and the pitch standard deviation is used to measure the volatility of the speaker's voice. By extracting all the pitch values ​​in the audio data, the average value of these values ​​is calculated to obtain the basic pitch attribute of the speaker. By analyzing the variation and distribution of the pitch values, the standard deviation of the pitch values ​​is calculated to reflect the volatility of the speaker's voice.

[0063] The audio data is processed using a pitch detection tool to extract pitch values ​​(such as F0 frequency) in the audio, which reflect the fundamental frequency of the speech. The system records the changes in pitch according to the time axis to form a pitch sequence. All pitch values ​​in the pitch sequence are summarized and the average of these pitch values ​​is calculated as the average pitch attribute of the speaker.

[0064] Analyze the distribution of pitch values, calculate the deviation of each pitch value from the average pitch, and then calculate the standard deviation to obtain the range and characteristics of pitch fluctuations.

[0065] In one specific embodiment, the system uses an autocorrelation function to identify periodic signals in the audio and extract the pitch value. The autocorrelation function can suppress the noise part of the audio signal, so it is particularly suitable for use in environments with high background noise, such as telephone calls or voice message systems. The system combines formant analysis and autocorrelation pitch detection to further improve the accuracy of pitch detection in high-noise environments. It is suitable for audio processing in the telephone customer service system of financial institutions, enabling the system to accurately analyze customer voice characteristics even in the case of high background noise.

[0066] Example: In a voice transaction system of a financial institution, a customer makes a large transfer instruction over the phone. The system analyzes the pitch features in the customer's voice and extracts the average pitch and pitch standard deviation. If it detects that the customer's pitch fluctuates greatly and does not match the usual voice data, the system will automatically determine that the customer is nervous or anxious, triggering additional identity verification steps or requiring manual customer service to intervene to confirm the transaction instruction. This process not only improves transaction security, but also prevents potential fraud.

[0067] Through the above steps, the pitch characteristics of the speaker can be accurately analyzed, and the two key attributes of average pitch and pitch standard deviation can be extracted. This helps to better understand the speaker's identity characteristics, emotional fluctuations, and voice patterns in speech analysis. In the financial field, pitch analysis can be used for customer identity authentication and emotional analysis. For example, in telephone transactions, the system can determine whether the customer is in a state of tension or anxiety by analyzing the customer's pitch fluctuations, thereby enhancing risk management and customer service responsiveness.

[0068] S60, analyzing the ratio of the number of phonemes in the audio data to the audio length to determine a speaking speed attribute;

[0069] In the present embodiment, the speaking speed attribute of the speaker is calculated by analyzing the ratio between the number of phonemes in the audio data and the total length of the audio. Phoneme is the smallest unit of speech in speech, and the speaking speed reflects the amount of speech output of the speaker in a specific time. By calculating the ratio of the number of phonemes to the length of the audio, the speaking speed of the speaker can be accurately measured and classified into different speaking speed levels (such as "fast", "medium" and "slow"). Through speech recognition technology, the system can identify each phoneme in the audio and count the total number of phonemes. The audio length is the total duration of the audio, that is, the actual pronunciation time of the speaker in a specific audio. By calculating the ratio of the number of phonemes to the length of the audio, the system obtains a quantitative result of the speaking speed, and then determines the speaking speed attribute of the speaker.

[0070] The audio data is processed using speech recognition technology to extract the phonemes in each pronunciation and count the number of phonemes in the entire audio. Commonly used technologies include HMM (Hidden Markov Model) and Deep Neural Network (DNN) models. The system accurately calculates the total duration of the audio by analyzing the duration of the audio file, which can be done by directly reading the metadata of the audio file. The system divides the number of phonemes by the length of the audio to obtain the number of phoneme pronunciations per second (phonemes / second), which is the speaking speed attribute of the speaker. Based on the calculated speaking speed value, the system classifies it into different speaking speed levels, such as: "fast" (high number of phonemes per second), "medium" (moderate phoneme pronunciation), and "slow" (fewer phoneme pronunciations).

[0071] In one specific implementation, the audio data is analyzed by a hidden Markov model (HMM) to extract the phoneme sequence in the audio. The system captures the dynamic changes in the speech signal through the HMM model, counts the frequency of occurrence of each phoneme, and calculates the speaking speed based on the audio duration. The system can adjust the parameters of the HMM model according to the context of different audio to ensure accurate recognition of phonemes in noisy environments, especially in the application of telephone calls or meeting records in the financial industry.

[0072] By analyzing the ratio of the number of phonemes to the length of audio, more accurate speech speed recognition and analysis can be achieved in the financial field, helping financial institutions improve overall efficiency and security in customer service, transaction security and risk control.

[0073] S70, using the category name of the gender attribute as a keyword of the gender attribute, and using the category name of the accent attribute as a keyword of the accent attribute;

[0074] In this embodiment, the gender attribute is to determine the gender of the speaker by analyzing the voice signal in the audio data. Common gender classifications include "male" and "female". The category name is a text label of the classification result, i.e. "male" or "female". Such names are usually generated by a gender classifier, which determines gender based on voice features (such as pitch, frequency, etc.). Keywords are identifiers used by the generated category names for subsequent data processing. The category name of the gender attribute is directly used as a keyword for the gender attribute, i.e. "Male" or "Female" can be used as a label to mark the gender information of the audio data.

[0075] The accent attribute determines the speaker's accent by analyzing specific characteristics of pronunciation in audio data, such as voice patterns, pronunciation habits, rhythm, and intonation.

[0076] Accent attributes usually include differences in Mandarin and local dialects in various regions. The main categories can be divided into:

[0077] Standard Mandarin: Official standard Mandarin, with clear pronunciation and no obvious local accent.

[0078] Mandarin with dialect: Mandarin mixed with local accents, such as northern accent, southern accent, etc.

[0079] Specific dialect accents: such as Northeastern accent, Cantonese accent, Minnan accent, Sichuan accent, Hunan accent, Hakka accent, etc.

[0080] Category name: The classifier outputs a category name, such as "Standard Mandarin", "Mandarin with a Northeastern accent", or "Cantonese", by analyzing the pronunciation features in the audio data.

[0081] The category name is the result of the classifier's judgment based on the pronunciation characteristics of the audio data, such as "Mandarin", "Mandarin with a Northeastern accent", or "Cantonese". The keyword is the category name of the accent attribute as the identifier of the audio data. By generating keywords such as "Mandarin" or "Cantonese", the classification and retrieval of audio data can be achieved.

[0082] For example, the audio data is input into an accent classifier, which extracts pronunciation features from the audio, such as syllables, tones, and pronunciation rhythm, for identifying the type of accent. A trained accent classification model (such as a deep learning-based model) is used to analyze the pronunciation pattern of the audio to determine whether the speaker is using standard Mandarin, Mandarin with a local accent, or a specific dialect, such as Cantonese or Minnan dialect. Based on the classification results, an accent category name is generated, such as "Mandarin", "Mandarin with a Northeastern accent", or "Cantonese". The generated accent category name is used as a keyword for the accent attribute to annotate the accent features of the audio data.

[0083] By automatically classifying and labeling the gender and accent attributes of audio data, supporting Mandarin and dialects in various regions, and generating corresponding keywords, the accuracy and efficiency of data labeling are significantly improved, labor costs are reduced, and the retrieval and management capabilities of audio data are enhanced. It is suitable for scenarios that need to process large amounts of audio data.

[0084] S80, mapping the speaking speed attribute, the signal-to-noise ratio attribute, the early-late reflection ratio attribute, the speaker average pitch attribute, and the pitch standard deviation attribute to corresponding preset discrete category names, and using the preset discrete category names as keywords of the corresponding attributes respectively;

[0085] In this embodiment, the speaking speed attribute of the audio data is analyzed and mapped to preset discrete categories, such as "fast", "medium", and "slow" according to the number of phonemes per minute. By counting the number of phonemes per unit time and according to a preset threshold, the speaking speed is classified as "fast", "medium", or "slow". The corresponding keywords generated are "fast", "medium", or "slow".

[0086] The signal-to-noise ratio attribute measures the ratio of the audio signal to the background noise, and maps it to preset discrete categories such as "high", "medium", and "low" based on the value of the signal-to-noise ratio. By performing spectral analysis on the audio data, the strength of the signal and noise is calculated, and it is divided into "high", "medium", and "low" according to the preset signal-to-noise ratio interval. The corresponding generated keywords are "high signal-to-noise ratio", "medium signal-to-noise ratio", or "low signal-to-noise ratio".

[0087] The early-late reflection ratio reflects the echo characteristics of the environment. According to the size of this ratio, it is mapped to preset discrete categories, such as "strong reflection", "medium reflection", and "weak reflection". By analyzing the acoustic reflection characteristics in the audio data, the ratio of early and late reflections is calculated, and according to the set standards, it is divided into "strong reflection", "medium reflection", and "weak reflection". The generated keywords are "strong reflection", "medium reflection" or "weak reflection".

[0088] The average pitch of the speaker is analyzed and mapped to "high", "middle", and "bass" according to the range of pitch values. The pitch value is calculated by analyzing the fundamental frequency (F0) and divided into "high", "middle", or "bass" according to the pitch range. The generated keywords are "high", "middle", or "bass".

[0089] The standard deviation of pitch reflects the fluctuation of pitch, and according to the fluctuation amplitude, it is mapped into "large fluctuation", "medium fluctuation" and "small fluctuation". By calculating the standard deviation of the fundamental frequency in the audio, the fluctuation amplitude of the pitch is evaluated, and according to the preset standard, it is divided into "large fluctuation", "medium fluctuation" or "small fluctuation". The generated keywords are "large fluctuation", "medium fluctuation" or "small fluctuation".

[0090] Discretization can simplify complex continuous data, making processing and analysis more efficient. For example, continuous values ​​of speaking speed, pitch, or signal-to-noise ratio may have a very large range of values, and directly processing these continuous data may be computationally complex. Mapping them into discrete categories such as "fast" and "medium" can reduce processing complexity and facilitate rapid calculation and classification by the system.

[0091] For many users or application scenarios, discretized data is easier to understand. For example, "fast" and "slow" are more intuitive than specific values ​​(such as 200 words / minute). Especially in application scenarios that require business-oriented decisions, the discretized results are more in line with the user's understanding and needs, helping them to respond quickly. Mapping continuous attributes to discrete categories can facilitate classification and indexing by the system. For example, when processing large amounts of audio data, classifying the audio according to discrete categories such as "fast", "slow", "high pitch", and "bass" can greatly improve the efficiency of data retrieval and help the system quickly find audio files that meet specific conditions.

[0092] Discretization of continuous data can reduce the impact of noise or extreme values. In some cases, continuous data may be affected by external factors and experience abnormal fluctuations. Discretization can reduce the impact of these fluctuations on subsequent processing or analysis and improve the stability of the system.

[0093] By discretizing continuous attributes into preset categories, the data processing process is simplified, the efficiency and stability of the system are improved, and the interpretability and usability of the data are enhanced. The generated keywords help improve the retrieval and classification capabilities of audio data, facilitate the application of business rules, reduce the interference of noise and data fluctuations, and are particularly suitable for the annotation and management of large-scale audio data.

[0094] S90: Generate annotation information of the audio data based on the keyword of each attribute.

[0095] In this embodiment, the system combines the keywords generated by each attribute to form structured annotation information. For example, "male, Cantonese, fast, high pitch, high signal-to-noise ratio" represents a segment of audio data with a male speaker, speaking Cantonese, fast speaking speed, high pitch and clear sound quality.

[0096] In one specific implementation, when generating keywords, the system combines the contextual information in the audio to ensure that the keywords not only reflect a single attribute, but also improve the accuracy of the annotation information through the context. For example, when analyzing the speech rate, more accurate annotations are generated by combining the speaker's emotional information. By introducing speech emotion recognition technology, the system can reflect the speaker's emotional changes and tone characteristics when generating keywords, further improving the comprehensiveness of the annotation information.

[0097] By generating keywords for each attribute and constructing structured annotation information for audio data, the system can greatly improve the usability and operability of audio data. This annotation method provides a strong foundation for subsequent speech analysis, retrieval, and classification. In the financial field, the system can not only improve the accuracy of speech recognition by generating detailed annotation information for customers' speech data, but also provide more intelligent support for scenarios such as personalized customer service and voice transaction confirmation.

[0098] The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses an audio data annotation method, which obtains audio data, determines gender attributes using a gender classifier, determines accent attributes using an accent classifier, analyzes audio quality to obtain signal-to-noise ratio and early-late reflection ratio, analyzes pitch features to obtain the speaker's average pitch and pitch standard deviation, and calculates the ratio of the number of phonemes to the audio length to determine the speaking speed attribute. Corresponding keywords are generated for each attribute, and annotation information of audio data is generated based on these keywords. By analyzing and annotating multiple attributes in audio data, the present invention can accurately annotate multiple key features in audio data, ensure the comprehensive expression of each attribute, and improve the processing efficiency and accuracy of voice data.

[0099] In one embodiment, the above S80 includes:

[0100] S801, mapping the speaking speed attribute into discrete category names including fast, medium and slow;

[0101] S802, mapping the signal-to-noise ratio attribute into discrete category names including high, medium and low;

[0102] S803, mapping the early and late reflection ratio attributes into discrete category names including strong reflection, medium reflection and weak reflection;

[0103] S804, mapping the speaker's average pitch attribute into discrete category names including high pitch, middle pitch and low pitch;

[0104] S805, mapping the pitch standard deviation attribute into discrete category names including large fluctuation, medium fluctuation and small fluctuation;

[0105] S806: Use the discrete category name as a keyword for the corresponding attribute.

[0106] In this embodiment, the speaking speed attribute in the audio data is analyzed, and the number of phonemes per unit time is calculated and mapped into discrete categories of "fast", "medium" and "slow". The number of phonemes in the audio data is counted by a speech analysis algorithm, and thresholds of three discrete categories of "fast", "medium" and "slow" are set based on the number of phonemes per minute. For example, more than 200 phonemes per minute are classified as "fast", between 100 and 200 as "medium", and less than 100 as "slow". The speaking speed is mapped into these three discrete categories according to the calculation results, and the corresponding keywords "fast", "medium" or "slow" are finally generated.

[0107] Analyze the signal-to-noise ratio in audio data, measure the ratio between signal strength and noise, and map the signal-to-noise ratio into three discrete categories: "high", "medium", and "low". Through the signal-to-noise ratio analysis tool, extract the strength of the effective signal and background noise in the audio, and calculate the signal-to-noise ratio. According to the preset threshold, for example, a signal-to-noise ratio greater than 50dB is "high", between 30 and 50dB is "medium", and below 30dB is "low". Based on this analysis result, the signal-to-noise ratio attribute is mapped to "high", "medium", and "low", and the corresponding keywords "high signal-to-noise ratio", "medium signal-to-noise ratio", or "low signal-to-noise ratio" are generated.

[0108] Analyze the early and late reflection ratio attributes in the audio data. This ratio is used to measure the reflection echo characteristics of the environment and divide it into "strong reflection", "medium reflection" and "weak reflection". Use the acoustic analysis tool to calculate the ratio of early and late reflection sound energy. According to the set threshold, the reflection ratio value above a certain value is defined as "strong reflection", within a certain range is "medium reflection", and below a certain value is "weak reflection". Generate the corresponding keywords "strong reflection", "medium reflection" or "weak reflection" based on the calculation results.

[0109] By analyzing the fundamental frequency in the audio data, the average pitch of the speaker is calculated and mapped into discrete categories of "high", "middle" and "bass". Through the fundamental frequency extraction algorithm in the voice signal, the average pitch value of the speaker during the call is calculated and compared with the preset fundamental frequency range. For example, a fundamental frequency above 300Hz is "high", between 150 and 300Hz is "middle", and below 150Hz is "bass". Generate the corresponding keywords "high", "middle" or "bass".

[0110] Analyze the fluctuation of pitch in audio data, use standard deviation to measure the fluctuation of pitch, and divide it into "large fluctuation", "medium fluctuation" and "small fluctuation". Calculate the standard deviation of pitch to measure the variation of pitch. According to the standard deviation value, divide it into three discrete categories, such as "large fluctuation" when it is greater than a certain standard deviation, "medium fluctuation" when it is within a certain range, and "small fluctuation" when it is less than a certain standard deviation. Generate the corresponding keywords "large fluctuation", "medium fluctuation" or "small fluctuation".

[0111] After discretizing each attribute, the result is mapped to the corresponding keyword. These keywords serve as the attribute annotation information of the audio data. The discrete category name after each attribute mapping is directly recorded as a keyword in the audio data. For example, the speaking speed attribute of an audio is mapped to "fast", and the signal-to-noise ratio attribute is mapped to "high signal-to-noise ratio", then the keywords of the audio data will eventually include information such as "fast" and "high signal-to-noise ratio".

[0112] Example description: In the financial field, customer service centers usually collect a large number of customer call records. Through this technical solution, the system can automatically analyze the speaking speed, pitch, signal-to-noise ratio and other attributes in the call records to generate corresponding keyword annotations. For example, a customer's call record shows that his speaking speed is fast, the pitch is high and the signal-to-noise ratio is low. The system will automatically generate "fast", "high pitch" and "low signal-to-noise ratio" as keywords. This annotation method not only improves data management efficiency, but also helps financial institutions identify customers' emotions or urgency, thereby providing personalized services and improving customer satisfaction.

[0113] This embodiment simplifies the processing of continuous attributes and improves the efficiency and accuracy of audio data annotation by mapping multiple attributes of audio data into discrete category names and generating Chinese keywords. Discrete processing makes the classification and retrieval of audio data more efficient, and the generated Chinese keywords are intuitive and easy to understand, which facilitates users to quickly understand audio features. The automated processing flow reduces the workload of manual annotation and improves data consistency, which is particularly suitable for scenarios of large-scale audio data management.

[0114] In one embodiment, in the above S40, analyzing the audio quality of the audio data to obtain a signal-to-noise ratio attribute includes:

[0115] S401, using a spectrum analysis module to separate the audio data into a valid speech part and a background noise part;

[0116] S402, measuring the power of the effective speech part and the background noise part respectively;

[0117] S403, analyzing the ratio of the effective speech power to the background noise power to determine a signal-to-noise ratio attribute value;

[0118] S404: Classify the signal-to-noise ratio attribute value into a corresponding audio quality level according to a predetermined signal-to-noise ratio threshold range.

[0119] In this embodiment, the audio data is divided into the effective speech part and the background noise part by the spectrum analysis module, and the power of each part is measured, and finally the signal-to-noise ratio (SNR) attribute value is determined by calculating the ratio of the two. According to the preset threshold range, the signal-to-noise ratio value is classified into different audio quality levels to help the system judge the clarity and usability of the audio.

[0120] The system first uses the spectrum analysis module to process the audio data in the frequency domain and decompose the audio signal into different frequency components. By analyzing the energy distribution of each frequency band, the system can identify the effective speech part (mainly concentrated in a specific frequency range) and the background noise part (usually low-frequency or high-frequency noise).

[0121] The system calculates the power of the effective speech part (i.e., the signal energy in the main speech frequency band) and the power of the background noise part. Power is an important physical quantity that reflects the signal strength and is usually estimated by the energy density function of the spectrum.

[0122] The system determines the signal-to-noise ratio attribute value by calculating the ratio of effective speech power to background noise power. The signal-to-noise ratio is usually measured in decibels (dB) and is used to measure the clarity of speech signals. If the signal-to-noise ratio is high, the speech signal is clear; if the signal-to-noise ratio is low, the background noise has a greater impact.

[0123] The system classifies the calculated signal-to-noise ratio attribute values ​​into different audio quality levels according to the preset signal-to-noise ratio threshold range. For example:

[0124] High signal-to-noise ratio (>30dB): indicates high audio quality with less background noise.

[0125] Medium SNR (10-30dB): indicates that the audio quality is average and background noise has some impact on the voice.

[0126] Low signal-to-noise ratio (<10dB): indicates poor audio quality and high background noise.

[0127] The system uses short-time Fourier transform (STFT) to perform spectrum analysis on the input audio data. By converting the audio signal from the time domain to the frequency domain, the system can clearly identify the energy distribution of different frequency bands and separate the effective speech part and the background noise part according to the unique frequency characteristics of the speech.

[0128] The system measures the power of the effective speech part and the background noise part respectively. The power calculation of the effective speech part is mainly concentrated in the mid-frequency band, while the power of the background noise is usually located in the low-frequency band or the high-frequency band. Through the energy density function (Power Spectral Density, PSD), the system can accurately estimate the power value of each frequency band.

[0129] The system calculates the signal-to-noise ratio attribute value through the ratio of effective speech power to background noise power.

[0130] The system compares the signal-to-noise ratio attribute value with the preset threshold range to determine the level of audio quality. If the signal-to-noise ratio is high, the system will mark the audio as high quality; if the signal-to-noise ratio is low, it will be marked as low quality.

[0131] In one specific implementation, the spectrum of the audio is estimated by an autoregressive (AR) model, which can more accurately identify the effective speech part in the audio containing strong noise, and further improve the calculation accuracy of the signal-to-noise ratio. By introducing the autoregressive model, random background noise can be effectively suppressed, ensuring that the speech signal can be accurately separated even in a low signal-to-noise ratio environment.

[0132] This embodiment can effectively evaluate the quality of audio data through spectrum analysis, power measurement and signal-to-noise ratio calculation, and provide high-quality input for subsequent speech recognition and processing. In the financial field, especially in scenarios such as telephone transactions and speech recognition identity authentication, accurate calculation of the signal-to-noise ratio can help the system determine the reliability of audio data and improve the accuracy and security of speech recognition.

[0133] In one embodiment, in the above S40, analyzing the audio quality of the audio data to obtain the early and late reflection ratio attributes includes:

[0134] S405, using a time-frequency analysis technique to identify early reflections and late reflections in the audio data;

[0135] S406, measuring the energy of the early reflection and the late reflection respectively;

[0136] S407, analyzing the ratio of early reflection energy to late reflection energy, and determining an early-late reflection ratio attribute value;

[0137] S408: Classify the early-late reflection ratio attribute value into corresponding reverberation degree levels according to a predetermined early-late reflection ratio threshold range.

[0138] In this embodiment, the system identifies early reflections and late reflections in the audio data through time-frequency analysis technology, measures their respective energies, and determines the early-late reflection ratio attribute through the energy ratio of the two. Early reflections and late reflections are two key components of audio reverberation, which represent the sound directly and indirectly reflected back from the sound source, respectively, affecting the spatial sense and clarity of the audio. By analyzing the early-late reflection ratio, the system can determine the degree of reverberation in the audio data and classify it into different levels according to the degree of reverberation.

[0139] The system uses time-frequency analysis technology to convert audio data into the time-frequency domain, and identifies early reflections (usually reflected sounds within a short period of time after speech is uttered) and late reflections (sounds that reach the ear after multiple reflections) that occur in the audio by analyzing changes in frequency and time. Early reflections generally last less than 50 milliseconds, while late reflections last longer.

[0140] The system measures the power of the early and late reflections in the audio. By analyzing the energy density in the time-frequency graph, the system can estimate the energy of the early and late reflections.

[0141] The system determines the ratio property by calculating the ratio of early reflection energy to late reflection energy. This ratio reflects the spatial sense and reverberation strength of the audio. A high ratio indicates that the early reflection energy is strong and the reverberation is weak; a low ratio indicates that the late reflection energy is strong and the reverberation is heavy.

[0142] The system classifies the calculated early and late reflection ratio values ​​into different reverberation levels according to the preset early and late reflection ratio threshold range. For example:

[0143] Mild reverberation (high ratio): The early reflection energy is strong and the speech is clear.

[0144] Moderate reverberation (medium ratio): The energy of early and late reflections is balanced, and the speech has some reverberation but is still intelligible.

[0145] Strong reverberation (low ratio): The late reflection energy is stronger and the reverberation is more severe, which may affect speech understanding.

[0146] The system uses time-frequency analysis techniques such as short-time Fourier transform (STFT) or wavelet transform on the audio data to convert the audio signal from the time domain to the time-frequency domain. By analyzing the changes in the frequency spectrum of the audio signal over time, the system can identify the time periods of early reflections and late reflections.

[0147] The frequency range and time period of early reflections and late reflections are obtained through time-frequency analysis, and the system measures the energy of these two reflection areas respectively. The energy of early reflections is mainly concentrated in the high-frequency band in a short period of time, while the energy of late reflections is more widely distributed and lasts longer. The system divides the total energy of early reflections by the total energy of late reflections to obtain the early-to-late reflection ratio. Based on the calculated reflection ratio, the system compares it with the predetermined threshold range and divides the audio data into different levels such as slight reverberation, moderate reverberation or strong reverberation according to the results. For telephone calls or meeting records in financial applications, audio with low reverberation will be marked as high quality.

[0148] In this embodiment, by analyzing the energy ratio of early reflections to late reflections in the audio data, the system can accurately evaluate the reverberation degree of the audio, thereby judging the clarity and spatial sense of the audio.

[0149] In one embodiment, the above 50 includes:

[0150] S501, extracting pitch values ​​from the audio data using a pitch detection tool, and recording the pitch values ​​to form a pitch sequence that changes over time;

[0151] S502, summarizing all extracted pitch values ​​in the audio data, analyzing the average of all pitch values, and determining the average pitch attribute value of the speaker;

[0152] S503, analyzing the distribution of the pitch values, calculating the standard deviation of the pitch values, and determining the pitch standard deviation attribute value;

[0153] S504, classifying the average pitch attribute value into corresponding pitch levels according to a predetermined pitch range;

[0154] S505: Classify the pitch standard deviation attribute value into corresponding fluctuation degree levels according to a predetermined standard deviation range.

[0155] In this embodiment, the pitch features in the audio data are extracted by using a pitch detection tool, and then the average pitch and pitch fluctuation of the speaker are analyzed. Pitch is one of the important attributes of speech. By analyzing it, it can provide important references for subsequent speech recognition, emotion recognition and other scenarios. The specific steps include extracting pitch values, calculating average pitch, calculating pitch standard deviation, and classifying these values ​​according to preset standards.

[0156] The system uses pitch detection tools such as autocorrelation, HPS (harmonic cepstrum) method or deep learning models to extract pitch values ​​from audio data. Each pitch value reflects the fundamental frequency (F0) of the speaker's voice at a specific moment. The system extracts a series of pitch values ​​from the entire audio signal to form a pitch sequence that changes over time.

[0157] The system summarizes the extracted pitch sequences and calculates the average of all pitch values ​​to obtain the speaker's average pitch attribute. This value reflects the basic pitch used by the speaker throughout the speech process and is an important indicator for identifying pronunciation features.

[0158] The system analyzes the distribution of pitch sequences and calculates the standard deviation of pitch values. The standard deviation of pitch reflects the degree of voice fluctuation of the speaker. The larger the standard deviation, the greater the pitch variation of the pronunciation and the more obvious the fluctuation; the smaller the standard deviation, the smoother the pronunciation and the less pitch variation.

[0159] The system classifies the average pitch attribute values ​​into different pitch classes according to the preset pitch range.

[0160] For example:

[0161] Bass: Below a certain preset frequency threshold.

[0162] Midrange: Within a certain preset middle frequency range.

[0163] Treble: Above a certain frequency threshold.

[0164] Pitch standard deviation classification:

[0165] The system divides the pitch fluctuation into different levels according to the size of the pitch standard deviation. For example:

[0166] Stable: A small standard deviation indicates little variation in pitch.

[0167] Moderate Fluctuation: The standard deviation is moderate, indicating that there is some variation in pitch.

[0168] Large fluctuations: A large standard deviation indicates significant pitch changes and greater volatility.

[0169] The system processes the audio signal frame by frame through a pitch detection algorithm (such as the autocorrelation method or a pitch detection tool based on a convolutional neural network), extracts the pitch value (F0) of each frame, and forms a pitch sequence. The accuracy of pitch detection depends on the algorithm and sampling frequency. The system summarizes each pitch value in the pitch sequence and calculates its arithmetic mean. The average pitch reflects the overall pitch characteristics of the speaker, and the system determines whether the speaker's pitch is low or high based on this result. The system calculates the deviation of each pitch value relative to the average pitch, and calculates the pitch standard deviation based on the deviation value. The standard deviation reflects the degree of fluctuation of the speaker's pitch throughout the speech process and is an important indicator for judging the stability of speech. The system compares the average pitch attribute value with the preset pitch range and classifies it as bass, middle, or high. Similarly, the system divides the degree of pitch fluctuation into three levels: stable, medium fluctuation, and large fluctuation according to the range of the pitch standard deviation.

[0170] Example description: A bank's telephone banking system uses this technical solution to analyze customer voice data and extract the customer's average pitch and pitch standard deviation attributes. For example, when the system detects that the customer's pitch standard deviation is large, it determines that the customer's mood fluctuates greatly and may be in an anxious state. The system will prompt the customer service staff to improve customer satisfaction by slowing down the tone or adjusting the communication method.

[0171] Through the above steps, the system of this embodiment can accurately extract and analyze the pitch characteristics of the speaker, including the average pitch and the pitch standard deviation, which is helpful to identify the speaker's emotional state and voice characteristics and improve the accuracy of the speech recognition system.

[0172] In one embodiment, the above S60 includes:

[0173] S601, applying speech recognition technology to the audio data to recognize and extract a phoneme sequence;

[0174] S602, determining the number of phonemes in the phoneme sequence to obtain the total number of phonemes in the audio data;

[0175] S603, measuring the total duration of the audio data to determine the audio length;

[0176] S604, analyzing the ratio of the total number of phonemes to the audio length to determine a speaking speed attribute value;

[0177] S605: Classify the speaking speed attribute value into corresponding speed levels according to a predetermined ratio range.

[0178] In this embodiment, the phoneme sequence is extracted by speech recognition technology, and the speaking speed attribute is calculated according to the ratio of the number of phonemes to the audio length. Speaking speed is an important feature of speech, reflecting the number of phonemes emitted by the speaker in a unit time. By analyzing the speaking speed attribute, the speaker's speech speed changes can be accurately identified, which has important application value in real-time speech processing, sentiment analysis and other scenarios.

[0179] The system first applies speech recognition technology to process the input audio data, identify and extract phoneme sequences. Phonemes are the smallest pronunciation units in speech. The system converts audio into a series of phonemes by analyzing the speech features in the audio signal.

[0180] The system counts the number of phonemes in the phoneme sequence to obtain the total number of phonemes in the audio data. This step ensures that the system can accurately analyze the actual volume of the speaker in a speech.

[0181] The system determines the duration of the audio by analyzing the total length of the audio file. The audio length represents the complete speech time period from the beginning to the end of the speaker.

[0182] The system divides the total number of phonemes by the audio length to obtain the speaking speed attribute value. The larger the ratio, the faster the speaker speaks; the smaller the ratio, the slower the speaker speaks.

[0183] The system classifies the speaking speed attribute values ​​into different speed levels according to a predetermined ratio range.

[0184] For example:

[0185] Slow: A smaller ratio indicates that the speaker speaks slowly.

[0186] Medium speed: The ratio is moderate, indicating that the speaker speaks at a moderate speed.

[0187] Fast: A larger ratio indicates that the speaker speaks faster.

[0188] The system uses speech recognition technology (such as HMM, DNN and other models) to process audio data and identify the phoneme sequence in the audio. The speech recognition model accurately identifies each phoneme uttered by the speaker by analyzing the frequency, duration and energy changes of the audio signal. The system counts each phoneme in the phoneme sequence to obtain the total number of all phonemes in the audio. By accurately calculating the number of phonemes, the system can identify the actual volume of the speaker in the voice. The system reads the metadata of the audio file to obtain the total duration of the audio. The audio length is one of the key variables to measure the speaker's speaking speed. The system calculates the number of phonemes emitted per second by dividing the total number of phonemes by the audio length. This ratio directly reflects the speaker's speaking speed. The system classifies the speaking speed as "slow", "medium" or "fast" according to the ratio range of the speaking speed. For example, when the ratio is less than a certain threshold, the system marks the speaking speed as "slow"; when the ratio is higher than a certain threshold, it is marked as "fast".

[0189] By analyzing the ratio of the number of phonemes in the audio data to the audio length, the system can accurately calculate the speaker's speaking speed attribute. It has important application value in scenarios such as real-time speech recognition, speech analysis, and customer emotion recognition, especially in the financial field, it can help the system improve the accuracy of voice command processing and transaction confirmation.

[0190] In one embodiment, the above S90 includes:

[0191] S901, obtaining a natural language description template, wherein the natural language description template includes a placeholder for a keyword of each attribute;

[0192] S902, filling the keyword of each attribute into the placeholder of the natural language description template to generate a structured prompt;

[0193] S903, inputting the structured prompt into a language model to generate a natural language sentence containing all attribute descriptions;

[0194] S904: Use the natural language sentence as annotation information of the audio data.

[0195] In this embodiment, a natural language description template is designed to generate structured annotation information based on each attribute of the audio data. The system fills the keywords of different attributes into the placeholders of the natural language description template to generate structured prompts. Finally, the system generates a natural language sentence containing all attribute descriptions through a language model and uses it as the annotation information of the audio data.

[0196] The system first designs a natural language description template, which contains multiple placeholders, each of which represents a certain attribute of the audio data. Attributes include gender, speaking speed, pitch, signal-to-noise ratio, etc. The keywords of these attributes will replace the placeholders in the template to form a structured natural language description.

[0197] The system selects corresponding keywords based on the attribute values ​​of each audio data and fills these keywords into the placeholders of the natural language description template. For example, the keyword for the gender attribute may be "male" or "female", and the keyword for the speech speed attribute may be "fast" or "slow".

[0198] After filling in the keywords of all attributes, the system generates a structured prompt. This prompt consists of keywords of multiple attributes, presenting a concise and structured description that reflects the characteristics of the audio data.

[0199] The system inputs the generated structured prompts into the language model (such as GPT, etc.), and the model generates complete natural language sentences based on the prompts. These sentences are more fluent and conform to the habits of natural language expression. The sentences contain all the attribute descriptions of the audio data and are presented in natural language, which is convenient for users to understand and use.

[0200] Finally, the system stores the generated natural language sentences as the annotation information of the audio data. The annotation information can be used in subsequent query, analysis, speech recognition and other tasks to provide the system with structured description content.

[0201] The system designs a natural language description template for each audio attribute, which contains multiple placeholders for attribute keywords to fill in. For example: Template sentence: "The speaker of this audio is {gender}, speaking at {speech speed}, pitch {pitch}, audio quality {signal-to-noise ratio}." Each attribute keyword will be filled in the corresponding placeholder.

[0202] The system selects appropriate keywords based on the attribute values ​​of the audio data. For example, for audio with a gender attribute of "male", a speaking speed attribute of "fast", and a signal-to-noise ratio of "high", the system will fill these keywords into the corresponding template placeholders. When all placeholders are filled, the system generates a structured prompt, such as: "The speaker of this audio is a male, speaking at a fast speed, with a medium pitch, and high audio quality." The system inputs the above structured prompt into the language model to generate a fluent natural language sentence. For example: "This audio is spoken by a male, who speaks at a fast speed, with a moderate pitch, and high audio quality." The system stores the generated natural language sentences as annotation information for the audio data for subsequent retrieval, classification, and analysis.

[0203] Example description: In a financial customer service center, the system generates natural language annotation information for each customer's call data through this solution. For example, the system generates a natural language sentence by analyzing the speech speed, pitch, and emotional fluctuations of the voice: "The customer is very emotional, speaks quickly, and the voice quality is clear." This annotation information can help customer service staff better understand customer needs and adjust service strategies.

[0204] This embodiment designs a natural language description template and generates structured prompts and natural language sentences based on each attribute, so that the system can generate more readable and understandable annotation information for audio data. These annotation information provides strong support for the query, retrieval and analysis of audio data, especially in the financial field, and can improve the processing efficiency and accuracy of voice trading systems and customer service systems.

[0205] In one embodiment, an audio data annotation device is provided, and the audio data annotation device corresponds one-to-one to the audio data annotation method in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of the audio data annotation device of the present invention. Data acquisition module 10, gender recognition module 20, accent recognition module 30, audio quality analysis module 40, pitch feature analysis module 50, speaking speed analysis module 60, gender and accent classification module 70, attribute mapping module 80 and annotation generation module 90. The functional modules are described in detail as follows:

[0206] A data acquisition module 10, used to acquire audio data;

[0207] A gender identification module 20, for determining a gender attribute using a gender classifier based on the audio data;

[0208] an accent recognition module 30 for determining an accent attribute based on the audio data using an accent classifier;

[0209] An audio quality analysis module 40, configured to analyze the audio quality of the audio data and obtain a signal-to-noise ratio attribute and an early-late reflection ratio value attribute;

[0210] A pitch feature analysis module 50, used to analyze the pitch features of the audio data and obtain the average pitch attribute and the pitch standard deviation attribute of the speaker;

[0211] A speaking speed analysis module 60, configured to analyze the ratio of the number of phonemes in the audio data to the audio length, and determine a speaking speed attribute;

[0212] a gender and accent classification module 70, configured to use the category name of the gender attribute as a keyword of the gender attribute, and use the category name of the accent attribute as a keyword of the accent attribute;

[0213] An attribute mapping module 80, for mapping a speaking speed attribute, a signal-to-noise ratio attribute, an early-late reflection ratio attribute, a speaker average pitch attribute, and a pitch standard deviation attribute to corresponding preset discrete category names, and using the preset discrete category names as keywords of the corresponding attributes;

[0214] The annotation generation module 90 is used to generate annotation information of the audio data based on the keyword of each attribute.

[0215] In one embodiment, the attribute mapping module 80 is specifically configured to:

[0216] Mapping the speaking speed attribute to discrete category names including fast, medium, and slow;

[0217] Mapping the signal-to-noise ratio attribute to discrete category names including high, medium, and low;

[0218] Mapping the early and late reflection ratio value attributes into discrete category names including strong reflection, medium reflection and weak reflection;

[0219] Mapping the speaker average pitch attribute to discrete category names including high, middle, and low;

[0220] Mapping the pitch standard deviation attribute to discrete category names including large fluctuation, medium fluctuation, and small fluctuation;

[0221] Use discrete category names as keys for corresponding attributes.

[0222] In one embodiment, the audio quality analysis module 40 is specifically configured to:

[0223] Using a spectrum analysis module to separate the audio data into an effective speech portion and a background noise portion;

[0224] respectively measuring the power of the effective speech part and the background noise part;

[0225] By analyzing the ratio of the effective speech power to the background noise power, the signal-to-noise ratio attribute value is determined;

[0226] The signal-to-noise ratio attribute value is classified into a corresponding audio quality level according to a predetermined signal-to-noise ratio threshold range.

[0227] In one embodiment, the audio quality analysis module 40 is specifically configured to:

[0228] identifying early reflections and late reflections in the audio data using time-frequency analysis techniques;

[0229] measuring the energy of the early reflection and the late reflection respectively;

[0230] By analyzing the ratio of early reflection energy to late reflection energy, the early-late reflection ratio attribute value is determined;

[0231] According to a predetermined early-late reflection ratio threshold range, the early-late reflection ratio attribute value is classified into corresponding reverberation degree levels.

[0232] In one embodiment, the pitch feature analysis module 50 is specifically used for:

[0233] Extracting pitch values ​​from the audio data using a pitch detection tool, and recording the pitch values ​​to form a pitch sequence that changes over time;

[0234] Summarize all extracted pitch values ​​in the audio data, analyze the average of all pitch values, and determine the average pitch attribute value of the speaker;

[0235] Analyze the distribution of pitch values, calculate the standard deviation of the pitch values, and determine the pitch standard deviation attribute value;

[0236] Classifying the average pitch attribute value into corresponding pitch classes according to a predetermined pitch range;

[0237] According to a predetermined standard deviation range, the pitch standard deviation attribute values ​​are classified into corresponding fluctuation degree levels.

[0238] In one embodiment, the speaking speed analysis module 60 is specifically used to:

[0239] Applying speech recognition technology to the audio data to identify and extract phoneme sequences;

[0240] Determine the number of phonemes in the phoneme sequence to obtain the total number of phonemes in the audio data;

[0241] Measuring the total duration of the audio data to determine the audio length;

[0242] Analyze the ratio of the total number of phonemes to the audio length to determine a speaking speed attribute value;

[0243] According to a predetermined ratio range, the speaking speed attribute values ​​are classified into corresponding speed levels.

[0244] In one embodiment, the annotation generation module 90 is specifically used to:

[0245] Design a natural language description template, wherein the natural language description template includes a placeholder for a keyword of each attribute;

[0246] Filling the keywords of each attribute into the placeholder of the natural language description template to generate a structured prompt;

[0247] Inputting the structured prompt into a language model to generate a natural language sentence containing all attribute descriptions;

[0248] The natural language sentence is used as the annotation information of the audio data.

[0249] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, the functions or steps on the server side of an audio data annotation method are implemented.

[0250] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a user-side of an audio data annotation method

[0251] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0252] Get audio data;

[0253] Based on the audio data, determining a gender attribute using a gender classifier;

[0254] determining an accent attribute using an accent classifier based on the audio data;

[0255] Analyze the audio quality of the audio data to obtain a signal-to-noise ratio attribute and an early-late reflection ratio value attribute;

[0256] Analyze the pitch characteristics of the audio data to obtain the average pitch attribute and the pitch standard deviation attribute of the speaker;

[0257] Analyzing the ratio of the number of phonemes in the audio data to the length of the audio data to determine a speaking speed attribute;

[0258] Using the category name of the gender attribute as a keyword of the gender attribute, and using the category name of the accent attribute as a keyword of the accent attribute;

[0259] Mapping the speaking speed attribute, the signal-to-noise ratio attribute, the early-late reflection ratio attribute, the speaker average pitch attribute, and the pitch standard deviation attribute to corresponding preset discrete category names, and using the preset discrete category names as keywords for the corresponding attributes respectively;

[0260] The tag information of the audio data is generated based on the keyword of each attribute.

[0261] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0262] Get audio data;

[0263] Based on the audio data, determining a gender attribute using a gender classifier;

[0264] determining an accent attribute using an accent classifier based on the audio data;

[0265] Analyze the audio quality of the audio data to obtain a signal-to-noise ratio attribute and an early-late reflection ratio value attribute;

[0266] Analyze the pitch characteristics of the audio data to obtain the average pitch attribute and the pitch standard deviation attribute of the speaker;

[0267] Analyzing the ratio of the number of phonemes in the audio data to the length of the audio data to determine a speaking speed attribute;

[0268] Using the category name of the gender attribute as a keyword of the gender attribute, and using the category name of the accent attribute as a keyword of the accent attribute;

[0269] Mapping the speaking speed attribute, the signal-to-noise ratio attribute, the early-late reflection ratio attribute, the speaker average pitch attribute, and the pitch standard deviation attribute to corresponding preset discrete category names, and using the preset discrete category names as keywords for the corresponding attributes respectively;

[0270] The tag information of the audio data is generated based on the keyword of each attribute.

[0271] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0272] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0273] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0274] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for annotating audio data, characterized in that: The following steps are involved: Get audio data; Based on the audio data, determining a gender attribute using a gender classifier; determining an accent attribute using an accent classifier based on the audio data; Analyze the audio quality of the audio data to obtain a signal-to-noise ratio attribute and an early-late reflection ratio value attribute; Analyze the pitch characteristics of the audio data to obtain the average pitch attribute and the pitch standard deviation attribute of the speaker; Analyzing the ratio of the number of phonemes in the audio data to the length of the audio data to determine a speaking speed attribute; Using the category name of the gender attribute as a keyword of the gender attribute, and using the category name of the accent attribute as a keyword of the accent attribute; Mapping the speaking speed attribute, the signal-to-noise ratio attribute, the early-late reflection ratio attribute, the speaker average pitch attribute, and the pitch standard deviation attribute to corresponding preset discrete category names, and using the preset discrete category names as keywords for the corresponding attributes respectively; The tag information of the audio data is generated based on the keyword of each attribute.

2. The audio data annotation method according to claim 1, characterized in that: Mapping the speaking speed attribute, the signal-to-noise ratio attribute, the early-late reflection ratio attribute, the speaker average pitch attribute, and the pitch standard deviation attribute to corresponding preset discrete category names, and using the preset discrete category names as keywords for the corresponding attributes, respectively, includes: Mapping the speaking speed attribute to discrete category names including fast, medium, and slow; Mapping the signal-to-noise ratio attribute to discrete category names including high, medium, and low; Mapping the early and late reflection ratio value attributes into discrete category names including strong reflection, medium reflection and weak reflection; Mapping the speaker average pitch attribute to discrete category names including high, middle, and low; Mapping the pitch standard deviation attribute to discrete category names including large fluctuation, medium fluctuation, and small fluctuation; Use discrete category names as keys for corresponding attributes.

3. The audio data annotation method according to claim 1, characterized in that: Analyzing the audio quality of the audio data to obtain a signal-to-noise ratio attribute includes: Using a spectrum analysis module to separate the audio data into an effective speech portion and a background noise portion; respectively measuring the power of the effective speech part and the background noise part; Analyze the ratio of the effective speech power to the background noise power to determine the signal-to-noise ratio attribute value; The signal-to-noise ratio attribute value is classified into a corresponding audio quality level according to a predetermined signal-to-noise ratio threshold range.

4. The audio data tagging method as claimed in claim 1, characterized in that: Analyze the audio quality of the audio data to obtain early and late reflection ratio attributes, including: identifying early reflections and late reflections in the audio data using time-frequency analysis techniques; measuring the energy of the early reflection and the late reflection respectively; Analyze the ratio of early reflection energy to late reflection energy to determine the attribute value of the early-late reflection ratio; According to a predetermined early-late reflection ratio threshold range, the early-late reflection ratio attribute value is classified into corresponding reverberation degree levels.

5. The audio data annotation method according to claim 1, characterized in that: Analyzing the pitch features of the audio data to obtain the average pitch attribute and the pitch standard deviation attribute of the speaker includes: Extracting pitch values ​​from the audio data using a pitch detection tool, and recording the pitch values ​​to form a pitch sequence that changes over time; Summarize all extracted pitch values ​​in the audio data, analyze the average of all pitch values, and determine the average pitch attribute value of the speaker; Analyze the distribution of pitch values, calculate the standard deviation of the pitch values, and determine the pitch standard deviation attribute value; Classifying the average pitch attribute value into corresponding pitch classes according to a predetermined pitch range; According to a predetermined standard deviation range, the pitch standard deviation attribute values ​​are classified into corresponding fluctuation degree levels.

6. The audio data annotation method according to claim 1, characterized in that: Analyzing the ratio of the number of phonemes in the audio data to the audio length to determine the speaking speed attribute includes: Applying speech recognition technology to the audio data to identify and extract phoneme sequences; Determine the number of phonemes in the phoneme sequence to obtain the total number of phonemes in the audio data; Measuring the total duration of the audio data to determine the audio length; Analyze the ratio of the total number of phonemes to the audio length to determine a speaking speed attribute value; According to a predetermined ratio range, the speaking speed attribute value is classified into corresponding speed levels.

7. The audio data annotation method according to claim 1, characterized in that: Generating the annotation information of the audio data based on the keyword of each attribute includes: Obtain a natural language description template, wherein the natural language description template includes a placeholder for a keyword of each attribute; Filling the keywords of each attribute into the placeholder of the natural language description template to generate a structured prompt; Inputting the structured prompt into a language model to generate a natural language sentence containing all attribute descriptions; The natural language sentence is used as the annotation information of the audio data.

8. An audio data annotation device, characterized in that: The audio data annotation device comprises: A data acquisition module, used for acquiring audio data; A gender recognition module, configured to determine a gender attribute based on the audio data using a gender classifier; an accent recognition module for determining accent attributes using an accent classifier based on the audio data; An audio quality analysis module, used to analyze the audio quality of the audio data and obtain a signal-to-noise ratio attribute and an early-late reflection ratio value attribute; A pitch feature analysis module, used to analyze the pitch features of the audio data and obtain the average pitch attribute and the pitch standard deviation attribute of the speaker; A speaking speed analysis module, used to analyze the ratio of the number of phonemes in the audio data to the audio length, and determine the speaking speed attribute; a gender and accent classification module, used to use the category name of the gender attribute as a keyword of the gender attribute, and use the category name of the accent attribute as a keyword of the accent attribute; An attribute mapping module, used to map a speaking speed attribute, a signal-to-noise ratio attribute, an early-late reflection ratio attribute, a speaker average pitch attribute, and a pitch standard deviation attribute to corresponding preset discrete category names, and use the preset discrete category names as keywords for the corresponding attributes respectively; The annotation generation module is used to generate annotation information of the audio data based on the keyword of each attribute.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and an audio data annotation program stored in the memory and executable on the processor. When the audio data annotation program is executed by the processor, the steps of the audio data annotation method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores an audio data annotation program, which, when executed by a processor, implements the steps of the audio data annotation method according to any one of claims 1 to 7.

Images