Audio theme classification method and system

Abstract

The invention provides an audio theme classification method and system, and the method comprises the steps: extracting the time axis data of a speaker from a to-be-classified audio file, the time axis data comprising the identifier of the speaker and the starting and ending time of a speaking segment of the speaker; according to the time axis data, converting the to-be-classified audio file into a transliteration text; a structured text is generated according to the time axis data and the transliteration text, the structured text comprises the transliteration text of speaking segments and speakers corresponding to the speaking segments, and the speaking segments are arranged according to a time sequence; and according to the structured text construction prompt, obtaining an audio file theme classification result output by the large language model. According to the method, the transliteration text in the structured text can represent the speaking content, and the speaker and the speaking sequence can represent information such as who is speaking and a dialogue interaction mode, so that key discrimination features are provided for a subsequent classification model, and the accuracy of theme classification of the audio file can be improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of artificial intelligence and audio processing technology, and specifically relates to an audio theme classification method and system. BACKGROUND

[0002] With the rapid development of Internet audio platforms, the number of user-generated content (UGC) such as podcasts, audiobooks, online courses, etc. is growing explosively. In order to improve user experience and achieve personalized recommendation and precise search, it is necessary to effectively classify these massive audio content and label them with tags such as "news", "entertainment", "education", etc. Currently, the realization of audio theme classification mainly relies on the following technical solutions: (1) Artificial classification scheme: This scheme relies on platform operators or users to classify and label audio content. Operators need to listen to audio content and manually add labels according to the preset classification system. Or the platform allows the content uploader to choose the label. The artificial classification scheme has the following significant defects: manual classification by operators, huge workload, high cost, low efficiency, and unable to adapt to the growth of massive data; self-labeling by users, label quality is uneven, poor standardization, and there are a lot of errors or irrelevant labels, which are difficult to use for downstream precise recommendation and search.

[0003] The artificial classification scheme has low automation and processing flow bottlenecks: whether relying on operators or users, the artificial classification scheme cannot achieve scale and standardization in essence. From a technical point of view, this scheme lacks effective quality control and consistency guarantee mechanism, resulting in low signal-to-noise ratio of label data; at the same time, its processing efficiency is completely limited by manpower, and in the face of tens of thousands of new audio every day, there is a serious processing flow bottleneck, which cannot meet the technical requirements of the platform for real-time understanding and distribution of content.

[0004] (2) Traditional machine learning-based text classification scheme: This is the current industry more common automatic classification scheme. Its main process is: first, use automatic speech recognition (ASR) technology to convert audio content into pure text; Then, this classification problem is regarded as a standard natural language processing (NLP) text classification task; Finally, use traditional machine learning models (such as support vector machine SVM, naive Bayes) or deep learning models (such as convolutional neural network CNN, recurrent neural network RNN, Transformer, etc.) to train and classify the transcribed text. Although the traditional machine learning-based text classification scheme has realized automation, it has the following technical limitations: severe dependence on large-scale labeled data: training classification models requires a large amount of high-quality labeled audio text data, the data labeling cost is extremely high, and the cycle is long; Limited generalization ability: for new topics or network hot content, the model has poor generalization ability, and is prone to classification errors, which requires re-labeling data and training the model to adapt.

[0005] In summary, the existing technical solutions are either too high in cost to be scaled up or have insufficient classification accuracy and flexibility due to technical limitations, making it difficult to meet the current demand for efficient, accurate, and flexible automated topic classification of massive user-generated content on large audio platforms. SUMMARY

[0006] The embodiments of the present application provide an audio topic classification method and system, which realizes efficient, accurate and automated topic classification of massive audio content.

[0007] In a first aspect, the embodiments of the present application provide an audio topic classification method, comprising: extracting the time axis data of the speaker from the audio file to be classified, the time axis data comprising the identifier of the speaker and the start and end time of the speech segment of the speaker; converting the audio file to be classified into a transcribed text according to the time axis data; generating a structured text according to the time axis data and the transcribed text, the structured text comprising the transcribed text of the speech segment and the speaker corresponding to the speech segment, and the speech segment being arranged in chronological order; constructing a prompt according to the structured text to obtain an audio file topic classification result output by a large language model.

[0008] Among them, the conversion of the audio file to be classified into a transcribed text according to the time axis data comprises: Based on the speaker's identifier and the start and end times of the speech segments in the timeline data, independent speech segments are cut out from the audio files to be classified. The spoken segment is processed by an automatic speech recognition engine to obtain the transcribed text of the spoken segment.

[0009] The step of generating structured text based on the timeline data and the transcribed text includes: The speaker and the start and end times of the speech segment are obtained from the timeline data. The speaker corresponding to the speech segment is combined with the transcribed text corresponding to the speech segment to obtain the speech segment unit; The playback order of the speech segments is determined based on the start and end times of the speech segments; The speech segments are assembled in chronological order to generate structured text.

[0010] The method further includes: Based on the start and end times of the speech segments, extract the identifier of the speaker who started speaking earliest from the timeline data; Based on the timeline data, count the number of times the speaker speaks; Add at least one of the following to the structured text: the number of times a speaker has spoken and the identifier of the speaker who first started speaking.

[0011] The method further includes: Detect overlapping segments spoken by multiple speakers simultaneously; Obtain the speaker's identifier and the speaker's speech content from the overlapping segments; The number of overlapping segments, the total duration of overlapping segments, and the overlap ratio are determined, wherein the overlap ratio is the ratio of the total duration of overlapping segments to the total duration of speech, and the total duration of speech is the total duration of the speech segment; Add at least one of the following to the structured text: the number of overlapping segments, the total duration of the overlapping segments, the overlap ratio, the speaker's identifier in the overlapping segments, and the speaker's speech content.

[0012] The method further includes: Determine the time interval between two adjacent speech segments; Based on multiple time intervals, at least one of the following is determined: average interval duration, time interval distribution, and short interval proportion of the audio file to be classified, wherein the short interval refers to an interval with a duration less than a threshold. Add at least one of the average interval duration, the time interval distribution, and the proportion of short intervals to the structured text.

[0013] The method further includes: Based on the speaker's identifier and the start and end times of the speaker's speech segments in the timeline data, determine the speaker's total speaking time and the average speaking time of each speech. Calculate the ratio of the speaker's total speaking time to the total speech duration to obtain the speaker's speaking time percentage, where the total speech duration is the total duration of the speaking segment; Add at least one of the speaker's total speaking time, the average speaking time, and the speaking time percentage to the structured text.

[0014] The method further includes: Based on the speaker identifiers and the start and end times of the speech segments in the timeline data, the order in which the speakers alternate speaking is obtained; The speaker's continuity rate is obtained by calculating the ratio of the number of consecutive times a speaker speaks to the total number of times they speak. Add at least one of the speaker alternation order and speaker continuity rate to the structured text.

[0015] The method further includes: Extract non-verbal content features from the audio files to be classified, wherein the non-verbal content features include at least one of voice emotion, speech rate, pitch, energy, and prosodic variation; The non-linguistic content features are added to the structured text.

[0016] The method further includes: Detect whether the audio file to be classified contains background music or ambient sound; Identify the environment type of the audio file to be classified; The results of the detection and at least one of the identified environment types are added to the structured text.

[0017] Secondly, this application provides an audio subject classification system, including: The extraction unit is used to extract the speaker's timeline data from the audio file to be classified. The timeline data includes the speaker's identifier and the start and end times of the speaker's speech segments. A conversion unit is used to convert the audio file to be classified into transcribed text based on the timeline data; The generation unit is used to generate structured text based on the timeline data and the transcribed text. The structured text includes the transcribed text of the speech segments and the speakers corresponding to the speech segments, and the speech segments are arranged in chronological order. The classification unit is used to construct prompts based on the structured text to obtain the audio file subject classification results output by the large language model.

[0018] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0019] Fourthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods described above.

[0020] The audio subject classification method and system of this application have the following beneficial effects: In this application, structured text is generated based on timeline data and transcribed text obtained from audio files. The structured text includes not only the transcribed text but also the speaker and speaking order. The transcribed text can represent the content of the speech, while the speaker and speaking order can represent who is speaking and the dialogue interaction pattern, thus providing key discriminative features for subsequent classification models and helping to improve the accuracy of subject classification of audio files. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the audio subject classification method in Embodiment 1 of this application. Figure 1 ; Figure 2 This is a schematic diagram of the audio subject classification method in Embodiment 1 of this application. Figure 2 ; Figure 3 This is a schematic diagram of the audio subject classification method in Embodiment 1 of this application. Figure 3 ; Figure 4 This is a schematic diagram of the audio subject classification method in Embodiment 2 of this application; Figure 5 This is a schematic diagram of the audio subject classification method in Embodiment 3 of this application; Figure 6 This is a schematic diagram of the audio subject classification system structure in this application. Detailed Implementation

[0022] The present application will be further described below with reference to the accompanying drawings and embodiments.

[0023] In the following description, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The following description provides multiple embodiments of the invention, which can be substituted or combined with each other. Therefore, this application can also be considered to include all possible combinations of the same and / or different embodiments described. Thus, if one embodiment includes features A, B, and C, and another embodiment includes features B and D, then this application should also be considered to include embodiments containing one or more other possible combinations of features A, B, C, and D, even if such embodiments are not explicitly described in the following text.

[0024] For audio subject classification, the commonly used automated classification scheme is a text classification scheme based on traditional machine learning, but this scheme has the following problems: High model training costs and difficulties in iterating classification systems: The effectiveness of text classification schemes based on traditional machine learning heavily relies on large-scale, high-quality labeled data. Training a usable classification model requires a significant investment of manpower for data collection and meticulous labeling, leading not only to extremely high upfront R&D costs but also making iterative classification system iteration exceptionally difficult. When new subject categories need to be added or existing category definitions need to be optimized, large-scale data labeling and model training must be restarted, resulting in rigid technical processes that cannot respond nimbly to business needs.

[0025] The coarse-grained classification granularity fails to effectively handle cross-topic content: Current technologies simplify audio classification to a single-modal text classification problem, making it difficult to address the prevalent phenomenon of cross-topic content. Most audio content is not a single topic; for example, a technology interview program may contain both "technology" and "interview" attributes. Single-label classification models suffer from information loss, while multi-label models exponentially increase the complexity of the labeling system and model training, making them difficult to implement in practice. This coarse-grained classification granularity cannot accurately depict the diversity of audio content, resulting in insufficient accuracy in downstream recommendation systems and impacting user experience.

[0026] The lack of key classification features limits the accuracy of existing solutions: The most fundamental flaw in current technologies lies in the dimensionality reduction of information modalities. They merely utilize the text transcribed by Automatic Speech Recognition (ASR), completely ignoring the structured information inherent in the audio itself—crucial for subject classification—such as the number of speakers, their relationships (e.g., host and guest), and the interactive patterns of the dialogue (e.g., monologue, conversation, debate). These features are key clues for distinguishing between audiobooks, interviews, courses, dramas, and debates. Due to the absence of these crucial classification features, the classification accuracy of existing solutions faces a significant technical ceiling.

[0027] Based on the inventors' analysis above, existing technologies have significant technical deficiencies in terms of model flexibility, classification precision, and feature completeness. Therefore, there is an urgent need in this field for a novel technical solution that can eliminate reliance on large-scale manual annotation, flexibly handle cross-topics, and fully utilize audio content and dialogue structure in audio recordings, thereby achieving efficient, accurate, and automated topic classification of massive amounts of audio content.

[0028] Example 1 like Figure 1 As shown, the audio subject classification method of this application includes: S101, extracting speaker timeline data from the audio file to be classified, wherein the timeline data includes the speaker's identifier and the start and end times of the speaker's speech segments; S103, converting the audio file to be classified into transcribed text based on the timeline data; S105, generating structured text based on the timeline data and the transcribed text, wherein the structured text includes the transcribed text of the speech segments and the speaker corresponding to the speech segments, and the speech segments are arranged in chronological order; S107, constructing prompts based on the structured text to obtain the audio file subject classification result output by a large language model. Each step is described below.

[0029] S101, Extract the speaker's timeline data from the audio file to be classified. The timeline data includes the speaker's identifier and the start and end times of the speaker's speech segments.

[0030] In this step, the audio files to be classified are processed through speaker diarization to extract the timeline data of the speakers. The timeline data includes each speaker in the audio file, such as speaker A and speaker B, and each speaker has a unique identifier that can distinguish different speakers. The timeline data also includes the start and end times of all speaking segments for each speaker. For example, if speaker A has three speaking segments, the timeline data records the start and end times of each of the three speaking segments; if speaker B has two speaking segments, the timeline data records the start and end times of each of the two speaking segments.

[0031] Among them, Speaker Diarization is used to automatically detect "who speaks and when" in an audio clip. It can divide the audio stream into multiple segments and aggregate segments belonging to the same speaker together without knowing the speaker's identity and number in advance.

[0032] S103, based on the timeline data, converts the audio files to be classified into transcribed text.

[0033] Based on the speaker's identifier and the start and end times of the speech segments in the speaker's timeline data obtained in step S101, each independent speech segment is segmented one by one from the audio file to be classified. The segmented speech segments are then sent to the automatic speech recognition engine, which processes each segment and outputs a transcribed text of each segment. Because the speech segments are segmented based on the speaker's identifier and the start and end times of the speech segments, the corresponding speaker can be identified from the transcribed text of each segment. This step can obtain the transcribed text of each speaker. Automatic Speech Recognition (ASR) technology converts human speech signals into corresponding text sequences for voice interaction and audio content textification.

[0034] S105, Generate structured text based on timeline data and transcribed text. The structured text includes transcribed text of speech segments and the speakers corresponding to the speech segments, with the speech segments arranged in chronological order.

[0035] like Figure 2 As shown, in step S1051, the speaker and the start and end times of the speech segment are obtained from the timeline data obtained in step S101. For example, a speech segment of 10-20 seconds corresponds to speaker A. In step S1052, the speaker corresponding to the speech segment is combined with the transcribed text corresponding to the speech segment to obtain a speech segment unit. When there are multiple speech segments, multiple speech segment units can be obtained. In step S1053, the playback time order of the speech segments is determined according to the start and end times of the speech segments, that is, the playback order of multiple speech segments in the audio file is determined according to the order of the start and end times of the speech segments. In step S1054, the speech segment units are concatenated according to the playback time order of the speech segments to generate structured text. That is, multiple speech segments are concatenated according to the playback time order of the speech segments, and the multiple speech segments are arranged in the playback time order. The speech segment unit includes the transcribed text corresponding to the speech segment and the speaker's identifier. The speech segment unit may also include the start and end times of the speech segment.

[0036] In this step, the transcribed text data obtained in step S103 and the timeline data obtained in step S101 are merged. Based on the playback time order of the audio file, the transcribed text corresponding to the speaker identifier and the speaker's speech segment is concatenated and formatted to generate structured text. The time order can be determined based on the start and end times of the speech segment in the timeline data, and the purpose of formatting is to ensure that the concatenated data meets the requirements of subsequent processing steps.

[0037] S107: Based on the structured text prompts, obtain the audio file subject classification results output by the large language model.

[0038] like Figure 3 As shown, in step S1071, the predefined topic classification system is combined with the structured text obtained in step S105 to construct a prompt for guiding the large language model in classification. The prompt may include at least one of the following: role setting, task description, contextual information, classification system definition, and output format requirements. The role setting assigns an expert role to the large language model; the task description is, for example, determining the topic category of audio and outputting a result with confidence in a predetermined format; the contextual information may include the structured text obtained in step S105; the classification system definition includes candidate topic categories and a feature description of each topic category, such as news: timeliness and objective narration; the output format requirements may include, for example, the output format, topic category, and confidence level.

[0039] In step S1072, the audio file subject classification results output by the large language model are obtained. The constructed prompts are sent to the large language model (LMM) via an API request. The large language model receives the prompts, comprehensively understands the structured text content, subject classification system definition, and task description contained in the prompts, and generates and outputs the corresponding audio file subject classification results. The audio file subject classification results output by the large language model are received; these results include one or more subject categories and the confidence level corresponding to each subject category.

[0040] In some embodiments, this application further includes: constructing batch processing or streaming processing tasks, processing each audio file in the audio database according to steps S101-S107, and storing a unique identifier for each audio file and the subject classification result of that audio file. The subject classification result of the audio file can be directly called by downstream recommendation, search and other business systems to achieve more refined operations and user services.

[0041] In this application, structured text is generated based on timeline data and transcribed text obtained from audio files. The structured text includes not only the transcribed text but also the speaker and speaking order. The transcribed text can represent what was said, while the speaker and speaking order can represent who is speaking and the dialogue interaction pattern, thus providing key discriminative features for subsequent classification models and fundamentally improving the accuracy of audio file classification.

[0042] Example 2 like Figure 4 As shown, the audio subject classification method of this application includes: S21, Speaker Segmentation and Clustering The purpose of this step is to automatically detect speaker information in the input audio, including the number of speakers and the start and end times of each speaker's speech.

[0043] The input for this step is the audio file to be categorized. This audio file can be in various common formats (such as WAV, MP3, M4A, etc.), and there are no restrictions on duration, number of channels, or number of speakers.

[0044] The processing steps in this step involve calling the Speaker Diarization module to process the input audio. This module uses a deep learning model (e.g., a model based on x-vector or ECAPA-TDNN (Emphasized Channel Attention, Propagation and Aggregation in TDNN, a time-delay neural network emphasizing channel attention, propagation, and aggregation)) to extract acoustic features from audio frames, and then uses clustering algorithms (such as spectral clustering, agglomerative hierarchical clustering, etc.) to group speech segments belonging to the same speaker together.

[0045] The output of this step is structured speaker timeline data. This data records the precise start and end timestamps of each identified speaker (identified by a unique ID, such as speaker A, speaker B, etc.) and all their speaking segments. Its data structure can be represented as {(speaker_id_1, start_time_1, end_time_1), (speaker_id_2, start_time_2, end_time_2), ...}.

[0046] S22, Segmented Automatic Speech Recognition (ASR) Transcription The purpose of this step is to accurately convert each speaker's speech segment into text content.

[0047] The inputs for this step are the original audio file and the speaker timeline data output from step S1.

[0048] The processing steps in this step are as follows: Based on the speaker's timeline data, each individual speech segment is extracted from the original audio. The extracted audio segments are then sent to the Automatic Speech Recognition (ASR) engine. The ASR engine processes each speech segment and outputs the corresponding transcribed text.

[0049] The output of this step is a dataset containing speaker identifiers and their corresponding transcribed text. Its data structure can be represented as {(speaker_id_1, text_1), (speaker_id_2, text_2), ...}.

[0050] S23, Generate structured text The purpose of this step is to fuse the timeline information and transcribed text information obtained from the first two steps to generate structured text that can fully reflect the audio content and dialogue structure and can be understood by large-scale language models.

[0051] The inputs for this step are: the speaker timeline data output from step S21 and the transcribed text data output from step S22.

[0052] The processing steps in this step involve concatenating and formatting the speaker identifiers and their corresponding transcribed text according to chronological order. In one embodiment, plain text in a script-like format can be generated to clearly display the dialogue flow. In another embodiment, to include richer information (such as start and end timestamps of speech segments), structured data in JSON (JavaScript Object Notation) format can be generated, which can improve the accuracy of the final audio file subject classification results.

[0053] Example 1 (plain text format): Speaker A: "Welcome to today's news program. I'm your anchor, Xiao Wang." Speaker B: "Hello everyone, I am Professor Li, your guest." Speaker A: "Today we're going to discuss the latest developments in artificial intelligence..." Example 2 (JSON format): JSON formatted structured text includes... [{"speaker": "A", "start": 0.5, "end": 4.2, "text": "Welcome to today's news program. I'm your anchor, Xiao Wang."}, {"speaker": "B", "start": 4.5, "end": 7.1, "text": "Hello everyone, I am Professor Li, your guest."}, ...] The output of this step is a string or a JSON object, which is structured text that incorporates speaker information.

[0054] The structured text in this step can be used not only for subsequent audio file subject classification, but also directly for a wider range of downstream tasks, such as content summarization, role analysis, and dialogue behavior analysis, providing a solid data foundation for in-depth understanding and multi-dimensional application of audio content.

[0055] In some embodiments, the audio subject classification method of this application further includes: extracting the identifier of the earliest speaker from the timeline data according to the start and end times of the speech segments; that is, the speaker corresponding to the speech segment with the earliest start time is the earliest speaker. Based on the timeline data, the number of times each speaker speaks is counted; for example, if speaker A has 15 speech segments, then speaker A speaks 15 times. At least one of the speaker's speaking count and the identifier of the earliest speaker is added to the structured text. The structured text includes at least one of the speaker's speaking count and the identifier of the earliest speaker.

[0056] From the output of speaker segmentation and clustering, we can extract the number of times each speaker spoke and the earliest speaker to start speaking. We count the number of times each speaker speaks throughout the entire audio recording. Then, we sort the speakers chronologically to determine the earliest speaker to start speaking.

[0057] The value of the number of times each speaker speaks and the identifier of the earliest speaker in the structured text for the subject classification of audio files is shown in the table below:

[0058] In subsequent steps, structured text can be used for role analysis. The value of information such as the number of times a speaker speaks and the earliest speaker in the structured text for role analysis is as follows: If speaker A is both the earliest speaker and the speaker with the most speaking times, then speaker A is highly likely to be the host or main speaker. If speaker B is both the earliest speaker and the speaker with the fewest speaking times, speaker B may be the opening guest or a narrator. If speaker C has a limited number of speaking times, speaker C may be a supporting character, a student, or an occasional interjector.

[0059] Example of adding a new field to structured text: First speaker: SPEAKER_00 Speaker statistics: SPEAKER_00: Speaked 12 times, was the first speaker to start. SPEAKER_01: Speaks 10 times.

[0060] In some embodiments, the audio subject classification method further includes: detecting overlapping segments where multiple speakers speak simultaneously; obtaining the speaker identifiers and speaker content in the overlapping segments; calculating the number of overlapping segments, the total duration of the overlapping segments, and the overlap ratio, where the overlap ratio is the ratio of the total duration of the overlapping segments to the total speech duration, and the total speech duration is the total duration of the speaking segments. At least one of the following—the number of overlapping segments, the total duration of the overlapping segments, the overlap ratio, the speaker identifiers in the overlapping segments, and the speaker content—is added to the structured text. The structured text includes at least one of the following: the number of overlapping segments, the total duration of the overlapping segments, the overlap ratio, the speaker identifiers in the overlapping segments, and the speaker content.

[0061] Speaker overlap refers to a situation where two or more speakers are speaking simultaneously. Modern speaker segmentation systems (such as pyannote.audio 3.0 and above) support overlap detection and can output: the start and end times of the overlap, a list of speakers involved in the overlap, and the proportion of the overlap duration to the total speech duration. Speech transcription of the overlapping portion includes: after detecting the overlap time period, extracting the audio for that time period, using speech separation techniques (such as the SepFormer model) to separate the overlapping speech into independent single-speaker speech, transcribing each separated speech individually, and labeling the transcription results with the corresponding speaker.

[0062] The value of overlap detection for audio file classification is shown below:

[0063] If the overlap is greater than 15% and the intervals between speaking segments are short, it is highly likely to be a debate or heated discussion. If the overlap is between 5% and 15% and is mainly short-duration, it is likely an interview. If the overlap is less than 5%, it is an orderly speaking scenario, such as a lecture or news report.

[0064] Example of adding a new field to structured text: Overlap statistics: Number of overlapping segments: 8 Total overlap duration: 12.5 seconds Overlap percentage: 4.2% Overlap strength: Moderate Overlapping fragment details: 14.8 seconds - 15.2 seconds: SPEAKER_00 and SPEAKER_01 speak simultaneously. SPEAKER_00: "...viewed from multiple angles" SPEAKER_01: "No, no..." In some embodiments, the audio subject classification method of this application further includes: calculating the time interval between two adjacent speech segments by subtracting the end time of the previous speech segment from the start time of the current speech segment, where the time interval refers to the silence time between two adjacent speech segments. Multiple time intervals are then added together to obtain the total time interval.

[0065] Calculate the average interval duration of the audio files to be classified based on the total time intervals and the number of speech segments (number of time intervals). And / or calculate the time interval distribution, which may include, for example, minimum, maximum, and standard deviation values. And / or calculate the percentage of short intervals with durations less than a threshold based on the durations of multiple time intervals and the number of time intervals. The threshold may be, for example, 0.5 seconds. Add at least one of the average interval duration, time interval distribution, and short interval percentage to the structured text. The structured text includes at least one of the average interval duration, time interval distribution, and short interval percentage.

[0066] The value of time interval features for audio file classification is shown below:

[0067] If the average time interval is less than 0.3 seconds and the variance of the time interval is small, it is highly likely to be a debate or a fast-paced interview. If the average time interval is greater than 3 seconds and it is mainly dominated by one person speaking, it is highly likely to be an audiobook or course. If the time interval decreases from large to small, it indicates that the discussion is becoming increasingly intense.

[0068] Example of adding a new field to structured text: Interval statistics: Average interval: 0.45 seconds Short interval percentage: 76% Interval mode: Rapid exchange The structured text of this application can provide interactive structural features of dialogues, providing richer judgment criteria for large-scale language models, greatly improving the accuracy of audio file classification, and also improving the accuracy of subsequent role analysis.

[0069] S24, Constructing a large language model prompt (Prompt) The purpose of this step is to build a high-quality prompt containing complete task instructions and contextual information to guide a large language model to complete the final classification task.

[0070] The inputs for this step are: a predefined subject classification system and the structured text output from step S23.

[0071] The processing steps for this step are as follows: A Prompt is dynamically constructed according to a preset template. This template may include at least one of the following components: Role setting: Assigning an expert role to the large language model, such as "You are a professional audio content analyst." Task description: Clearly stating the task to be performed, namely, determining the subject matter category of the audio, and requiring the output of results with confidence scores in a specific format. Contextual information: Embedding the structured text generated in step S23 into the Prompt as the primary basis for model analysis. Classification system definition: Listing all candidate subject matter categories and providing a concise feature description for each category, such as: "News: Timely, objective narration, multiple citations of data and authoritative sources." Output format requirements: For example, specifying the output as JSON format, including the subject matter category and its corresponding confidence score (between 0 and 1).

[0072] The output of this step is a complete text string (i.e., a Prompt) that can be directly sent to the large language model API (Application Programming Interface).

[0073] S25, obtain the audio file subject classification results output by the large language model.

[0074] This step leverages the power of large-scale language models to perform the final intelligent classification of the audio content. Prompts are sent to the large-scale language model (such as OpenAI's GPT series, Google's Gemini series, and Anthropic's Claude series) via API requests. Upon receiving the prompts, the large-scale language model comprehensively understands the structured text content, subject classification system definitions, and task requirements, generating and outputting the corresponding classification results. Large-scale language models can understand dialogue structures (such as monologues or interviews) and make more accurate judgments than traditional text classification. The audio file subject classification results output by the large-scale language model are received. These results include one or more subject categories and their corresponding confidence scores. For example, a subject classification result of {"News": 0.8, "Interview": 0.4} indicates that the primary subject category of the audio file is news, while also having the secondary attribute of an interview. This "soft classification" result can effectively handle cross-subject content.

[0075] S26, Fully Automated Classification and Application This step aims to extend the single-process flow described above to the platform's full audio data, enabling large-scale automated classification.

[0076] The input for this step is a massive amount of audio files in an audio database.

[0077] The process for this step is as follows: Build a batch processing or streaming processing task and sequentially execute the complete process from S21 to S25 for each audio file in the database.

[0078] The output of this step is a persistent mapping database that stores a unique identifier (Audio ID) for each audio file and its corresponding genre classification result (a JSON object output in step S25). This mapping database can be directly accessed by downstream business systems (such as personalized recommendation systems, search engines, and content moderation systems) to achieve more refined operations and user services.

[0079] This application solves the following technical problems: (1) Problem of limited accuracy due to ignoring key classification features: To address the shortcomings of existing technologies that rely solely on plain text content for classification, this application proposes a novel "segmentation-transcription-structuring" processing link. This link first extracts structured information such as the number of speakers and dialogue interaction patterns from the audio through speaker segmentation and clustering technology. Then, it combines the ASR (Automatic Speech Recognition) transcription content to generate structured text containing multimodal features. Finally, this structured text is input into a large-scale language model for comprehensive analysis and classification, thereby significantly improving the accuracy of classification. (2) The problems of high model training cost and difficulty in classifying the system: In view of the shortcomings of existing technologies that rely on large-scale labeled data and complex model training processes, this application utilizes the zero-shot or few-shot learning capabilities of large language models and guides the model to complete the classification task by designing specific prompts. When it is necessary to add or adjust the subject category, only the description in the prompt needs to be updated, without the need to re-label the data and retrain the model, which greatly reduces the maintenance cost and improves the iteration efficiency and flexibility of the classification system.

[0080] (3) The problem of coarse classification granularity and inability to effectively handle cross-topic content: In view of the shortcomings of existing technologies in handling cross-topic content, this application guides a large language model to output multi-label classification results with confidence or weight, thereby achieving "soft classification" of audio content. It can accurately identify and quantify multiple subject attributes coexisting in the work (e.g., "news" confidence 0.7, "interview" confidence 0.3), solve the problem of coarse classification granularity of existing technologies, and provide richer information for downstream refined recommendations.

[0081] (4) Low level of automation and bottleneck in scalability of the processing flow: In view of the inefficiency of existing manual classification schemes, this application constructs an end-to-end fully automated processing flow by organically combining technologies such as speaker segmentation, automatic speech recognition and large-scale language models. This flow does not require any manual intervention and can perform efficient and consistent automated classification of massive audio data, thus solving the problems of low level of automation and bottleneck in scalability of existing technologies.

[0082] Example 3 like Figure 5 As shown, the audio subject classification method of this application includes: Step 31: Receive the audio file to be processed and call the speaker segmentation and clustering module to process it to obtain the structured speaker timeline data contained therein. For example, a 60-second interview audio file is input into the system. For example, a pre-trained model based on the pyannote.audio open-source library is used to process the audio. After processing, the system outputs a list containing all speech segments and their speaker identities and timestamps, such as [('SPEAKER_00', 0.5, 15.2), ('SPEAKER_01', 15.8, 35.5), ('SPEAKER_00', 36.1, 58.9)].

[0083] This step aims to extract structured information from the audio signal, providing a foundation for subsequent steps. The speaker segmentation and clustering algorithm in this application is not limited to open-source solutions; it can also be based on commercial cloud service APIs, etc. The speaker segmentation and clustering algorithm can be any speaker segmentation and clustering technique capable of achieving the same purpose.

[0084] Step 32: Based on the speaker timeline data obtained in Step 31, extract individual speech segments from the original audio file and call the Automatic Speech Recognition (ASR) engine to convert these speech segments into text data. Continuing the previous example, the system segments 0.5-15.2 seconds, 15.8-35.5 seconds, and 36.1-58.9 seconds of the audio. Subsequently, OpenAI's Whisper model can be used to transcribe these three segments, obtaining the corresponding text content. Automatic speech recognition engines could also include iFlytek's Hearing or Google's Speech-to-Text.

[0085] Step 33: The speaker identification information (identifier) ​​obtained in Step 31 and the text data obtained in Step 32 are merged and formatted in chronological order to generate a unified structured text containing multimodal information. Continuing the previous example, the system integrates the speaker ID with the transcribed text, for example, to generate a plain text dialogue record: SPEAKER_00: "Today we are honored to have Professor Li with us..." SPEAKER_01: "Hello, everyone. It's great to be here..." SPEAKER_00: "Professor, what are your thoughts on the recently released AI model?..." To preserve more accurate time information, the speaker ID is integrated with the transcribed text, for example, by generating a JSON array format: [ {"speaker": "SPEAKER_00", "start": 0.5, "end": 15.2, "text": "Today we are honored to have Professor Li with us..."}, {"speaker": "SPEAKER_01", "start": 15.8, "end": 35.5, "text": "Hello everyone, I'm very happy to be here..."}, {"speaker": "SPEAKER_00", "start": 36.1, "end": 58.9, "text": "Professor, what are your thoughts on the recently released AI model?..."}] This step primarily involves combining speaker structural information with speech content information to generate input (structured text) that can be understood by large language models. Structured text can be plain text, JSON, XML, or any other format.

[0086] In some embodiments, the audio subject classification method of this application further includes: calculating the sum of the speaker's speech segments based on the speaker's identifier in the timeline data and the start and end times of the speaker's speech segments to obtain the speaker's total speaking time; calculating the ratio of the total speaking time to the number of times the speaker speaks to obtain the speaker's average speaking time; and calculating the ratio of the speaker's total speaking time to the total audio duration to obtain the speaker's speaking time percentage, where the total audio duration is the total duration of the speech segments. At least one of the speaker's total speaking time, average speaking time, and speaking time percentage is added to the structured text. The structured text includes at least one of the speaker's total speaking time, average speaking time, and speaking time percentage. The longest or shortest speech for each speaker can also be calculated.

[0087] The value of duration features for audio file classification is shown below:

[0088] If a single speaker accounts for more than 95% of the total speaking time and their average speaking time is greater than 30 seconds, it is likely an audiobook. If a single speaker accounts for more than 80% of the total speaking time and their average speaking time is 5-30 seconds, it is likely a course or lecture. If two speakers each account for 40-60% of the total speaking time and their average speaking time is similar, it is likely a debate. If one speaker accounts for 20-30% of the total speaking time and their average speaking time is short, while the other speaker accounts for 70-80%, it is likely an interview (host and guest).

[0089] Example of adding a new field to structured text: Speaking duration statistics: SPEAKER_00: Total duration 85.2 seconds, accounting for 29.9%, with an average of 7.1 seconds per segment. SPEAKER_01: Total duration 200.1 seconds, accounting for 70.1%, with an average of 20.0 seconds per segment. Duration distribution pattern: Type: Interview Lead speaker: SPEAKER_01 Balance: 0.43 (0 for complete imbalance, 1 for complete balance) In some embodiments, the audio subject classification method of this application further includes: obtaining the order of speaker alternation based on the speaker's identifier and the start and end times of the speech segments in the timeline data; calculating the ratio of the number of consecutive speeches by a speaker to the total number of speeches to obtain the speaker's continuity rate. At least one of the speaker alternation order and the speaker continuity rate is added to the structured text. The structured text includes at least one of the speaker alternation order and the speaker continuity rate.

[0090] Analyzing the turn-taking patterns between speakers, or the alternation patterns between speakers, includes: turn sequence, i.e. the order in which speakers alternate, such as A→B→A→B; self-continuation rate, i.e. the proportion of speakers who speak twice consecutively (i.e., the proportion of speakers who continue speaking without changing speakers); and alternation regularity, i.e., whether the alternation is regular or random.

[0091] The value of round-based models for audio file classification is as follows:

[0092] If the self-continuation rate is greater than 80% and there is only one speaker, it is very likely to be an audiobook or a lecture. If the self-continuation rate is less than 20% and there are two speakers, alternating regularly, it is very likely to be a debate. If the self-continuation rate is between 30% and 50% and there are multiple speakers, it is very likely to be a roundtable discussion.

[0093] Example of adding a new field to structured text: Round pattern analysis: Total number of rounds: 22 Round sequence: A→B→A→B→A→B→A→B... Self-continuation rate: 13.6% (the percentage of speakers who speak consecutively) Alternation regularity: 0.85 (0 for completely random, 1 for completely regular) Round pattern: Regular alternation (regular_alternation) In some embodiments, this application further includes: extracting non-linguistic content features from the audio file to be classified, the non-linguistic content features including at least one of voice emotion, speech rate, pitch, energy, and prosodic variation; and adding the non-linguistic content features to structured text. The structured text includes non-linguistic content features. Non-linguistic content features include at least one of the following features: voice emotion, i.e., the speaker's emotional state (calm, excited, sad, angry, happy, etc.); speech rate, i.e., the number of words or syllables spoken per minute; pitch / fundamental frequency, i.e., the variation in the high and low pitches of the voice; energy / loudness, i.e., the variation in the intensity of the voice; and prosodic variation, i.e., the pattern of change of the above features over time. These features can be analyzed using specialized voice emotion recognition models (such as emotion2vec, Wav2Vec2, etc.) and acoustic feature extraction tools.

[0094] The value of non-linguistic content features for audio file classification is as follows:

[0095] If the emotional variance is large and the speaker is a single person, it is likely an audiobook with a performative reading style; if the emotional variance is small, the speech rate is stable and the speaker is a single person, it is likely a news broadcast; if the emotional "excitement" accounts for a high proportion, there is a lot of speech overlap and the speaker is a multi-person event, it is likely a debate.

[0096] Here is an example of adding a new field to a structured text document: Rhythmic features: overall: Dominant sentiment: Neutral Emotional distribution: Neutral 65%, Happy 15%, Excited 12%, Sad 5%, Angry 3% Emotional change level: 0.23 (higher values ​​indicate richer emotions) Average speaking speed: 145 words / minute Prosody Pattern: Normal According to the speaker: SPEAKER_00: Dominant sentiment - neutral, speaking speed 160 words / minute SPEAKER_01: Dominant emotion - excitement, speaking speed 130 words / minute.

[0097] In some embodiments, this application further includes: detecting whether there is background music or ambient sound in the audio file to be classified; identifying the environment type of the audio file to be classified; and adding at least one of the detection results and the identified environment type to structured text. The structured text includes the detection results and at least one of the identified environment types. Identifying sound events in the audio other than human voices includes, for example, music detection, ambient sound detection, and environment type identification. Music detection detects the presence of background music (opening theme, ending theme, background music, etc.), ambient sound detection detects the presence of applause, laughter, cheers, noise, etc., and environment type identification includes identifying recording studio, indoor, outdoor, and live recording environments. These features can be analyzed using audio event detection models (such as YAMNet, PANNs, etc.).

[0098] The value of background noise features for audio file classification is as follows:

[0099] If there is background music and laughter, it may be a variety show or talk show; if there is applause and it is mainly from a single person, it may be a speech or a TED Talk (Technology, Entertainment, Design); if there is no background music, the speaker is a single person, and the emotions are rich, it may be an audiobook; if there is no background music, the speaker is a group of people, and the emotions are stable, it may be an interview or a course.

[0100] Here is an example of adding a new field to a structured text document: Background sound analysis: Is there background music? Yes Background music percentage: 25% Music appearance periods: 0-5.5 seconds (opening credits), 295-300.5 seconds (closing credits) Sound events: Applause: Appeared 3 times, total duration 8.5 seconds Laughter: Appears 12 times, total duration 15.3 seconds Environment type: Recording studio Background mode: Entertainment show.

[0101] The structured text of this application can provide interactive structural features of dialogues, providing richer judgment criteria for large language models and greatly improving the accuracy of audio file classification.

[0102] Step 34: Based on the preset template, combine the structured text generated in Step 33 with the predefined topic classification system to construct a complete prompt to guide the large language model in classification. This topic classification system can be dynamically adjusted according to business needs, and may include, but is not limited to, the following: News: Timely and objectively narrating events.

[0103] Entertainment: Celebrity interviews, variety shows, film and television reviews, etc.

[0104] Education: Online courses, professional lectures, knowledge explanations, etc.

[0105] Novels: audiobooks, storytelling, etc.

[0106] The above classification system is integrated with the structured text generated in step 33 to form the final prompt.

[0107] Unlike traditional methods that require retraining the model, the classification system proposed in this application is highly scalable. When a new subject (such as "sports") needs to be added, or the definition of an existing subject needs to be modified, the operator only needs to add or modify a single line of text description in the Prompt template in natural language. Without any code changes or model retraining, the new classification system can be implemented immediately, achieving high flexibility.

[0108] Step 35: Obtain the audio file's subject matter classification results. Send the prompt constructed in Step 34 to a large language model (e.g., via API call to GPT-4 or Claude 3 Opus models) and receive the classification results returned by the model. The large language model will generate a JSON-formatted output based on the input dialogue content (e.g., identifying the roles of "host" and "professor," and the topic of "AI model") and a predefined classification system. This output contains one or more subject matter categories and their corresponding confidence scores. For example, for the interview audio above, the model might return: {"Education": 0.9, "Technology": 0.8, "Interview": 0.6}.

[0109] In this application, a large language model is used as an inference engine to process the structured text generated in step 33, which incorporates multimodal information, and combined with the carefully designed Prompt template containing subject matter definitions in step 34 to achieve accurate classification of the subject matter of the original audio file.

[0110] The structured text in this application provides richer criteria for judgment in large language models. An example of the classification logic is as follows: Debate identification: The interval pattern is rapid exchange (interval less than 0.5 seconds), the overlap intensity is high (overlap ratio greater than 15%), the duration distribution is balanced between the two sides (each accounting for 40-60%), and the round pattern is regular alternation, which indicates a debate.

[0111] Audiobook recognition: If the number of speakers is 1, the duration is greater than 95%, the emotional change level is high (performative reading), and the background mode is clean (no background noise), it is determined to be an audiobook.

[0112] Variety show identification: If there is background music, frequent laughter, and multiple people talking, it is identified as a variety show or talk show.

[0113] Step 36: Associate the classification result returned in Step 35 with the unique identifier of the audio file and store it in a database for downstream applications (such as recommendation systems, content search, etc.) to query and use.

[0114] The audio subject classification method in this application has the following effects: (1) Full-process automation is achieved, significantly reducing classification costs and time: The end-to-end processing pipeline constructed in this application can complete the subject classification of massive audio data without any manual intervention. By adopting a zero-shot classification method based on a large-scale language model, the reliance on large-scale manually labeled training data is completely eliminated. This fundamentally solves the problems of high model training costs and long cycles in existing machine learning solutions, and also avoids the scalability bottleneck of manual classification solutions, greatly reducing the cost of technical implementation and operation and maintenance.

[0115] (2) Improved classification accuracy: This application innovatively introduces a speaker segmentation and clustering step into the processing flow and integrates the results with the ASR transcription content into structured text. This enables the large language model, which serves as the classification engine, to not only understand "what was said" but also "who is speaking" and "what the structure of the dialogue is." This multimodal information input provides the model with richer judgment criteria than plain text (e.g., it can distinguish between "single-person lecture" and "multi-person interview"), thereby breaking through the accuracy limit of existing technologies due to the lack of key features and significantly improving the accuracy of classification.

[0116] (3) Refined processing of cross-topics: This application utilizes a large-scale language model to output multi-label results with confidence scores, achieving "soft classification" of audio topics. This application can flexibly and accurately characterize the multiple topic attributes of coexisting audio content, solving the technical pain point that traditional single-label or complex multi-label classification models cannot effectively handle cross-topics. This refined classification result can provide higher quality data support for downstream applications such as personalized recommendation and content profiling.

[0117] (4) Greatly improves the flexibility and scalability of the classification system: In the technical solution of this application, the basis for subject classification is defined in the prompt template. When it is necessary to add, delete or modify subject categories, the operators only need to modify the text description in the prompt, and it will take effect immediately. Reclassifying the entire dataset only requires a one-time batch inference task. This "modify and use immediately" feature completely avoids the rigid process of relabeling data and retraining models in traditional solutions, making the iteration of the classification system extremely agile and low-cost, and able to quickly adapt to the needs of business development.

[0118] like Figure 6 As shown, the audio subject classification system of this application includes: an extraction unit 401, used to extract speaker timeline data from the audio file to be classified, the timeline data including the speaker's identifier and the start and end times of the speaker's speech segments; a conversion unit 402, used to convert the audio file to be classified into transcribed text based on the timeline data; a generation unit 403, used to generate structured text based on the timeline data and the transcribed text, the structured text including the transcribed text of the speech segments and the speaker corresponding to the speech segments, the speech segments being arranged in chronological order; and a classification unit 404, used to construct prompts based on the structured text to obtain the audio file subject classification result output by a large language model.

[0119] In this application, the embodiments of the audio subject classification system are basically similar to the embodiments of the audio subject classification method. For relevant details, please refer to the description of the embodiments of the audio subject classification method.

[0120] This application also provides an electronic device, including a memory, a processor, and a software program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the above methods.

[0121] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described audio subject classification method. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0122] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, indirect coupling or communication connection of devices or units, and may be electrical, mechanical, or other forms.

[0123] In the various embodiments of the present invention, all functional units can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0124] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An audio subject classification method, characterized in that, include: Extract speaker timeline data from the audio file to be classified. The timeline data includes the speaker's identifier and the start and end times of the speaker's speech segments. Based on the timeline data, the audio files to be classified are converted into transcribed text; Structured text is generated based on the timeline data and the transcribed text. The structured text includes the transcribed text of the speech segments and the speakers corresponding to the speech segments. The speech segments are arranged in chronological order. Based on the structured text construction prompts, the audio file subject classification results output by the large language model are obtained.

2. The audio subject classification method according to claim 1, characterized in that, The step of converting the audio file to be classified into transcribed text based on the timeline data includes: Based on the speaker's identifier and the start and end times of the speech segments in the timeline data, separate speech segments are cut out from the audio files to be classified. The spoken segment is processed by an automatic speech recognition engine to obtain the transcribed text of the spoken segment.

3. The audio subject classification method according to claim 1 or 2, characterized in that, The step of generating structured text based on the timeline data and the transcribed text includes: The speaker and the start and end times of the speech segment are obtained from the timeline data. The speaker corresponding to the speech segment is combined with the transcribed text corresponding to the speech segment to obtain the speech segment unit; The playback order of the speech segments is determined based on the start and end times of the speech segments; The speech segments are assembled in chronological order to generate structured text.

4. The audio subject classification method according to claim 3, characterized in that, The method further includes: Based on the start and end times of the speech segments, extract the identifier of the speaker who started speaking earliest from the timeline data; Based on the timeline data, count the number of times the speaker speaks; Add at least one of the following to the structured text: the number of times a speaker has spoken and the identifier of the speaker who first started speaking.

5. The audio subject classification method according to claim 3, characterized in that, The method further includes: Detect overlapping segments spoken by multiple speakers simultaneously; Obtain the speaker's identifier and the speaker's speech content from the overlapping segments; The number of overlapping segments, the total duration of overlapping segments, and the overlap ratio are determined, wherein the overlap ratio is the ratio of the total duration of overlapping segments to the total duration of speech, and the total duration of speech is the total duration of the speech segment; Add at least one of the following to the structured text: the number of overlapping segments, the total duration of the overlapping segments, the overlap ratio, the speaker's identifier in the overlapping segments, and the speaker's speech content.

6. The audio subject classification method according to claim 3, characterized in that, The method further includes: Determine the time interval between two adjacent speech segments; Based on multiple time intervals, at least one of the following is determined: average interval duration, time interval distribution, and short interval proportion of the audio file to be classified, wherein the short interval refers to an interval with a duration less than a threshold. Add at least one of the average interval duration, the time interval distribution, and the proportion of short intervals to the structured text.

7. The audio subject classification method according to claim 3, characterized in that, The method further includes: Based on the speaker's identifier and the start and end times of the speaker's speech segments in the timeline data, determine the speaker's total speaking time and the average speaking time of each speech. Calculate the ratio of the speaker's total speaking time to the total speech duration to obtain the speaker's speaking time percentage, where the total speech duration is the total duration of the speaking segment; Add at least one of the speaker's total speaking time, the average speaking time, and the speaking time percentage to the structured text.

8. The audio subject classification method according to claim 3, characterized in that, The method further includes: Based on the speaker identifiers and the start and end times of the speech segments in the timeline data, the order in which the speakers alternate speaking is obtained; The speaker's continuity rate is obtained by calculating the ratio of the number of consecutive times a speaker speaks to the total number of times they speak. Add at least one of the speaker alternation order and speaker continuity rate to the structured text.

9. The audio subject classification method according to claim 3, characterized in that, The method further includes: Extract non-verbal content features from the audio files to be classified, wherein the non-verbal content features include at least one of voice emotion, speech rate, pitch, energy, and prosodic variation; The non-linguistic content features are added to the structured text.

10. The audio subject classification method according to claim 3, characterized in that, The method further includes: Detect whether the audio file to be classified contains background music or ambient sound; Identify the environment type of the audio file to be classified; The results of the detection and at least one of the identified environment types are added to the structured text.

11. An audio subject classification system, characterized in that, include: The extraction unit is used to extract the speaker's timeline data from the audio file to be classified. The timeline data includes the speaker's identifier and the start and end times of the speaker's speech segments. A conversion unit is used to convert the audio file to be classified into transcribed text based on the timeline data; The generation unit is used to generate structured text based on the timeline data and the transcribed text. The structured text includes the transcribed text of the speech segments and the speakers corresponding to the speech segments, and the speech segments are arranged in chronological order. The classification unit is used to construct prompts based on the structured text to obtain the audio file subject classification results output by the large language model.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-10.

13. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1-10.

Images