Multi-language audio quality inspection method and system based on voice processing and large language model

Abstract

The invention discloses a multi-language audio quality inspection method and system based on voice processing and a large language model, and a set of efficient quality inspection system is constructed through collaborative design of four stages of voice preprocessing, text conversion and optimization, violation detection and result output. The optimized voice processing module significantly reduces noise interference and recognition errors; an LLM calling optimization framework based on Prompt improves the accuracy of a detection task through rule design and a context reasoning strategy; and the standardized output and post-processing module ensures the reliability and practicability of the quality inspection result. The overall process design realizes efficient illegal behavior detection in a complex multi-language scene, the comprehensive performance of the system is greatly improved, the high efficiency, adaptability and accuracy of a multi-language customer service quality inspection task are realized, the actual requirements of a complex business scene can be met, and the method has remarkable technical advantages and wide application prospects.

Description

Technical Field

[0001] The present invention relates to the technical field of audio quality inspection, and in particular to a multi-language audio quality inspection method and system based on speech processing and a large language model. Background Art

[0002] In the field of customer service quality inspection, especially for the automation task of collaborative analysis of speech and text, a variety of technologies and methods have been widely studied and applied.

[0003] Customer service conversation data usually has the following characteristics: the text content is relatively lengthy and has strong context relevance, which places high demands on context understanding ability; the language expression forms are complex and diverse, often involving multi-language switching or mixed language use; the text generated by speech recognition may contain certain errors, such as typos, incomplete grammar, or missing content. In addition, customer service conversations are usually accompanied by complex scenes such as background noise and multi-person mixed voices. These characteristics significantly increase the technical difficulty of customer service quality inspection tasks, and at the same time place higher requirements on the applicability and accuracy of existing quality inspection technologies.

[0004] Existing customer service quality inspection technologies usually rely on the following methods:

[0005] 1. Rule matching and regular expressions: Rule matching technology detects violations in conversations through predefined keywords or fixed sentences, and is widely used in structured and templated customer service scenarios. This method performs well in scenarios where specific rules are clear, but lacks adaptability to the flexibility and diversity of natural language expressions. For example, customer service conversations often contain sentence deformations, synonym substitutions, or language switching, making it difficult for rule matching to effectively cover these changes. In addition, the inevitable transcription errors in speech recognition texts (such as typos and missing words) further increase the complexity and maintenance difficulty of rule matching. In multilingual scenarios, the cost of updating and maintaining rules rises rapidly with the increase in the number of languages, resulting in limited practical applications in complex voice scenarios.

[0006] 2. Vector similarity matching: With the development of natural language processing technology, semantic matching technology based on vector representation has gradually been applied to customer service quality inspection tasks. These methods rely on pre-trained language models (such as Word2Vec, BERT) to generate semantic vectors of text, and identify violations in conversations by calculating similarities. Compared with rule matching, vector similarity matching is more robust in sentence deformation and synonymous expression processing. However, this technology has difficulty capturing deep semantic associations in complex contexts in long text scenarios. In addition, in multilingual conversation scenarios, due to the large differences in the distribution of semantic features in different languages, vector similarity matching often cannot take into account the characteristics of multiple languages ​​at the same time, resulting in a significant decrease in detection performance.

[0007] 3. Traditional machine learning models: Traditional machine learning models (such as support vector machines and decision trees) were once the mainstream methods in the field of customer service quality inspection. These models classified texts by manually designing features and were suitable for early structured dialogue detection tasks. However, traditional machine learning methods have significant limitations when processing natural language. These models are highly dependent on manual feature design and are difficult to adapt to flexible and changeable language expressions and context dependencies. In addition, in the face of multilingual dialogue scenarios, these models usually need to design features separately for each language, resulting in poor scalability and difficulty in meeting the needs of complex multilingual scenarios.

[0008] In the field of customer service quality inspection, rule matching, vector similarity matching and traditional machine learning models have their own characteristics, but they have obvious limitations when dealing with complex, multilingual customer service dialogue scenarios. Those skilled in the art have been seeking more efficient text processing technologies and model optimization strategies to address the shortcomings of existing methods in terms of applicability, accuracy and scalability. Summary of the invention

[0009] In view of this, the present invention provides a multilingual audio quality inspection method and system based on speech processing and a large language model, focusing on violation detection and customer service quality assessment in complex multilingual conversations. The present invention uses voice activation detection (VAD), two-channel speech separation, speaker separation and speech-to-text (ASR) technology, combined with context-aware dialogue flow optimization, to achieve accurate analysis and efficient preprocessing of audio data in multilingual customer service scenarios.

[0010] The present invention innovatively solves the problems of hallucination and recognition bias that may be caused by large language models in customer service scenarios. By introducing the prompt strategy of business knowledge enhancement, hierarchical rule constraints, and step-by-step reasoning, the accuracy, coverage, and credibility of violation detection are effectively improved. At the same time, the present invention adopts dynamic quality scoring and speech recognition consistency analysis to ensure the high adaptability and reliability of the quality inspection process for multilingual dialogues. The present invention also fully considers practical problems such as multilingual mixing and audio noise in the customer service environment. Before model reasoning, the ASR output is formatted and converted and noise is filtered, so that the model can accurately locate violations in complex scenarios.

[0011] A multilingual audio quality inspection method based on speech processing and a large language model specifically comprises the following steps:

[0012] S1, extracting the conversation speech stream containing only the target speaker from the audio to be inspected;

[0013] S2, using multilingual ASR technology to convert the conversation speech stream into text data and optimize the text data based on word-level timing information and role annotation, and reconstruct the standard conversation text;

[0014] S3, the large language model identifies and extracts illegal dialogues in the standard dialogue text according to the set quality inspection rules, and generates quality inspection data;

[0015] S4, stores the quality inspection data in the database and simultaneously pushes it to the quality inspection business system for business personnel to review, analyze and provide feedback.

[0016] Preferably, in step S1, the specific steps of extracting the conversation voice stream containing only the target speaker from the audio to be inspected are:

[0017] The dual-channel separation technology is used to extract multiple target speaker speech streams with time stamps from the audio to be inspected.

[0018] The speech activation detection algorithm is used to remove non-human background noise from the speech streams of multiple target speakers and extract the effective speech streams.

[0019] Speaker separation technology is used to remove the speech of non-target speakers from the effective speech stream, and extract the dialogue speech stream containing only the target speaker.

[0020] Preferably, in step S2, the text data is optimized based on the Word-Level timing information and the role annotations, and the specific steps of reconstructing the standard dialogue text are:

[0021] Segment text data at word level;

[0022] The voice activation detection algorithm determines whether the time interval between two adjacent characters or words is less than the set VAD threshold based on the timestamp of each character or word. If so, it means that the two characters or words belong to the voice activity of the same speaker, and they are merged into a text speech paragraph, and the speaker role and the start and end time of the speech are marked in the speech paragraph; otherwise, the two characters belong to the voice activities of different speakers, and they are divided into different text speech paragraphs, and the corresponding speaker role and speech start and end time are marked in each paragraph;

[0023] All the obtained text speech paragraphs marked with speaker roles and speech start and end times are sorted in chronological order to reconstruct the standard dialogue text.

[0024] Preferably, in step S3, the large language model identifies and extracts the illegal dialogues in the standard dialogue text according to the set quality inspection rules, and the specific steps of generating quality inspection data are:

[0025] Inject business knowledge and quality inspection rules into large language models;

[0026] The large language model analyzes the standard dialogue text step by step based on the step-by-step reasoning method, and scores the dialogue quality according to the context semantics. If the dialogue quality score is greater than the set score, the dialogue quality is considered to be good, and a formatted quality inspection result text containing an empty violation list is output;

[0027] If the conversation quality score is less than the set score, the set quality inspection rules will be used to determine whether there is any violation. If there is a violation, the text content corresponding to the violation, the type of violation, and the time when the violation occurred will be extracted, and a formatted quality inspection result text containing a list of violations will be output.

[0028] Preferably, the quality inspection result text containing an empty violation list and the quality inspection result text containing a violation behavior list have the same text format, and both include the violation behavior type, the violation behavior occurrence time, the violation behavior description, and the dialogue quality score value.

[0029] Preferably, after outputting the quality inspection result text containing an empty violation list or containing a violation behavior list, the quality inspection result text needs to be converted into a standard language text.

[0030] Preferably, before converting the quality inspection result text into a standard language text, the text format needs to be verified to ensure that the text format meets the requirements.

[0031] Preferably, the quality inspection rules include static quality inspection rules and dynamic quality inspection rules, both of which are used to clarify the types of illegal behaviors, illegal detection instructions and the text output format of the quality inspection result text.

[0032] Static quality inspection rules are quality inspection rules that cover the requirements of general business scenarios;

[0033] Dynamic quality inspection rules are quality inspection rules applicable to specific business scenarios. Each rule in the dynamic quality inspection rules has a priority setting.

[0034] Preferably, the rules in the dynamic quality inspection rules can be added or deleted according to business needs.

[0035] Preferably, in step S4, the quality inspection data supports batch push transmission to the quality inspection business system.

[0036] A multi-language audio quality inspection system based on speech processing and a large language model, comprising a speech processing module, an ASR module, a large language model, an output post-processing module and a database;

[0037] The speech processing module is used to extract multiple target speaker speech streams with time stamps from the audio to be inspected by using dual-channel separation technology, and perform multi-layer filtering on the multiple target speaker speech streams to extract a dialogue speech stream containing only the target speaker;

[0038] The ASR module is used to convert the conversation speech stream containing only the target speaker into text data using multilingual ASR technology, and optimize the text data based on Word-Level timing information and role annotation to reconstruct a standard conversation text;

[0039] The large language model is used to identify and extract illegal dialogues in the standard dialogue text according to the set quality inspection rules to generate quality inspection data;

[0040] The output post-processing module is used to convert the quality inspection data output by the large language model into a standard language format, store the quality inspection data converted into the standard language in a database, and simultaneously push it to the quality inspection business system for business personnel to review, analyze and provide feedback.

[0041] Preferably, before converting the quality inspection data output by the large language model into a standard language format, the output post-processing module needs to verify the text format of the quality inspection data output by the large language model to ensure that the text format meets the requirements.

[0042] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0043] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0044] The beneficial effects of the present invention are:

[0045] 1. The present invention uses technologies such as voice activation detection (VAD), speaker separation and binaural separation to purify the signals of multi-speaker aliasing and noisy speech in customer service calls, thereby reducing errors in the speech processing stage, reducing the misjudgment of large language models due to inaccurate speech recognition, and improving the accuracy of quality inspection tasks. At the same time, it combines multiple technical processing in the text conversion and optimization stage to provide high-quality text input for subsequent violation detection.

[0046] VAD dynamically screens valid speech segments to reduce interference from background noise. Speaker separation technology uses a deep learning-based speech separation model to accurately distinguish the voice trajectories of customers and customer service staff, ensuring that the text data generated in the speech-to-text (ASR) stage has high-quality role annotations.

[0047] In the text conversion and optimization stage, the audio signal after speech processing is converted into text data through the ASR module. At the same time, the text is optimized and standardized in combination with Word-Level timing information and role annotation. Based on advanced multilingual ASR technology, the system can accurately identify the text content in multilingual and mixed language conversations. Through timestamp annotation, it ensures that each sentence in the text corresponds to the exact start and end time of the speech, separates the text content according to the dialogue role (such as customer service and customer), and reconstructs the dialogue text, which effectively solves the problem of missing alternating dialogue information caused by overlapping speech between customers and customer service, and provides high-quality, standardized input data for the large language model, significantly improving the overall accuracy and robustness of quality inspection tasks.

[0048] 2. The present invention uses a large language model to analyze the converted speech text. By optimizing the contextual reasoning capability and dynamic rule loading mechanism, it achieves efficient adaptation to multiple languages, long texts and complex dialogue scenarios, enhances the adaptability and robustness of the large language model to complex scenarios, and ensures the credibility and consistency of the quality inspection results.

[0049] The contextual reasoning optimization strategy combines business knowledge injection and instruction reinforcement to improve the large language model's (LLM) ability to deeply understand contextual semantics. For example, the Chain-of-Thought (CoT) method guides the model to analyze the conversation context step by step, accurately identifies violations and generates standardized output.

[0050] The dynamic rule loading mechanism can adjust the detection logic in real time according to business needs, ensuring that the system performs well in dealing with multi-language switching and complex conversation expressions. This strategy significantly reduces the hallucination phenomenon of large language models in multi-language and long text processing, thereby improving the credibility and consistency of model detection results.

[0051] 3. The present invention constructs a set of flexibly adjustable quality inspection logic framework. Through hierarchical rule design, static core instructions are combined with dynamic rule tables to achieve dynamic adaptation of quality inspection rules, thereby improving the flexibility of the system.

[0052] The dynamic rule table is designed in layers according to priority, so that the system can give priority to completing key violation detection tasks, while supporting real-time updates of dynamic rule content to adapt to dynamic changes in business needs and maintain a high degree of fit with business needs. For example, during peak customer service business volume, the system can quickly load new violation detection rules to ensure efficient adaptation to new scenarios. Through this mechanism, the present invention achieves the flexibility and accuracy of quality inspection logic, significantly improving the fit between the system and business needs.

[0053] 4. The large language model of the present invention outputs the detection results in a structured JSON format, including key fields such as the type of violation, the time when the violation occurred, the description of the violation, and the dialogue quality score. This formatting design not only facilitates the integration and data analysis of subsequent systems, but also improves the traceability of the detection results. For example, it is convenient to uniformly convert to English format in multilingual scenarios, ensure the consistency of the detection results in cross-language analysis, and significantly improve the analysis and integration efficiency in cross-language scenarios.

[0054] In addition, the integrity and standardization of the test results are automatically verified to ensure that the generated test results meet the preset standards, reducing data processing errors caused by format problems. It also further enhances the reliability and operational efficiency of the testing tasks and improves the traceability and operability of the quality inspection results.

[0055] 5. The present invention builds an efficient quality inspection system through the collaborative design of four stages: speech preprocessing, text conversion and optimization, violation detection and result output. The optimized speech processing module significantly reduces noise interference and recognition errors; the LLM call optimization framework based on Prompt improves the accuracy of the detection task through rule design and contextual reasoning strategy; the standardized output and post-processing module ensure the reliability and practicality of the quality inspection results. The overall process design realizes efficient violation detection in complex multilingual scenarios, greatly improves the overall performance of the system, realizes the efficiency, adaptability and accuracy of multilingual customer service quality inspection tasks, can meet the actual needs of complex business scenarios, and has significant technical advantages and broad application prospects.

[0056] 6. The present invention can not only significantly reduce the cost of manual quality inspection, but also has the characteristics of high efficiency, flexibility and scalability. It is suitable for automated quality inspection needs in logistics, customer service and other fields, and is of great significance to improving customer experience and reducing operating costs. At the same time, the system can flexibly adapt to changes in rules and business needs to ensure the stability and controllability of detection results. Compared with traditional solutions, the present invention not only demonstrates superior quality inspection results in complex scenarios, but also has extremely high scalability and business adaptation capabilities, providing an advanced solution for multilingual customer service quality inspection tasks that is both flexible and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is described below by the specific embodiments shown in the accompanying drawings. However, it should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0060] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0061] The present invention relates to the fields of artificial intelligence (AI), natural language processing (NLP) and speech processing, and in particular to the application of a large language model (LLM) based on speech preprocessing technology and a pre-trained language model in customer service quality inspection. Specifically, the present invention is oriented to a multilingual environment, and by combining speech preprocessing (such as speech activation detection and speaker separation) with large language model reasoning, an automated quality inspection system and method suitable for complex dialogue scenarios is constructed. By adopting multimodal collaborative analysis technology and combining speech and text information, the recognition accuracy of illegal behaviors in noisy environments and multilingual customer service dialogues is significantly improved, and the hallucination problem and the analysis deviation problem in multilingual scenarios are particularly solved. The system is suitable for industries that require efficient quality inspection, such as logistics, e-commerce and customer service outsourcing.

[0062] The present invention realizes purification in complex voice scenarios through binaural separation, voice activation detection (VAD), and speaker separation technology, and cooperates with dialogue stream format conversion to make ASR output more in line with the needs of quality inspection tasks, provide LLM with more accurate and appropriate text input, and reduce quality inspection misjudgment. That is, through binaural separation technology, according to the binaural characteristics in the channel recording, the voice signals of the customer and the customer service are extracted respectively to reduce aliasing interference; then the voice activation detection (VAD) algorithm is used to remove the background noise and silent segments in the voice signals of the customer and the customer service, and retain high-quality voice data; and then the speaker separation technology is used to further distinguish the various sound sources involved in the conversation, and in the multi-speaker scenario, the voices of other people other than the customer and the customer service are removed, effectively solving the common background noise, multi-speaker aliasing and other problems in the customer service scenario, providing accurate dialogue voice for downstream processing, thereby reducing the error rate of the subsequent speech-to-text (ASR) stage, reducing the large model misjudgment caused by inaccurate speech recognition, and providing high-quality input data for subsequent quality inspection.

[0063] Then, the ASR module converts the above audio processed conversational speech into high-quality text data and standardizes the conversation structure. That is, the speech signal is converted into text through automatic speech recognition (ASR), supporting multi-language recognition and optimizing the recognition quality of long conversations; and based on the word-level timing information, the transcribed text is reconstructed into a standardized conversation flow format by timestamp and role separation to solve the problem of missing alternating conversation information caused by overlapping speech between customers and customer service, and it also facilitates the contextual reasoning of large language models.

[0064] Then, the large language model is based on the LLM call optimization framework of Prompt, and through refined Prompt design and dynamic rule loading, guides LLM to detect violations of input text and generate high-quality output results. That is, through contextual reasoning optimization strategy, dynamic hierarchical rule table and structured output generation, the LLM call process is fully optimized; the contextual reasoning optimization strategy enhances LLM's semantic understanding ability of complex contexts through business knowledge injection, instruction reinforcement and step-by-step reasoning, effectively reducing hallucinations, thereby ensuring the accuracy and consistency of violation detection results; the dynamic hierarchical rule table adopts a priority design and flexibly loads detection content according to business needs, which not only ensures the adaptability of the system, but also meets the ever-changing scenario needs.

[0065] Prompt follows the following structure:

[0066] 1 Task:Detecting Violations in Customer Service Conversations

[0067] 2├──Additional Instructions:

[0068] 3├──Steps:

[0069] 4├──Violation Types:

[0070] 5├──Example 1:

[0071] 6├──Example 2:

[0072] 7└──Conversation Transcripts: 8

[0074] The design principles of Prompt are:

[0075] 1. Role setting: clarify the role of the large language model. In this embodiment, the large language model is set as a professional customer service quality inspection robot of J&TExpress.

[0076] 2. Background information: Provide necessary background information to help the model understand the scenario, such as the conversation being processed is transcribed in Tagalish and the output needs to be in English.

[0077] 3. Mission instructions:

[0078] Steps (CoT, Chain of Thought): Detailed definition of the execution steps of the task, including the complete process from dialogue scoring to violation analysis.

[0079] Violation Types Definition: Explicitly lists possible violation types and provides positive and negative examples for each violation to help the model understand the violation criteria.

[0080] 4. Output format requirements: The format of the model output is clearly specified in the step part of the task instructions, and all outputs are required to be in standardized JSON format.

[0081] 5. Supplementary Examples (Few-Shot Learning): Example 1 and Example 2 provide specific Few-Shot examples to help the model better understand how to process inputs and generate correct outputs.

[0082] The LLM call optimization framework based on Prompt can improve the semantic understanding ability of the model in long text and multilingual scenarios through the Chain-of-Thought (CoT) method, effectively reduce the risk of hallucination, and ensure the accuracy and consistency of violation detection. At the same time, the quality inspection results in a unified structured JSON format are output, including key fields such as violation type, specific description, and dialogue score, thereby ensuring the standardization and consistency of the output results, and ensuring the traceability and business adaptability of the violation detection results (quality inspection tasks).

[0083] Then, to adapt to multilingual scenarios, all outputs of the large language model are converted into standardized speech - English, ensuring the convenience of subsequent analysis and integration.

[0084] In order to better understand the technical solution of the present invention, the present invention is described in detail below with reference to the accompanying drawings.

[0085] The present invention provides a multilingual audio quality inspection method based on speech processing and a large language model, which specifically includes the following steps:

[0086] S1, extracting the conversation speech stream containing only the target speaker from the audio to be inspected.

[0087] Specifically, the steps include:

[0088] First, obtain the audio to be inspected, and use the dual-channel separation technology to extract multiple target speaker voice streams with timestamps from the audio to be inspected (the dual-channel separation technology uses the physical characteristics of the left and right channels in the audio to be inspected to extract the voice streams of all target speakers separately). For example, if the audio to be inspected is a logistics customer service call recording, the target speakers are customer service A and the customer. The dual-channel separation technology is used to extract the voice streams of customer service A and the customer from the logistics customer service call recording. The extracted voice streams carry timestamps, and the extracted voice streams may contain non-human voice noise and voice noise of non-target speakers;

[0089] Then, the voice activation detection algorithm (VAD) is used to remove non-human background noise (such as background music or environmental noise) from the speech streams of multiple target speakers to extract the effective speech streams;

[0090] The Voice Activation Detection (VAD) algorithm can distinguish the speech and non-speech parts in the audio signal through energy threshold, spectrum characteristics and context semantic analysis, separate the background noise from the effective speech, and thus filter out high-quality effective speech streams;

[0091] For example, suppose that the multiple target speaker speech streams extracted using the dual-channel separation technology are:

[0092] 【00:32-00:38】Agent: verify ko lang po is made deliver pa rin ito okayone moment lang po ma'am please see the line

[0093] 【00:35-00:35】Customer:okay sige

[0094] 【00:49-00:51】Agent: oh

[0095] 【00:56-00:56】Agent: oh

[0096] 【01:02-01:03】Agent: nu

[0097] 【01:04-01:05】Agent: yes

[0098] 【01:05-01:07】Agent: yes

[0099] 【01:08-01:14】Agent: anu po oh

[0100] [01:19-01:22]Agent: oo oo

[0101] Among them, the time between 00:49 and 1:22 is all background noise. In order to distinguish the background noise (such as silence and music) from the valid voice, the voice activation detection algorithm (VAD) is used to remove the voice between 00:49 and 1:22. The optimized valid voice stream is as follows:

[0102] 【00:32-00:38】Agent: verify ko lang po is made deliver pa rin ito okayone moment lang po ma'am please see the line

[0103] 【00:35-00:35】Customer:okay sige

[0104] 【00:37-04:24】[on-hold music]

[0105] 【04:26-04:59】Agent: uh um hello po miss thank you so much for face and waiting line on the line po now i fully to

[0106] 【04:32-04:32】Customer:oo

[0107] Then, speaker separation technology (such as a separation network based on a Conformer structure) is used to extract the speech trajectory of each speaker, eliminate the speech of non-target speakers in the effective speech stream, and extract a conversational speech stream containing only the target speaker (for example, if the audio to be inspected is a logistics customer service call recording, the target speakers are customer service A and the customer, and the effective speech stream processed by the voice activation detection algorithm (VAD) contains the speech of customer service A and the customer, as well as the speech of other people, such as customer service B. In this case, speaker separation technology is needed to remove the audio of customer service B).

[0108] For example, suppose that after being processed by the voice activation detection algorithm (VAD), the valid voice stream is:

[0109] [01:01-01:10] Customer: It would be best if it were within half an hour, because I called many times and the number I left was either invalid, no one answered, or the call was hung up.

[0110] 【01:07-01:09】Customer Service: Yes, we have already responded to your feedback.

[0111] [01:10-01:17] Customer Service: Well, we have already made it clear for you and urged you to call back within half an hour. Can you please wait patiently and keep the line open?

[0112] 【01:10-01:17】Customer: OK, OK.

[0113] 【01:18-01:20】Customer Service: Okay, how can I help you?

[0114] The speaker separation technology determines that "【01:07-01:09】Customer Service: Well, the feedback has been received." is the speech of other non-target speakers, so it will delete this segment of speech and obtain the following conversation voice stream containing only the target speaker: [01:01-01:10] Customer: It would be best if it is within half an hour, because I called many times and the number he left was either invalid, no one answered, or he hung up.

[0115] [01:10-01:17] Customer Service: Well, we have already made it clear for you and urged you to call back within half an hour. Can you please wait patiently and keep the line open?

[0116] 【01:10-01:17】Customer: OK, OK.

[0117] 【01:18-01:20】Customer Service: Okay, how can I help you?

[0118] S2 uses multilingual ASR technology to convert the conversation voice stream into text data and optimizes the text data based on word-level timing information and role annotation to reconstruct the standard conversation text.

[0119] Specifically, the steps include:

[0120] First, the multilingual ASR technology is used to convert the conversational speech stream into text data.

[0121] Then, the text data is segmented at the word level to ensure that the boundaries of each character and word can be accurately captured and processed;

[0122] Then, the voice activation detection algorithm determines whether the time interval between two adjacent characters or words is less than the set VAD threshold (for example, 0.3 seconds) based on the timestamp of each character or word. If so, it means that the two characters or words belong to the speech activity of the same speaker, and they are merged into a text speech paragraph, and the speaker role and speech start and end time are marked in the speech paragraph; otherwise, the two characters belong to the speech activities of different speakers, and they are divided into different text speech paragraphs, and the corresponding speaker role and speech start and end time are marked in each paragraph;

[0123] All the obtained text speech paragraphs marked with speaker roles and speech start and end times are sorted in chronological order to reconstruct the standard dialogue text.

[0124] The text format of the standard dialogue text includes the start and end time of the speech, the speaker's role, and the speech content.

[0125] In real conversation audio, the voices of customers and customer service staff often overlap, resulting in missing information in alternating conversations. This step can effectively solve this problem to ensure that when speech overlap occurs, the voice activation detection algorithm can correctly capture and transcribe each valid voice segment to avoid information loss or truncation.

[0126] For example, suppose that the conversation speech stream obtained in step S1 that only contains the target speaker is:

[0127] 【00:00-00:04】Agent:GNT Express Philippines this is Tony how can I help you good afternoon

[0128] 【00:05-00:26】Customer:hello good afternoon ma'am I would like to ahinquire about the parcel

[0129] 【00:06-00:36】Agent:iyas po I mhm I see I atleast that you have concerned regarding your pick up is at Hilongos leyte mhm to Liloan SouthernLeyte ma'am I see I atleast that you have concerned regarding your pick uprequest no

[0130] Multilingual ASR technology optimizes text data based on word-level timing information and role annotations, and reconstructs the following standard dialogue text:

[0131] 【00:00-00:04】Agent:GNT Express Philippines this is Tony how can I help you good afternoon

[0132] 【00:05-00:13】Customer:hello good afternoon ma'am I would like to ahinquire about the parcel

[0133] 【00:00-00:07】Agent:iyas po

[0134] 【00:05-00:13】Customer:and then I ah yeah I'm I'm calling here in Cebucity

[0135] 【00:14-00:14】Agent: mhm ano

[0136] 【00:18-00:23】Customer: and then the parcel that we wanted to be pickedup is at Hilongos leyte

[0137] 【00:18-00:19】Agent: mhm

[0138] 【00:24-00:26】Customer:to Liloan Southern Leyte ma'am

[0139] 【00:26-00:35】Agent: I see I atleast that you have concerned regarding your pick up request no

[0140] S3, the large language model identifies and extracts illegal dialogues in the standard dialogue text according to the set quality inspection rules, and generates quality inspection data.

[0141] Specifically, the steps include:

[0142] First, business knowledge and quality inspection rules are injected into the large language model;

[0143] The large language model analyzes the standard dialogue text step by step based on the step-by-step reasoning method, and scores the dialogue quality according to the context semantics. If the dialogue quality score is greater than the set score, the dialogue quality is considered to be good, and a formatted quality inspection result text containing an empty violation list is output;

[0144] If the conversation quality score is less than the set score, the set quality inspection rules are used to determine whether there is any violation. If there is a violation, the text content corresponding to the violation, the type of violation, and the time when the violation occurred are extracted, and a formatted quality inspection result text containing a list of violations is output. If there is no violation, a formatted quality inspection result text containing an empty violation list is output.

[0145] The preset quality inspection rules include static quality inspection rules and dynamic quality inspection rules, both of which are used to clarify the types of violations, violation detection instructions, and the text output format of the quality inspection result text. Among them, static quality inspection rules are quality inspection rules that cover the needs of general business scenarios, which provide basic support for dynamic quality inspection rules; dynamic quality inspection rules are quality inspection rules applicable to specific business scenarios. Each rule in the dynamic quality inspection rules is set with a priority, and supports real-time updates and dynamic adjustments to ensure that key detection tasks are completed first. For example, for violations with higher priorities, the system will prioritize loading related rules and allocate more resources for analysis. This mechanism significantly improves the adaptability of quality inspection tasks to changing business needs.

[0146] The rules in the dynamic quality inspection rules can be added or deleted according to business needs.

[0147] The large language model gradually detects and analyzes each dialogue text according to the above quality inspection rules, combines context information to accurately determine the time and specific description of the violation, extracts the core information of the violation, and outputs the quality inspection result text. Regardless of whether the output is a quality inspection result text containing an empty violation list or a quality inspection result text containing a violation list, the output text format is the same, including the violation type (violation_type), violation occurrence time (timestamp), violation description (violation_description), and dialogue quality score (overall_score). Violation types such as "Improper Opening / Closing Remarks" are accompanied by specific detection requirements and examples to ensure the stability and consistency of the detection task.

[0148] The quality inspection result text output by the large language model will be formatted into a unified JSON format to ensure the standardization and traceability of the inspection results and their high degree of alignment with business needs, facilitating subsequent system processing and data integration.

[0149] Preferably, after the large language model outputs the quality inspection result text containing an empty violation list or a violation list, the quality inspection result text needs to be converted into a standard language text. For example, in a multilingual scenario, the quality inspection result text can be uniformly converted into English to ensure the consistency of results across languages ​​and facilitate subsequent analysis.

[0150] Preferably, before converting the quality inspection result text output by the large language model into a standard language text, the text format needs to be automatically checked to ensure that the text format meets the requirements, that is, to ensure that the record of each violation in the quality inspection result text is complete, and whether it includes the type of violation, the time when the violation occurred, the description of the violation, and the conversation quality score.

[0151] S4, stores the quality inspection data in the database and simultaneously pushes it to the quality inspection business system for business personnel to review, analyze and provide feedback.

[0152] Preferably, the quality inspection data supports batch push transmission to the quality inspection business system.

[0153] The present invention realizes an automated, efficient and flexible multilingual customer service quality inspection system based on speech preprocessing and large language model (LLM), which is particularly suitable for violation detection and business quality inspection tasks in complex multilingual scenarios.

[0154] A multi-language audio quality inspection system based on speech processing and a large language model, comprising a speech processing module, an ASR module, a large language model, an output post-processing module and a database;

[0155] The speech processing module is used to extract multiple target speaker speech streams with time stamps from the audio to be inspected by using dual-channel separation technology, and perform multi-layer filtering on the multiple target speaker speech streams to extract a dialogue speech stream containing only the target speaker;

[0156] The ASR module is used to convert the conversation speech stream containing only the target speaker into text data using multilingual ASR technology, and optimize the text data based on Word-Level timing information and role annotation to reconstruct a standard conversation text;

[0157] The large language model is used to identify and extract illegal dialogues in the standard dialogue text according to the set quality inspection rules to generate quality inspection data;

[0158] The output post-processing module is used to convert the quality inspection data output by the large language model into a standard language format, store the quality inspection data converted into the standard language in a database, and simultaneously push it to the quality inspection business system for business personnel to review, analyze and provide feedback.

[0159] Preferably, before converting the quality inspection data output by the large language model into a standard language format, the output post-processing module needs to verify the text format of the quality inspection data output by the large language model to ensure that the text format meets the requirements.

[0160] The present invention also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method when executing the computer program.

[0161] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device to perform all or part of the steps of the method described in the embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0162] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0163] It should be clear that the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

Claims

1. A multilingual audio quality inspection method based on speech processing and a large language model, characterized in that: The specific steps include: S1, extracting the conversation speech stream containing only the target speaker from the audio to be inspected; S2, using multilingual ASR technology to convert the conversation speech stream into text data and optimize the text data based on word-level timing information and role annotation, and reconstruct the standard conversation text; S3, the large language model identifies and extracts illegal dialogues in the standard dialogue text according to the set quality inspection rules, and generates quality inspection data; S4, stores the quality inspection data in the database and simultaneously pushes it to the quality inspection business system for business personnel to review, analyze and provide feedback.

2. The multilingual audio quality inspection method based on speech processing and large language model according to claim 1 is characterized in that: In step S1, the specific steps of extracting the conversation speech stream containing only the target speaker from the audio to be inspected are: The dual-channel separation technology is used to extract multiple target speaker speech streams with time stamps from the audio to be inspected. The speech activation detection algorithm is used to remove non-human background noise from the speech streams of multiple target speakers and extract the effective speech streams. Speaker separation technology is used to remove the speech of non-target speakers from the effective speech stream, and extract the dialogue speech stream containing only the target speaker.

3. The multilingual audio quality inspection method based on speech processing and large language model according to claim 1 is characterized in that: In step S2, the text data is optimized based on the Word-Level timing information and role annotations, and the specific steps of reconstructing the standard dialogue text are as follows: Segment text data into word level; The voice activation detection algorithm determines whether the time interval between two adjacent characters or words is less than the set VAD threshold based on the timestamp of each character or word. If so, it means that the two characters or words belong to the voice activity of the same speaker, and they are merged into a text speech paragraph, and the speaker role and the start and end time of the speech are marked in the speech paragraph; otherwise, the two characters belong to the voice activities of different speakers, and they are divided into different text speech paragraphs, and the corresponding speaker role and speech start and end time are marked in each paragraph; All the obtained text speech paragraphs marked with speaker roles and speech start and end times are sorted in chronological order to reconstruct the standard dialogue text.

4. The multilingual audio quality inspection method based on speech processing and large language model according to claim 1 is characterized in that: In step S3, the large language model identifies and extracts illegal dialogues in the standard dialogue text according to the set quality inspection rules. The specific steps of generating quality inspection data are as follows: Inject business knowledge and quality inspection rules into large language models; The large language model analyzes the standard dialogue text step by step based on the step-by-step reasoning method, and scores the dialogue quality according to the context semantics. If the dialogue quality score is greater than the set score, the dialogue quality is considered to be good, and a formatted quality inspection result text containing an empty violation list is output; If the conversation quality score is less than the set score, the set quality inspection rules will be used to determine whether there is any violation. If there is a violation, the text content corresponding to the violation, the type of violation, and the time when the violation occurred will be extracted, and a formatted quality inspection result text containing a list of violations will be output.

5. The multilingual audio quality inspection method based on speech processing and large language model according to claim 4 is characterized in that: The quality inspection result text containing an empty violation list and the quality inspection result text containing a violation list have the same text format, both including the violation type, the time when the violation occurred, the violation description, and the dialogue quality score.

6. The multilingual audio quality inspection method based on speech processing and large language model according to claim 4 is characterized in that: After outputting the quality inspection result text containing an empty violation list or a violation list, you also need to convert the quality inspection result text into a standard language text.

7. The multilingual audio quality inspection method based on speech processing and large language model according to claim 6 is characterized in that: Before converting the quality inspection result text into a standard language text, the text format needs to be verified to ensure that the text format meets the requirements.

8. The multilingual audio quality inspection method based on speech processing and large language model according to claim 1 or 4, characterized in that: The quality inspection rules include static quality inspection rules and dynamic quality inspection rules, both of which are used to clarify the types of illegal behaviors, illegal detection instructions and the text output format of the quality inspection result text. Static quality inspection rules are quality inspection rules that cover the requirements of general business scenarios; Dynamic quality inspection rules are quality inspection rules applicable to specific business scenarios. Each rule in the dynamic quality inspection rules has a priority setting.

9. The multilingual audio quality inspection method based on speech processing and large language model according to claim 8, characterized in that: The rules in the dynamic quality inspection rules can be added or deleted according to business needs.

10. The multilingual audio quality inspection method based on speech processing and large language model according to claim 1, characterized in that: In step S4, the quality inspection data supports batch push transmission to the quality inspection business system.

11. A multi-language audio quality inspection system based on speech processing and large language model, characterized in that: Includes speech processing module, ASR module, large language model, output post-processing module and database; The speech processing module is used to extract multiple target speaker speech streams with time stamps from the audio to be inspected by using dual-channel separation technology, and perform multi-layer filtering on the multiple target speaker speech streams to extract a dialogue speech stream containing only the target speaker; The ASR module is used to convert the conversation speech stream containing only the target speaker into text data using multilingual ASR technology, and optimize the text data based on Word-Level timing information and role annotation to reconstruct a standard conversation text; The large language model is used to identify and extract illegal dialogues in the standard dialogue text according to the set quality inspection rules to generate quality inspection data; The output post-processing module is used to convert the quality inspection data output by the large language model into a standard language format, store the quality inspection data converted into the standard language in a database, and simultaneously push it to the quality inspection business system for business personnel to review, analyze and provide feedback.

12. The multi-language audio quality inspection system based on speech processing and large language model according to claim 11, characterized in that: Before converting the quality inspection data output by the large language model into a standard language format, the output post-processing module needs to verify the text format of the quality inspection data output by the large language model to ensure that the text format meets the requirements.

13. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 0 are implemented.

Images