Audio processing method, and training method and device of text quality inspection large model
By segmenting audio, splicing text and using LoRA fine-tuning text quality inspection model, the existing system has solved the problem of low accuracy when understanding complex text and specific domain dialogues, achieving higher text quality inspection accuracy and accurate results Spend.
Patent Information
- Application Number
- CN202410975698.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-11-26
AI Technical Summary
Existing text quality inspection systems have limitations in understanding the deep meaning and context of text, especially when dealing with complex text and domain-specific dialogues.
By obtaining the audio to be processed, the audio of each speaker is divided, and the spliced text is determined based on the preset speech set, and input it to the pre-trained text quality inspection model for quality inspection, and output the speech violation type, basis for violation and Conversation summary, LoRA fine-tuning method is used to train text quality inspection large models to improve the accuracy of quality inspection results.
It improves the accuracy of text and the accuracy of quality inspection results, can meet the specified format requirements, and is suitable for quality inspection of complex texts and specific domain dialogues.
Smart Images

Figure CN119028353A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of audio processing technology, and in particular to an audio processing method, a training method and a device for a large text quality inspection model. Background Art
[0002] In the current market, audio can be converted into text, and most text quality checks use rule-based methods or machine learning models to detect grammatical errors, spelling errors, sentence structure, illegal language problems, etc. Although existing technologies perform well in grammar and spelling detection, they still have limitations in understanding the deeper meaning and context of the text. For example, there is a lack of understanding of puns, field-specific terms, or culturally specific expressions. In addition, complex texts, such as technical conversations or texts of a specific style, remain a challenge for current text quality detection systems. These texts often require a higher level of understanding, including field-specific knowledge and complex language structure analysis. Therefore, it is difficult for related technologies to guarantee the accuracy of the text, and there is a lack of effective quality inspection methods for the text. Summary of the invention
[0003] The purpose of the present invention is to provide an audio processing method, a training method and a device for a large text quality inspection model to improve the accuracy of the text and the accuracy of the quality inspection results.
[0004] The present invention provides an audio processing method, which includes: obtaining audio to be processed; processing the audio to be processed and outputting audio corresponding to each speaker; determining a spliced text based on the audio corresponding to each speaker and a preset speech set; inputting the spliced text into a pre-trained text quality inspection model to output a quality inspection result of the spliced text through the text quality inspection model; wherein the quality inspection result includes: speech violation type, violation basis and conversation summary.
[0005] Furthermore, based on the audio corresponding to each speaker and a preset set of words, the step of determining the spliced text includes: based on the preset set of words, converting the audio corresponding to each speaker into a text corresponding to each speaker; wherein each piece of text carries corresponding timestamp information; and splicing the texts corresponding to all speakers according to the timestamp information to obtain a spliced text.
[0006] Furthermore, after the step of outputting the audio corresponding to each speaker, the method further includes: based on a preset speech set, using a regular expression to identify the audio corresponding to each speaker, so as to determine the role corresponding to each speaker.
[0007] Furthermore, the step of processing the audio to be processed and outputting the audio corresponding to each speaker includes: removing invalid audio in the audio to be processed to obtain a first audio; determining whether the first audio is a mono audio; if the first audio is a mono audio, performing speaker segmentation on the first audio to determine the audio corresponding to each speaker in the first audio.
[0008] Furthermore, the step of performing speaker segmentation on the first audio and determining the audio corresponding to each speaker in the first audio includes: extracting voiceprint features from the first audio through a preset voiceprint extraction model; performing hierarchical clustering on the voiceprint features to obtain clustering results; and determining the audio corresponding to each speaker in the first audio based on the clustering results.
[0009] The present invention provides a training method for a text quality inspection large model, the method comprising: obtaining a sample text; wherein the sample text is pre-annotated with standard quality inspection information of the sample text; wherein the standard quality inspection information includes: standard speech violation types, standard violation bases and standard dialogue summaries; inputting the sample text into an initial text quality inspection large model, and outputting a predicted quality inspection result; wherein the predicted quality inspection result includes: predicted speech violation types, predicted violation bases and predicted dialogue summaries of the sample text; based on the predicted quality inspection result and the standard quality inspection result, the LoRA fine-tuning method is adopted to train the initial text quality inspection large model to obtain a trained text quality inspection large model.
[0010] The present invention provides an audio processing device, which includes: a first acquisition module, used to acquire audio to be processed; a first output module, used to process the audio to be processed and output the audio corresponding to each speaker; a determination module, used to determine a spliced text based on the audio corresponding to each speaker and a preset speech set; a second output module, used to input the spliced text into a pre-trained text quality inspection model, so as to output the quality inspection result of the spliced text through the text quality inspection model; wherein the quality inspection result includes: speech violation type, violation basis and conversation summary.
[0011] The present invention provides a training device for a text quality inspection large model, the device comprising: a second acquisition module, used to acquire sample text; wherein the sample text is pre-annotated with standard quality inspection information of the sample text; wherein the standard quality inspection information includes: standard speech violation type, standard violation basis and standard dialogue summary; a third output module, used to input the sample text into the initial text quality inspection large model, and output a predicted quality inspection result; wherein the predicted quality inspection result includes: the predicted speech violation type, predicted violation basis and predicted dialogue summary of the sample text; a training module, used to train the initial text quality inspection large model based on the predicted quality inspection result and the standard quality inspection result, using the LoRA fine-tuning method, to obtain a trained text quality inspection large model.
[0012] The present invention provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement any of the above-mentioned audio processing methods, or the above-mentioned text quality inspection large model training method.
[0013] The present invention provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement any of the above-mentioned audio processing methods, or the above-mentioned training method of the large text quality inspection model.
[0014] The audio processing method, text quality inspection large model training method and device provided by the present invention obtain the audio to be processed; process the audio to be processed and output the audio corresponding to each speaker; determine the spliced text based on the audio corresponding to each speaker and the preset speech set; input the spliced text into the pre-trained text quality inspection large model to output the quality inspection result of the spliced text through the text quality inspection large model; wherein the quality inspection result includes: speech violation type, violation basis and dialogue summary. This method can determine the spliced text corresponding to the audio to be processed based on the preset speech set, improve the accuracy of the text, and use the text quality inspection large model to perform quality inspection on the spliced text, and can obtain the quality inspection result that meets the specified format requirements, thereby improving the accuracy of the quality inspection result. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0016] Figure 1 A flowchart of an audio processing method provided by an embodiment of the present invention;
[0017] Figure 2 A schematic diagram of the structure of a Titanet network model provided by an embodiment of the present invention;
[0018] Figure 3 A flowchart of a method for training a large text quality inspection model provided by an embodiment of the present invention;
[0019] Figure 4 A schematic diagram of LoRA fine-tuning provided by an embodiment of the present invention;
[0020] Figure 5A flowchart of an audio processing method provided by an embodiment of the present invention;
[0021] Figure 6 A schematic diagram of the structure of an audio processing device provided by an embodiment of the present invention;
[0022] Figure 7 A schematic diagram of the structure of a training device for a large text quality inspection model provided by an embodiment of the present invention;
[0023] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0024] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0025] The text quality inspection system in the related art lacks an effective user feedback mechanism to continuously optimize and adjust the text quality inspection performance. The self-learning ability and the mechanism of self-optimization based on user feedback have not been fully realized in many systems. Based on this, the embodiment of the present invention provides an audio processing method, a training method and device for a large text quality inspection model, and the technology can be applied to applications that require quality inspection of text converted from audio.
[0026] To facilitate understanding of this embodiment, an audio processing method disclosed in an embodiment of the present invention is first introduced. Figure 1 As shown, the method comprises the following steps:
[0027] Step S102, obtaining audio to be processed;
[0028] The above-mentioned audio to be processed is usually audio of conversations between multiple speakers, for example, it can be daily call audio between two or more speakers, it can be call audio between customer service and customers in the financial industry, or it can be related call audio in other industries, etc.
[0029] Step S104, processing the audio to be processed, and outputting the audio corresponding to each speaker;
[0030] Since the audio to be processed is usually the audio of a conversation between multiple speakers, including conversations between multiple speakers, that is, the audio to be processed is a fusion of the audio of each speaker; therefore, the audio to be processed needs to be processed to separate the audio corresponding to each speaker from the audio to be processed.
[0031] Step S106, determining a concatenated text based on the audio and preset speech set corresponding to each speaker;
[0032] The above preset script set usually includes specific scripts that often appear in specific industry scenarios, such as "Here I am", "My employee number is", "I received your work order", "Hello, this is", and other specific phrases. These specific phrases are usually standard scripts used by customer service in conversations to introduce themselves, confirm information, or establish the formality of the conversation. In actual implementation, the text of the audio corresponding to each speaker can be determined based on the audio corresponding to each speaker and the preset script set, and then the spliced text containing the text corresponding to each speaker can be obtained by splicing.
[0033] Step S108, input the spliced text into a pre-trained text quality inspection model to output the quality inspection result of the spliced text through the text quality inspection model; wherein the quality inspection result includes: the type of speech violation, the basis for the violation and the dialogue summary.
[0034] The above-mentioned text quality inspection model can be a Transformer model, etc. The text quality inspection model is a large machine learning model that can process and analyze customer service dialogues, and can be used to identify and summarize situations where the speech violates the rules, etc.; the above-mentioned speech violation type can be used to indicate the type of violation that occurs in the dialogue, which may include: inappropriate language use, non-compliance with customer service standards, provision of misleading information, etc. The above-mentioned violation basis can indicate the basis or reason for the violation. For example, if the violation type is "misleading information", the violation basis may be "providing wrong product information". The above-mentioned dialogue summary can be understood as a brief summary of the entire dialogue content, especially emphasizing the key points related to the violation. In actual implementation, after determining the above-mentioned spliced text, the spliced text can be input into a pre-trained text quality inspection model, and the text quality inspection model outputs the quality inspection results such as speech violation type, violation basis and dialogue summary, that is, the text quality inspection model in this embodiment can output relevant quality inspection results in a fixed and consistent format.
[0035] The above audio processing method obtains the audio to be processed; processes the audio to be processed and outputs the audio corresponding to each speaker; determines the spliced text based on the audio corresponding to each speaker and the preset speech set; inputs the spliced text into a pre-trained text quality inspection model to output the quality inspection result of the spliced text through the text quality inspection model; wherein the quality inspection result includes: speech violation type, violation basis and dialogue summary. This method can determine the spliced text corresponding to the audio to be processed based on the preset speech set, improve the accuracy of the text, and use the text quality inspection model to inspect the spliced text, and can obtain the quality inspection result that meets the specified format requirements, thereby improving the accuracy of the quality inspection result.
[0036] The embodiment of the present invention further provides another audio processing method, which is implemented on the basis of the method in the above embodiment, and comprises the following steps:
[0037] Step 1: Obtain the audio to be processed;
[0038] Step 2, removing invalid audio from the audio to be processed to obtain the first audio;
[0039] There are usually invalid audios such as noise and silent segments in the audio to be processed. It is necessary to remove the invalid audio that is not related to the conversation in the audio to be processed. In actual implementation, a pre-trained VAD (Voice Activity Detection) model can be used to remove the invalid audio in the audio to be processed to obtain the first audio. The specific removal method can refer to the relevant technology and will not be repeated here. The VAD model can identify and eliminate long silent periods from the audio to be processed, and remove noise, so as to save voice channel resources without reducing service quality.
[0040] Step 3, determining whether the first audio is a mono audio;
[0041] In actual implementation, the first audio may be mono audio or dual-channel audio. It is necessary to determine whether the first audio is mono audio or dual-channel audio. For mono audio, speaker segmentation is required later. For dual-channel audio, the audio on each side needs to be extracted for subsequent processing. Mono audio refers to an audio format with only one channel, while dual-channel audio refers to an audio format with two channels. Mono audio can only play sound through one speaker, while dual-channel audio can play the sound of the left and right channels through two speakers respectively, thereby achieving a more stereoscopic sound effect.
[0042] Step 4: If the first audio is a mono audio, perform speaker segmentation on the first audio to determine the audio corresponding to each speaker in the first audio.
[0043] The step 4 can be implemented by the following steps 40 to 42:
[0044] Step 40, extracting voiceprint features from the first audio using a preset voiceprint extraction model;
[0045] The above voiceprint extraction model can choose the Titanet network model, which is an advanced voiceprint extraction model designed for voiceprint recognition. It can effectively extract the unique voiceprint features of each speaker and is used to identify and distinguish different speakers from mono audio. It identifies different speakers by analyzing subtle differences in the first audio. The architecture of the Titanet network model includes multiple levels, each of which focuses on extracting different voiceprint features from the first audio. These voiceprint features can then be used to identify and distinguish speakers.
[0046] like Figure 2 A structural diagram of a Titanet network model is shown, which includes an Encoder encoder and a Decoder decoder; the Encoder encoder includes a multi-layer convolutional network Conv-BN-Relu, and the Decoder decoder includes: Attention-Pooling-BN, Linear-BN and Linear (logits N); in actual implementation, the first audio is input to the Encoder encoder, and the multi-layer convolutional network Conv-BN-Relu in the Encoder encoder outputs higher-dimensional audio feature information in the first audio, and the audio feature information is sent to the Decoder decoder, normalized by the pooling layer (Attention-Pooling-BN) based on the attention mechanism, and then mapped by the mapping layer (Linear-BN), and finally the linear layer (Linear) outputs the unique voiceprint features of each speaker.
[0047] Voiceprints are not only specific, but also relatively stable. The above-mentioned voiceprint features may include the wavelength, frequency, intensity, rhythm, etc. of the sound. In the application scenario of this embodiment, mono audio is usually faced, which usually involves multiple different speakers. In order to effectively distinguish these multiple roles from the audio, speaker segmentation is required first. The purpose of this step is to separate different speakers from the mono audio. This is particularly critical when the audio contains two or more speakers. The Titanet network model can be used to extract the unique voiceprint features of each speaker.
[0048] Step 41, hierarchical clustering is performed on the voiceprint features to obtain a clustering result;
[0049] After extracting the voiceprint features of each speaker, these voiceprint features can be classified using hierarchical clustering technology. Hierarchical clustering is a method of grouping voiceprint features according to similarity. It creates a hierarchical nested clustering tree by calculating the similarity between voiceprint features of different categories. For example, taking the example of two speakers, the customer and the customer service, this means that the extracted voiceprint features need to be divided into two categories, corresponding to the two speakers in the audio, namely the customer and the customer service.
[0050] The following takes two speakers as an example to illustrate the distance calculation method in the hierarchical clustering process. Here, the clustering takes the average value of the two clusters as the center point, and takes the distance between the two means as the distance between the two clusters. The specific formula is as follows:
[0051]
[0052] Among them, X corresponds to the first cluster corresponding to the first speaker; Y corresponds to the second cluster corresponding to the second speaker; d(X,Y) is the distance between the two clusters; x corresponds to each voiceprint feature vector in X; y corresponds to each voiceprint feature vector in Y; Σx∈X is the sum of the vector lengths of all voiceprint feature vectors in X, Σy∈Y is the sum of the vector lengths of all voiceprint feature vectors in Y; |X| is the number of voiceprint feature vectors in X; |Y| is the number of voiceprint feature vectors in Y.
[0053] Step 42: determine the audio corresponding to each speaker in the first audio according to the clustering result.
[0054] After the clustering result is obtained, the audio corresponding to each speaker in the first audio can be determined according to the clustering result. For example, for two speakers, customer service and client, the audio corresponding to customer service and the audio corresponding to client can be determined according to the clustering result.
[0055] Step 5: Based on a preset speech set, a regular expression is used to identify the audio corresponding to each speaker to determine the role corresponding to each speaker.
[0056] During the text quality inspection process, when processing the converted text, a key task is usually faced, that is, distinguishing which text corresponds to which speaker. For example, for two speakers, customer and client, it is necessary to distinguish which text is the voice of the customer service and which is the voice of the customer. This step is crucial for understanding and analyzing the conversation process, because the voice of the customer service and the customer is usually different in content and style. In order to accurately perform this task, this embodiment adopts a method based on regular expressions.
[0057] Regular expressions are a powerful text matching tool that allows you to define specific patterns to identify and extract specific information from text. In the scenario where customer service staff are talking to customers, you can define a series of specific speech patterns that often appear in customer service conversations. For example, you can look for text containing specific phrases, such as "Here I am," "My employee number is," "I received your ticket," "Hello, this is," etc. These phrases are usually standard phrases used by customer service staff in conversations to introduce themselves, confirm information, or establish the formality of the conversation.
[0058] When these specific phrases are detected through regular expressions, the corresponding text fragments can be automatically marked as the voice of the customer service. If these specific phrases do not appear in the text, it is usually assumed that the part of the text comes from the customer's voice. Related technologies usually need to convert audio into text and then distinguish roles based on the text. This embodiment provides a simple and effective way to distinguish different roles in a conversation. It does not need to convert audio into text, but can directly distinguish roles based on the audio of each speaker.
[0059] In general, by applying regular expressions to identify specific preset words, the speech of different speakers can be effectively distinguished. This not only improves the accuracy of conversation analysis, but is also crucial for subsequent data processing and quality inspection activities. The advantage of this method is its high efficiency and low error rate, which maintains efficiency and accuracy when processing large amounts of conversation data.
[0060] Step 6: based on the preset speech set, convert the audio corresponding to each speaker into text corresponding to each speaker; wherein each text carries corresponding timestamp information;
[0061] Since the text quality inspection model is used to analyze and inspect text, the voice data must first be accurately converted into text form. To achieve this goal, an advanced ASR (Automatic Speech Recognition) service can be used. The function of the ASR service is to process the input voice signal and convert it into the corresponding text signal. This process involves complex signal processing and language understanding technology to ensure that accurate text content is extracted from the voice. A key consideration in the transcription process is to retain timestamp information. Timestamp information is crucial for recording the specific start and end time of each speech, which is critical for maintaining the coherence and contextual understanding of the conversation.
[0062] The main purpose of retaining timestamp information is to correctly splice subsequent conversations. In conversational text quality inspection, knowing the specific speaking time of each sentence not only helps to understand the conversation flow, but also helps to analyze important features such as pauses and speech speed changes in the conversation. These features may be key evaluation indicators in the quality inspection process, for example, analyzing the response time and fluency in customer service conversations.
[0063] In summary, using the ASR service to transcribe the audio corresponding to each speaker into text and retain key timestamp information is a basic step in establishing an efficient and accurate text quality inspection system. This not only improves the efficiency of data processing, but also ensures the integrity of the conversation content and the depth of analysis.
[0064] Step seven, concatenate the texts corresponding to all speakers according to the timestamp information to obtain a concatenated text.
[0065] In actual implementation, once the audio corresponding to each speaker is successfully distinguished and converted into text, the next key step is to splice these text fragments into a complete conversation form based on the timestamp information to obtain the spliced text. This process is crucial to understanding the context and flow of the entire communication, and helps analyze the quality and efficiency of the conversation.
[0066] The basic logic of splicing the texts corresponding to all speakers is to arrange the conversations of each speaker in order according to the timestamp information. The timestamp information, as a key data point, records the start and end time of each utterance, so that the timeline of the conversation can be accurately reconstructed. In this way, the coherence and logical order of the conversation can be ensured, so that the conversation reads naturally and smoothly.
[0067] For example, suppose the system detects a customer service voice clip that starts at 10:00AM with a timestamp that contains the text "Hello, this is ***". Then, the system finds a customer response with a timestamp of 10:01AM that reads "Well, hello, what's up?" The system will concatenate these two clips into a conversation in this order, with the following format:
[0068] Customer Service (10:00AM): “Hello, this is ***.”
[0069] Customer (10:01AM): “Um, hi, what’s up?”
[0070] This type of conversation stitching not only provides a clear timeline of the conversation, but also helps us analyze the response time, communication fluency, and possible communication problems in the conversation. For example, if there is a significant time delay between the customer service's speech and the customer's response, this may indicate a communication efficiency problem or customer confusion.
[0071] Splicing customer service and customer texts into a conversational format based on timestamp information is a key step in the text quality inspection system. This not only helps reconstruct and understand the context of the entire conversation, but is also crucial for analyzing the efficiency and quality of the conversation. This approach can provide deeper insights to help improve the quality and efficiency of customer service exchanges.
[0072] Step eight, input the spliced text into a pre-trained text quality inspection model to output the quality inspection results of the spliced text through the text quality inspection model; the quality inspection results include: the type of speech violation, the basis for the violation and the conversation summary.
[0073] The above-mentioned audio processing method can determine the spliced text corresponding to the audio to be processed based on a preset set of words, thereby improving the accuracy of the text, and uses a large text quality inspection model to perform quality inspection on the spliced text, thereby obtaining quality inspection results that meet the specified format requirements, thereby improving the accuracy of the quality inspection results.
[0074] The present invention provides a method for training a large text quality inspection model. Figure 3 As shown, the method comprises the following steps:
[0075] Step S302, obtaining a sample text; wherein the sample text is pre-marked with standard quality inspection information of the sample text; wherein the standard quality inspection information includes: standard speech violation type, standard violation basis and standard dialogue summary;
[0076] The sample texts mentioned above can be texts of conversations between multiple speakers, for example, texts of conversations between customers, etc. In order for the large text quality inspection model to understand and generate outputs in a specified format, the training set needs to be manually annotated to provide the large text quality inspection model with the correct learning examples. This means that we must carefully analyze each conversation in each sample text in the initial data set and manually annotate each conversation with the following three key elements:
[0077] Standard script violation types: This section needs to clearly mark the types of violations that occurred in the conversation. For example, this may include inappropriate language use, failure to comply with customer service standards, provision of misleading information, etc. Each violation type needs to be accurately identified and classified.
[0078] Standard Violation Basis: Here you need to specify the basis or reason for the violation. For example, if the violation type is "Misleading Information", the violation basis may be "Providing False Product Information". This part of the annotation is crucial for training the model to understand what behavior constitutes a violation.
[0079] Standard conversation summary: This section needs to provide a brief summary of the entire conversation content, especially highlighting the key points related to the violation. This helps the model understand the overall context of the conversation and the specific manifestations of the violation in the conversation.
[0080] Through the manual annotation process of sample text, a high-quality training set can be created, which is essential for training the model to accurately and consistently output analysis results in the specified format. This meticulous annotation process, although time-consuming, is necessary to ensure the accuracy and effectiveness of model training. Such a training set can help the model better understand and recognize complex conversation patterns, especially in the identification of speech violations, thereby improving its accuracy and reliability in practical applications.
[0081] Step S304: input the sample text into the initial text quality inspection model and output the predicted quality inspection result; wherein the predicted quality inspection result includes: the predicted speech violation type of the sample text, the predicted violation basis and the predicted dialogue summary;
[0082] Step S306: Based on the predicted quality inspection results and the standard quality inspection results, the LoRA fine-tuning method is used to train the initial text quality inspection large model to obtain a trained text quality inspection large model.
[0083] The above-mentioned LoRA (Low-Rank Adaptation) is a parameter adaptation method for neural network models, especially widely used in large pre-trained models such as Transformer models. The core idea of LoRA is to introduce a small number of trainable parameters to adapt to specific downstream tasks on the basis of the initial text quality inspection large model, thereby avoiding retraining the entire initial text quality inspection large model, and significantly reducing the computing cost and time. In this embodiment, the sample text can be input into the initial text quality inspection large model, and the output includes the predicted speech violation type, the predicted violation basis and the predicted dialogue summary. The loss value between the predicted quality inspection result and the pre-labeled standard quality inspection result can be calculated based on the preset loss function, and the initial text quality inspection large model is trained by the LoRA fine-tuning method until the loss value converges to obtain a trained text quality inspection large model.
[0084] like Figure 4 A schematic diagram of LoRA fine-tuning is shown in FIG.
[0085] W∈R d*d is the pre-training weight, that is, the weight parameter of the initial text quality inspection model in its original state, and d is the data dimension.
[0086] △W∈R d*dThe weights for fine-tuning are usually only a small part of the weight parameters of the initial text quality inspection model in the original state, such as 1%, and the remaining weights generally do not need to be adjusted.
[0087] Suppose the input is x and the output is h. x is the input of the text quality inspection model during the training process, such as the sample text, and h is the result displayed in the specified format.
[0088] Then there is the following formula:
[0089] h=Wx+△Wx;
[0090] In LoRA, matrices A and B are used to approximate △W;
[0091] A∈R r*d : A low-rank matrix A, where r is called the rank, is initialized with a Gaussian.
[0092] B∈R d*r : Low rank matrix B, initialize B;
[0093] After such a split, △W is rewritten as △W=BA, which reduces the number of fine-tuning parameters from d*d to 2*r*d. At the same time, in order not to change the output dimension, the following formula transformation is required:
[0094] h=Wx+BAx;
[0095] In addition, there is a hyperparameter ɑ as the scaling ratio to get the final output as follows:
[0096]
[0097] The traditional application is the transformer model. Figure 4 The left frame corresponds to the traditional transformer model structure. If it is trained, there are many parameters and a large amount of memory is required. In this embodiment, the part in the right frame replaces the part in the left frame, and the simulation is achieved by the AB matrix.
[0098] The training method of the above large text quality inspection model uses LoRa as a fine-tuning method to adapt to specific quality inspection tasks. By applying LoRa fine-tuning to the initial large text quality inspection model, effective learning and fitting can be achieved on a smaller data set (about 10,000 data points).
[0099] The following is an explanation of the fine-tuning process and effects:
[0100] Data volume and training rounds: Since the initial data is small, fine-tuning may not meet the requirements, so the data needs to be augmented. The augmentation can be done while maintaining the original data ratio.
[0101] In the test, about 10,000 data were used, which is less than the amount of data usually required. Nevertheless, after only 1-2 rounds of training, the model has achieved a good fitting effect. This shows that the LoRA fine-tuning method performs well in improving data efficiency and can quickly adapt to new tasks with fewer training rounds.
[0102] For example, if the input content is: The following text is the conversation text in a collection call. Does the customer service conversation belong to third-party disclosure, threats and intimidation, false promises, insulting customers, illegal identity, sensitive identity, collecting cash from customers, or no violation?
[0103] Customer service: Hello, Mr. Ren. Customer: Yes. Customer service: Yes, Mr. Ren, you have a Meituan living allowance in your name. I haven't paid it back for eight days. Do you remember? Customer: Yes. Customer service: Yes, I will pay it back this morning. It's already eight days overdue. Customer: Yes, I will pay it back as soon as possible. Customer service: As soon as possible. When is the best time? Customer: Yes, I will pay it back as soon as possible. Customer service: I'm not going to work today. You, you, you, this is not the case. I see that you have a lot of money. Is it convenient? Customer service: You see, there is still a limit on your credit limit. You, you ... Customer Service: Okay, this must be 2863 Customer Service: Okay Customer Service: Yes Customer: Goodbye Customer Service: Goodbye
[0104] The quality inspection results of the above text are output as follows:
[0105] There was no violation. During this conversation, the customer service representative contacted a customer named Mr. Ren and reminded him that a Meituan living expense bill under his name was overdue for eight days. The customer service asked Mr. Ren if he could pay off the debt that morning. Mr. Ren responded that he would deal with it as soon as possible. Then, the customer service strengthened the tone, asking when "as soon as possible" meant, and pointed out that Mr. Ren's current financial situation seemed to be poor, and suggested that he transfer some funds to pay off the bill. The customer service also reminded Mr. Ren that if he continued to be overdue, his credit limit might be frozen, making it impossible to borrow money in the future.
[0106] Finally, the customer service gave a specific repayment time limit, requiring the outstanding amount of 2,863 yuan to be paid off before 5 p.m. that day. The conversation ended with a brief farewell between the customer and the customer service.
[0107] Overall, this was a conversation about overdue bill reminders and collections, with customer service trying to ensure that Mr. Ren understood his financial responsibilities and dealt with the debt as quickly as possible, while also reminding him about the potential impact on his credit record.
[0108] Fitting effect: Through LoRA fine-tuning, the model quickly learned the task-related features and patterns, and achieved effective fitting of the quality inspection task. This shows that LoRA effectively introduced task-specific adjustments while maintaining the original pre-trained model knowledge structure.
[0109] Quality inspection results: In terms of quality inspection, the test results show that the output of the model fine-tuned by LoRA is in line with expectations. This means that the model is not only adapted to new tasks in theory, but also produces high-quality and accurate results in practical applications.
[0110] In summary, the model optimization achieved by the LoRA fine-tuning method in this embodiment demonstrates the potential for efficient learning and accurate quality inspection with a smaller data set and limited training rounds. This method not only improves the adaptability and flexibility of the model, but also achieves the expected results in quality inspection, proving its effectiveness and practicality in practical applications.
[0111] For easier understanding, see Figure 5 The flowchart of an audio processing method shown in the figure first removes the mute on the quality inspection audio to obtain the mutated audio (corresponding to the first audio mentioned above), and determines whether the first audio is mono audio or dual-channel audio. If it is mono audio, speaker segmentation is required. If it is dual-channel audio, the audio on each side is extracted to determine the audio on both sides of the call (corresponding to the audio corresponding to each speaker mentioned above), and then role discrimination is performed to distinguish the audio on both sides of the customer service and the customer. Through ASR speech recognition, the call audio between the customer and the customer service is converted into text. The transcribed text content needs to be spliced according to the timestamp information and the order of the conversation to obtain the actual conversation order text. Finally, the fine-tuned ChatGLM model is used to output the results, wherein the ChatGLM model is trained based on the LoRA fine-tuning method.
[0112] The above-mentioned audio processing method is a quality inspection system based on a large text quality inspection model. It mainly involves silence detection, speaker segmentation and clustering, speech recognition, role discrimination, and fine-tuning of a large text quality inspection model. Among them, silence detection is mainly used for the front-end to uniformly remove silence from the quality inspection audio. Speaker segmentation and clustering divides the audio after silence removal into the audio of the customer service and customer sides. Speech recognition technology is used to transcribe the audio of the customer service and customer into text for subsequent input into the large text quality inspection model. Fine-tuning of the large text quality inspection model is used to fine-tune the initial large text quality inspection model according to a certain amount of real-world scenario data to achieve a predetermined output format.
[0113] A large text quality inspection model is used as the quality inspection model, which has a large number of parameters and strong generalization ability. The LoRA method is used to fine-tune the large text quality inspection model, which only requires about 10,000 data to achieve good performance. The fine-tuning data is manually structured output, so that the output of the large text quality inspection model can also meet the actual application needs.
[0114] The embodiment of the present invention provides an audio processing device, such as Figure 6 As shown, the device comprises:
[0115] A first acquisition module 60, used to acquire audio to be processed;
[0116] A first output module 61 is used to process the audio to be processed and output the audio corresponding to each speaker;
[0117] A determination module 62, configured to determine a concatenated text based on the audio and a preset speech set corresponding to each speaker;
[0118] The second output module 63 is used to input the spliced text into a pre-trained text quality inspection model to output the quality inspection result of the spliced text through the text quality inspection model; wherein the quality inspection result includes: the type of speech violation, the basis for the violation and the dialogue summary.
[0119] The above-mentioned audio processing device can determine the spliced text corresponding to the audio to be processed based on a preset set of words, thereby improving the accuracy of the text, and use a large text quality inspection model to perform quality inspection on the spliced text, thereby obtaining quality inspection results that meet the specified format requirements, thereby improving the accuracy of the quality inspection results.
[0120] Furthermore, the determination module is also used to: based on a preset set of words, convert the audio corresponding to each speaker into text corresponding to each speaker; wherein each piece of text carries corresponding timestamp information; and splice the texts corresponding to all speakers according to the timestamp information to obtain a spliced text.
[0121] Furthermore, the device is also used to: based on a preset speech set, use regular expressions to identify the audio corresponding to each speaker, so as to determine the role corresponding to each speaker.
[0122] Furthermore, the first output module is also used to: remove invalid audio from the audio to be processed to obtain the first audio; determine whether the first audio is mono audio; if the first audio is mono audio, perform speaker segmentation on the first audio to determine the audio corresponding to each speaker in the first audio.
[0123] Furthermore, the first output module is also used to: extract voiceprint features from the first audio through a preset voiceprint extraction model; perform hierarchical clustering on the voiceprint features to obtain clustering results; and determine the audio corresponding to each speaker in the first audio based on the clustering results.
[0124] The audio processing device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned audio processing method embodiment. For the sake of brief description, for matters not mentioned in the audio processing device embodiment, reference may be made to the corresponding contents in the aforementioned audio processing method embodiment.
[0125] The embodiment of the present invention provides a training device for a large text quality inspection model, such as Figure 7 As shown, the device comprises:
[0126] The second acquisition module 70 is used to acquire a sample text; wherein the sample text is pre-marked with standard quality inspection information of the sample text; wherein the standard quality inspection information includes: standard speech violation type, standard violation basis and standard dialogue summary;
[0127] The third output module 71 is used to input the sample text into the initial text quality inspection model and output the predicted quality inspection result; wherein the predicted quality inspection result includes: the predicted speech violation type of the sample text, the predicted violation basis and the predicted dialogue summary;
[0128] The training module 72 is used to train the initial text quality inspection model based on the predicted quality inspection results and the standard quality inspection results by using the LoRA fine-tuning method to obtain a trained text quality inspection model.
[0129] The training device of the above-mentioned large text quality inspection model uses LoRa as a fine-tuning method to adapt to specific quality inspection tasks. By applying LoRa fine-tuning to the initial large text quality inspection model, effective learning and fitting can be achieved on smaller data sets.
[0130] The embodiment of the present invention further provides an electronic device, see Figure 8 As shown, the electronic device includes a processor 130 and a memory 131, wherein the memory 131 stores machine executable instructions that can be executed by the processor 130, and the processor 130 executes the machine executable instructions to implement the above-mentioned audio processing method, or the training method of the large text quality inspection model.
[0131] Further, Figure 8 The electronic device shown further includes a bus 132 and a communication interface 133 , and the processor 130 , the communication interface 133 and the memory 131 are connected via the bus 132 .
[0132] The memory 131 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 133 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 132 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0133] The processor 130 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 130. The above processor 130 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 131, and the processor 130 reads the information in the memory 131 and completes the steps of the method of the above embodiment in combination with its hardware.
[0134] An embodiment of the present invention also provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned audio processing method, or the training method of the large text quality inspection model. The specific implementation can be found in the method embodiment, which will not be repeated here.
[0135] The computer program product of the audio processing method, text quality inspection large model training method and device provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.
[0136] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.
[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. An audio processing method, It is characterized in that The method comprises: Get the audio to be processed; Processing the audio to be processed, and outputting the audio corresponding to each speaker; Determine the spliced text based on the audio and preset speech set corresponding to each speaker; The spliced text is input into a pre-trained text quality inspection model to output the quality inspection result of the spliced text through the text quality inspection model; wherein the quality inspection result includes: the type of speech violation, the basis for the violation and the dialogue summary.
2. The method according to claim 1, It is characterized in that Based on the audio and preset speech set corresponding to each speaker, the steps of determining the spliced text include: Based on a preset speech set, the audio corresponding to each speaker is converted into text corresponding to each speaker; wherein each text carries corresponding timestamp information; The texts corresponding to all speakers are concatenated according to the timestamp information to obtain a concatenated text.
3. The method according to claim 1, It is characterized in that After the step of outputting the audio corresponding to each speaker, the method further includes: Based on a preset set of speech scripts, regular expressions are used to identify the audio corresponding to each speaker to determine the role corresponding to each speaker.
4. The method according to claim 1, It is characterized in that The step of processing the audio to be processed and outputting the audio corresponding to each speaker comprises: Removing invalid audio from the audio to be processed to obtain a first audio; Determine whether the first audio is mono audio; If the first audio is a mono audio, speaker segmentation is performed on the first audio to determine audio corresponding to each speaker in the first audio.
5. The method according to claim 4, It is characterized in that The step of performing speaker segmentation on the first audio to determine the audio corresponding to each speaker in the first audio comprises: Extracting voiceprint features from the first audio using a preset voiceprint extraction model; Performing hierarchical clustering on the voiceprint features to obtain a clustering result; According to the clustering result, the audio corresponding to each speaker in the first audio is determined.
6. A training method for a large text quality inspection model. It is characterized in that The method comprises: Obtaining a sample text; wherein the sample text is pre-marked with standard quality inspection information of the sample text; wherein the standard quality inspection information includes: standard speech violation type, standard violation basis and standard dialogue summary; Input the sample text into the initial text quality inspection model, and output the predicted quality inspection result; wherein the predicted quality inspection result includes: the predicted speech violation type of the sample text, the predicted violation basis and the predicted dialogue summary; Based on the predicted quality inspection results and the standard quality inspection results, the LoRA fine-tuning method is adopted to train the initial text quality inspection large model to obtain a trained text quality inspection large model.
7. An audio processing device, It is characterized in that The device comprises: A first acquisition module, used to acquire audio to be processed; A first output module, used to process the audio to be processed and output the audio corresponding to each speaker; A determination module, used to determine the concatenated text based on the audio and preset speech set corresponding to each speaker; The second output module is used to input the spliced text into a pre-trained text quality inspection model to output the quality inspection result of the spliced text through the text quality inspection model; wherein the quality inspection result includes: the type of speech violation, the basis for the violation and the dialogue summary.
8. A training device for a large text quality inspection model, It is characterized in that The device comprises: The second acquisition module is used to acquire a sample text; wherein the sample text is pre-marked with standard quality inspection information of the sample text; wherein the standard quality inspection information includes: standard speech violation type, standard violation basis and standard dialogue summary; The third output module is used to input the sample text into the initial text quality inspection model and output the predicted quality inspection result; wherein the predicted quality inspection result includes: the predicted speech violation type, predicted violation basis and predicted dialogue summary of the sample text; The training module is used to train the initial text quality inspection model based on the predicted quality inspection results and the standard quality inspection results by using the LoRA fine-tuning method to obtain a trained text quality inspection model.
9. An electronic device, It is characterized in that It includes a processor and a memory, the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the audio processing method described in any one of claims 1 to 5, or the training method of the large text quality inspection model described in claim 6.
10. A machine-readable storage medium, It is characterized in that The machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by the processor, the machine-executable instructions prompt the processor to implement the audio processing method described in any one of claims 1 to 5, or the training method of the large text quality inspection model described in claim 6.
