Customer Voice Analysis System Based on Large Model

Abstract

The present invention discloses a customer speech analysis system based on a large model, including: a speech data collection and preprocessing module, which is used to collect customer speech data and perform preprocessing; a speech recognition module, which is used to recognize the preprocessed customer speech data; a feature fusion module, which is used to extract the semantic features of text data, and fuse the features of customer speech data to generate speech text representation; a large model fine-tuning module, which is used to fine-tune the pre-trained Spark cognitive large model; a semantic analysis and emotion recognition module, which is used to process the speech text representation, extract semantic information, recognize the emotional state, and generate analysis results; a dialogue context management module, which is used to associate customer historical speech data, establish dialogue context information, and update speech analysis results. The present invention utilizes the Spark cognitive large model and improved quantized low-rank adaptation, and has the advantages of high precision, low computing cost, and strong intelligent adaptability.

Description

Technical Field

[0001] The present invention relates to the field of speech processing, and in particular to a customer speech analysis system based on a large model. Background Art

[0002] Against the backdrop of the rapid development of artificial intelligence and natural language processing, speech analysis technology has been widely used in many fields, including intelligent customer service, financial risk control, emotional computing, and human-computer interaction. Current speech analysis systems mainly rely on automatic speech recognition technology to convert speech data into text data, and further use natural language processing technology for analysis. Although such methods can perform semantic analysis on speech content, due to the lack of non-text information of speech, such as pitch, speech speed, energy changes, etc., the analysis results are prone to lose key information and cannot accurately identify the customer's emotional state, intention expression, and potential needs. In addition, existing speech analysis technologies also face many challenges in processing multimodal information fusion, model fine-tuning, and knowledge management, making it difficult to meet the needs of high-precision and highly intelligent speech analysis.

[0003] Traditional speech analysis methods mainly include rule-based speech analysis and emotion recognition methods based on shallow machine learning. Rule-based methods usually rely on preset speech feature parameters, such as volume, pitch fluctuation, etc., and classify speech emotions through fixed logical rules. This type of method can achieve certain results in specific application scenarios, but due to the limited nature of the rules, it is difficult to adapt to complex and changeable customer voice scenarios, and lacks generalization capabilities when processing multiple voice modes. Methods based on shallow machine learning, such as support vector machines, random forests, and traditional neural networks, usually rely on manually extracted speech features for emotion classification. Although this type of method has better data-driven capabilities than rule-based methods, due to the lack of contextual understanding capabilities and the inability to combine text information, there is still a serious information loss problem, making it difficult to accurately analyze complex speech situations.

[0004] In recent years, with the development of deep learning and large models, speech analysis methods based on end-to-end neural networks have gradually become mainstream. For example, convolutional neural networks, long short-term memory networks, and speech sentiment analysis models based on Transformer structures have improved the accuracy of speech sentiment recognition to a certain extent. However, these models still face multiple technical bottlenecks: on the one hand, traditional speech analysis systems usually rely on independent ASR modules and NLP modules. This separate processing method leads to data fragmentation, affecting the consistency and real-time nature of the analysis; on the other hand, although the existing large models have strong semantic understanding capabilities, they still focus on text when processing speech data and cannot fully utilize the non-text information in the speech. In addition, the existing large models have high computational overhead and high storage requirements during fine-tuning, resulting in high costs and technical barriers for enterprises during deployment and application.

[0005] In response to the above problems, the introduction of the Spark cognitive big model provides new possibilities for speech analysis. The Spark cognitive big model has powerful cross-modal learning capabilities and can process multimodal information such as text and speech at the same time. However, the existing model fine-tuning methods are difficult to efficiently adapt to speech analysis tasks when computing resources are limited. The traditional full-parameter fine-tuning method requires a lot of computing resources, which limits the feasibility of large models in enterprise-level applications. In addition, the existing ASR-NLP combination solution still has the problem of speech information loss. Only semantic analysis through text cannot accurately capture the true emotions of customers. Therefore, how to optimize the speech analysis process based on the Spark cognitive big model, realize the deep integration of speech and text, and reduce the computational cost of model fine-tuning has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] One purpose of the present invention is to propose a customer speech analysis system based on a large model. The present invention introduces an improved quantized low-rank adaptation method and adapter fusion to efficiently fine-tune the large model, while reducing computing requirements and maintaining the high performance expression capability of the model. Multimodal fusion technology is used to combine text information with speech data features to achieve more accurate speech sentiment analysis and customer intent recognition. A context-aware mechanism is introduced to use historical speech data to construct conversation context information, thereby improving the accuracy and consistency of speech analysis.

[0007] A customer speech analysis system based on a large model according to an embodiment of the present invention includes:

[0008] Voice data collection and preprocessing module, used to collect customer voice data and preprocess the customer voice data;

[0009] A speech recognition module is used to recognize the pre-processed customer speech data and generate text data;

[0010] The feature fusion module is used to extract the semantic features of text data and fuse the features of customer voice data to generate a fused voice-text representation;

[0011] The large model fine-tuning module is used to fine-tune the pre-trained Spark cognitive large model;

[0012] The semantic analysis and emotion recognition module is used to process the fused speech-text representation, extract semantic information, identify the emotional state, and generate analysis results;

[0013] The conversation context management module is used to associate customer historical voice data, establish conversation context information, and update voice analysis results.

[0014] Optionally, modules can be connected via the following methods:

[0015] S1. Collect customer voice data and pre-process the customer voice data;

[0016] S2, recognize the pre-processed customer voice data and generate text data;

[0017] S3, extracting semantic features of text data and obtaining features of customer voice data to generate fused voice-text representation;

[0018] S4, using improved quantized low-rank adaptation method and adapter fusion method to fine-tune the parameters of the pre-trained Spark cognitive large model;

[0019] S5. Use the Spark cognitive big model to process the fused speech and text representation, extract semantic information, identify emotional states, and generate speech analysis results;

[0020] S6. Associate the customer's historical voice data, establish conversation context information, and update the voice analysis results.

[0021] Optionally, the customer voice data features include pitch, frequency, energy and speaking speed.

[0022] Optionally, S4 includes the following specific steps:

[0023] S41. Constructing the parameter weight matrix of the Spark cognitive model , perform low-rank decomposition on W, use dynamic low-rank optimization method to calculate the optimal rank of W, and the d in the parameter weight matrix represents the dimension:

[0024] ;

[0025] Among them, W represents the parameter weight matrix, A represents the left matrix of low-rank decomposition, which is part of the weight after dimensionality reduction. , B represents the right matrix of low-rank decomposition, which is responsible for restoring the complete weight information, , r represents the low-rank dimension, and is the reduced parameter scale, which is optimized using the least squares error minimization method:

[0026] ;

[0027] in, represents the i-th row of the parameter weight matrix, express The i-th row of represents the norm, d represents the dimension of the parameter weight matrix, and argmin represents the variable value that minimizes the function value;

[0028] S42, quantize the parameter weight matrix using 4-bit quantization:

[0029] ;

[0030] in, represents the quantized parameter weight matrix, represents the minimum value in the parameter weight matrix, Indicates the maximum value in the parameter weight matrix, b indicates the number of quantization bits, and round indicates rounding the value to the nearest integer;

[0031] S43, using a group quantization method, independently quantizing the parameter weight matrices of different layers according to 4-bit quantization, and restoring the parameter weight matrices after group quantization:

[0032] ;

[0033] ;

[0034] in, represents the parameter weight matrix after group quantization, Quantize represents the quantization function, which is the 4-bit quantization in step S42, represents the parameter weight matrix of the lth layer, b represents the number of quantization bits, Represents the parameter weight matrix after inverse quantization;

[0035] S44, using error compensation method to reduce the impact of quantization error, calculating gradient adjustment, using dynamic gradient adjustment strategy, and making corrections when the error exceeds the threshold:

[0036] ;

[0037] ;

[0038] Among them, g' represents the corrected gradient, g represents the gradient of the parameter weight matrix update, represents the parameter weight matrix after inverse quantization, represents the error compensation coefficient, represents the error adjustment term, Represents the indicator function. When the error exceeds the threshold When the error correction is triggered, represents the quantization error threshold, d represents the dimension of the parameter weight matrix, Represents the parameter weight matrix after inverse quantization;

[0039] S45. Let the input feature be expressed as , the adapter layer parameter matrix is , calculate the adapted feature representation:

[0040] ;

[0041] Among them, X' represents the adapted feature representation, X represents the input feature representation, Represents the training parameters, ReLU represents the rectified linear unit activation function, represents the adapter layer parameter matrix;

[0042] S46. Assume that the output probability distribution of the pre-trained Spark cognitive model is , the output probability distribution of the pre-trained Spark cognitive model after fine-tuning is , the Kullback-Leibler divergence is used to calculate the distillation loss:

[0043] ;

[0044] Where L represents the distillation loss, represents the output probability distribution of the pre-trained Spark cognitive model, represents the output probability distribution of the pre-trained Spark Cognitive Large Model after fine-tuning, y represents the category index, and x represents the input data;

[0045] S47, using distillation temperature parameters Smooth the output probability distribution of the pre-trained Spark Cognitive Big Model:

[0046] ;

[0047] in, represents the logits value of the pre-trained Spark cognitive model, Represents the distillation temperature parameter, exp represents the natural exponential function, y' represents a variable when traversing all categories, and t represents the current time step.

[0048] Optionally, S5 includes the following specific steps:

[0049] S51, assuming that the fused speech text is represented by The fused speech-text representation includes the semantic features of the text data: And the customer voice data features are , and They are the dimensions of the semantic features of text data and the features of customer voice data:

[0050] ;

[0051] in, represents the fused speech-text representation, represents the vector concatenation operation, Represents the semantic features of text data, Indicates the characteristics of customer voice data;

[0052] S52. Use the embedding layer of the Spark cognitive model to encode the fused speech-text representation:

[0053] ;

[0054] Among them, H represents the hidden state, and E represents the embedding matrix, which is determined by the pre-trained weights of the Spark Cognitive Big Model;

[0055] S53. Use the multi-head attention mechanism to calculate the weight distribution of each input vector and calculate the attention score:

[0056] ;

[0057] ;

[0058] in, represents the output of the h-th attention head, Softmax represents normalization, , and vectors representing queries, keys, and values, , and weight matrices representing queries, keys, and values, Represents the query vector The dimension of represents the numerical stability correction term, represents the transpose of the key vector;

[0059] S54, using the multi-layer Transformer encoder of the Spark cognitive large model to perform multi-head attention calculation on the hidden state:

[0060] ;

[0061] in, Indicates the current layer hidden state, Represents the hidden state of the previous layer, LayerNorm represents layer normalization, and MultiHeadAttention represents multi-head attention calculation;

[0062] S55. Use gated networks to optimize attention output:

[0063] ;

[0064] ;

[0065] in, represents the gating factor of the lth layer, Sigmoid represents the activation function, represents the gating weight matrix, Indicates the current layer hidden state, represents the bias term, represents the optimized hidden state, Represents the hidden state of the previous layer;

[0066] S56. The output layer of the Spark Cognitive Model is used to calculate the sentiment classification score, and the Softmax function is used to calculate the probability distribution of the sentiment category:

[0067] ;

[0068] ;

[0069] Among them, Z represents the sentiment category score, represents the classification bias vector, represents the classification weight matrix, represents the probability distribution of sentiment category, and y represents the sentiment category index;

[0070] S57, extracting keywords using an attention-guided keyword extraction method, and selecting keywords using an adaptive threshold method:

[0071] ;

[0072] ;

[0073] in, represents the total attention weight of the k-th word, represents the keyword extraction transformation matrix, ReLU represents the rectified linear unit activation function, represents the attention score of the k-th word, represents the hidden state of the tth layer, T represents the number of layers of hidden states, represents the dynamic threshold, represents the mean value of keyword weight, Indicates the standard deviation of keyword weights, represents the adjustment factor;

[0074] Finally, we get the keyword set K:

[0075] ;

[0076] in, represents the kth keyword;

[0077] S58. Combine sentiment classification results with keyword extraction results to generate structured data:

[0078] ;

[0079] Among them, R represents structured data, and argmax represents the variable value when the function reaches the maximum value.

[0080] The beneficial effects of the present invention are:

[0081] First, by introducing an improved quantized low-rank adaptation method and adapter fusion technology, the present invention significantly reduces the computing resource requirements while ensuring the efficient operation of large models, allowing enterprises to deploy and optimize large models at a lower hardware cost, thereby improving the scalability and real-time performance of the speech analysis system. In addition, compared with the loss of speech information caused by text conversion in the traditional ASR-NLP processing flow, the present invention adopts a multimodal feature fusion strategy to deeply combine text information with speech features, so that the speech analysis system can focus on the details of the semantic level and the speech expression level at the same time, thereby achieving more accurate customer emotion recognition and intention analysis.

[0082] Secondly, the present invention utilizes a context management mechanism so that customer voice analysis is not limited to a single round of conversation, but can be combined with historical voice data to construct conversation context information and achieve more natural semantic understanding. The introduction of this mechanism enables the system to continuously optimize the voice analysis results, accurately identify changes in customer intentions, and dynamically adjust semantic weights through time window management to improve semantic coherence and accuracy in multiple rounds of conversations. In addition, the present invention adopts a keyword extraction method based on an attention mechanism, so that the system can automatically identify key information in customer voices, and construct structured data in combination with sentiment analysis results, making the analysis results more intuitive and explainable, which helps companies make accurate decisions in intelligent customer service, user feedback management, and market analysis.

[0083] Finally, the present invention achieves more accurate, intelligent and efficient customer voice analysis. Compared with the existing technology, the present invention can improve the accuracy of voice emotion recognition, effectively reduce voice information loss, improve the coherence of semantic understanding, and optimize the intelligent capabilities of the business system through adaptive updating of knowledge graphs. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0085] Figure 1 A method flow chart of a customer speech analysis system based on a large model proposed by the present invention;

[0086] Figure 2 The present invention proposes an optimization process for a large model fine-tuning module of a large model-based customer speech analysis system. DETAILED DESCRIPTION

[0087] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0088] refer to Figure 1 and Figure 2 , a customer speech analysis system based on a large model, comprising:

[0089] Voice data collection and preprocessing module, used to collect customer voice data and preprocess the customer voice data;

[0090] A speech recognition module is used to recognize the pre-processed customer speech data and generate text data;

[0091] The feature fusion module is used to extract the semantic features of text data and fuse the features of customer voice data to generate a fused voice-text representation;

[0092] The large model fine-tuning module is used to fine-tune the pre-trained Spark cognitive large model;

[0093] The semantic analysis and emotion recognition module is used to process the fused speech-text representation, extract semantic information, identify the emotional state, and generate analysis results;

[0094] The conversation context management module is used to associate customer historical voice data, establish conversation context information, and update voice analysis results.

[0095] In this implementation, the modules are implemented by the following method:

[0096] S1. Collect customer voice data and pre-process the customer voice data;

[0097] S2, recognize the pre-processed customer voice data and generate text data;

[0098] S3, extracting semantic features of text data and obtaining features of customer voice data to generate fused voice-text representation;

[0099] S4, using improved quantized low-rank adaptation method and adapter fusion method to fine-tune the parameters of the pre-trained Spark cognitive large model;

[0100] S5. Use the Spark cognitive big model to process the fused speech and text representation, extract semantic information, identify emotional states, and generate speech analysis results;

[0101] S6. Associate the customer's historical voice data, establish conversation context information, and update the voice analysis results.

[0102] In this implementation, the customer voice data features include pitch, frequency, energy and speaking speed.

[0103] In this implementation, S4 includes the following specific steps:

[0104] S41. Constructing the parameter weight matrix of the Spark cognitive model , perform low-rank decomposition on W, use dynamic low-rank optimization method to calculate the optimal rank of W, and the d in the parameter weight matrix represents the dimension:

[0105] ;

[0106] Among them, W represents the parameter weight matrix, A represents the left matrix of low-rank decomposition, which is part of the weight after dimensionality reduction. , B represents the right matrix of low-rank decomposition, which is responsible for restoring the complete weight information, , r represents the low-rank dimension, and is the reduced parameter scale, which is optimized using the least squares error minimization method:

[0107] ;

[0108] in, represents the i-th row of the parameter weight matrix, express The i-th row of represents the norm, d represents the dimension of the parameter weight matrix, and argmin represents the variable value that minimizes the function value;

[0109] S42, quantize the parameter weight matrix using 4-bit quantization:

[0110] ;

[0111] in, represents the quantized parameter weight matrix, represents the minimum value in the parameter weight matrix, Indicates the maximum value in the parameter weight matrix, b indicates the number of quantization bits, and round indicates rounding the value to the nearest integer;

[0112] S43, using a group quantization method, independently quantizing the parameter weight matrices of different layers according to 4-bit quantization, and restoring the parameter weight matrices after group quantization:

[0113] ;

[0114] ;

[0115] in, represents the parameter weight matrix after group quantization, Quantize represents the quantization function, which is the 4-bit quantization in step S42, represents the parameter weight matrix of the lth layer, b represents the number of quantization bits, Represents the parameter weight matrix after inverse quantization;

[0116] S44, using error compensation method to reduce the impact of quantization error, calculating gradient adjustment, using dynamic gradient adjustment strategy, and making corrections when the error exceeds the threshold:

[0117] ;

[0118] ;

[0119] Among them, g' represents the corrected gradient, g represents the gradient of the parameter weight matrix update, represents the parameter weight matrix after inverse quantization, represents the error compensation coefficient, represents the error adjustment term, Represents the indicator function. When the error exceeds the threshold When the error correction is triggered, represents the quantization error threshold, d represents the dimension of the parameter weight matrix, Represents the parameter weight matrix after inverse quantization;

[0120] S45. Let the input feature be expressed as , the adapter layer parameter matrix is , calculate the adapted feature representation:

[0121] ;

[0122] Among them, X' represents the adapted feature representation, X represents the input feature representation, Represents the training parameters, ReLU represents the rectified linear unit activation function, represents the adapter layer parameter matrix;

[0123] S46. Assume that the output probability distribution of the pre-trained Spark cognitive model is , the output probability distribution of the pre-trained Spark cognitive model after fine-tuning is , the Kullback-Leibler divergence is used to calculate the distillation loss:

[0124] ;

[0125] Where L represents the distillation loss, represents the output probability distribution of the pre-trained Spark cognitive model, represents the output probability distribution of the pre-trained Spark Cognitive Large Model after fine-tuning, y represents the category index, and x represents the input data;

[0126] S47, using distillation temperature parameters Smooth the output probability distribution of the pre-trained Spark Cognitive Big Model:

[0127] ;

[0128] in, represents the logits value of the pre-trained Spark cognitive model, Represents the distillation temperature parameter, exp represents the natural exponential function, y' represents a variable when traversing all categories, and t represents the current time step.

[0129] In this implementation, S5 includes the following specific steps:

[0130] S51, assuming that the fused speech text is represented by The fused speech-text representation includes the semantic features of the text data: And the customer voice data features are , and They are the dimensions of the semantic features of text data and the features of customer voice data:

[0131] ;

[0132] in, represents the fused speech-text representation, represents the vector concatenation operation, Represents the semantic features of text data, Indicates the characteristics of customer voice data;

[0133] S52. Use the embedding layer of the Spark cognitive model to encode the fused speech-text representation:

[0134] ;

[0135] Among them, H represents the hidden state, and E represents the embedding matrix, which is determined by the pre-trained weights of the Spark Cognitive Big Model;

[0136] S53. Use the multi-head attention mechanism to calculate the weight distribution of each input vector and calculate the attention score:

[0137] ;

[0138] ;

[0139] in, represents the output of the h-th attention head, Softmax represents normalization, , and vectors representing queries, keys, and values, , and weight matrices representing queries, keys, and values, Represents the query vector The dimension of represents the numerical stability correction term, represents the transpose of the key vector;

[0140] S54, using the multi-layer Transformer encoder of the Spark cognitive large model to perform multi-head attention calculation on the hidden state:

[0141] ;

[0142] in, Indicates the current layer hidden state, Represents the hidden state of the previous layer, LayerNorm represents layer normalization, and MultiHeadAttention represents multi-head attention calculation;

[0143] S55. Use gated networks to optimize attention output:

[0144] ;

[0145] ;

[0146] in, represents the gating factor of the lth layer, Sigmoid represents the activation function, represents the gating weight matrix, Indicates the current layer hidden state, represents the bias term, represents the optimized hidden state, Represents the hidden state of the previous layer;

[0147] S56. The output layer of the Spark Cognitive Model is used to calculate the sentiment classification score, and the Softmax function is used to calculate the probability distribution of the sentiment category:

[0148] ;

[0149] ;

[0150] Among them, Z represents the sentiment category score, represents the classification bias vector, represents the classification weight matrix, represents the probability distribution of sentiment category, and y represents the sentiment category index;

[0151] S57, extracting keywords using an attention-guided keyword extraction method, and selecting keywords using an adaptive threshold method:

[0152] ;

[0153] ;

[0154] in, represents the total attention weight of the k-th word, represents the keyword extraction transformation matrix, ReLU represents the rectified linear unit activation function, represents the attention score of the k-th word, represents the hidden state of the tth layer, T represents the number of layers of hidden states, represents the dynamic threshold, represents the mean value of keyword weight, Indicates the standard deviation of keyword weights, represents the adjustment factor;

[0155] Finally, we get the keyword set K:

[0156] ;

[0157] in, represents the kth keyword;

[0158] S58. Combine sentiment classification results with keyword extraction results to generate structured data:

[0159] ;

[0160] Among them, R represents structured data, and argmax represents the variable value when the function reaches the maximum value.

[0161] Embodiment 1:

[0162] In the outpatient and emergency departments of modern hospitals, doctors need to receive a large number of patients every day, record medical descriptions, analyze medical histories, develop treatment plans, and provide detailed health advice to patients. Traditional medical record keeping methods mainly rely on doctors' manual writing or text transcription based on automatic speech recognition systems. However, existing speech transcription systems can only extract text information and cannot effectively capture non-linguistic features in speech, such as the patient's speaking speed, pitch, energy changes, etc. This makes it difficult for doctors to fully understand the patient's actual condition in subsequent diagnosis and treatment, especially when dealing with complex symptoms, mental health problems or emergency conditions, which may lead to misjudgment due to the lack of speech features.

[0163] For example, when an emergency patient describes his symptoms, although the text content may be "I feel chest tightness and a little breathless in recent days", if the patient's voice shows obvious rapid, intermittent or low characteristics, it may indicate more serious respiratory or cardiovascular diseases. However, traditional automatic speech recognition and transcription systems only convert the voice into text without considering factors such as tone and speaking speed, which may cause doctors to mistakenly believe that the patient's condition is mild, affecting diagnostic decisions.

[0164] The speech analysis system of the present invention has been deployed in the outpatient department and emergency department of a tertiary hospital. The system accesses the voice interaction between doctors and patients, and records and analyzes the content of the conversation in real time. Unlike traditional automatic speech recognition systems, this system not only transcribes speech into text, but also uses multimodal feature fusion technology to combine pitch, speech rate, frequency, and energy speech features with text content to generate a more complete description of the patient's condition. When the patient describes his or her own symptoms, the system automatically extracts key information and calculates the emotional state based on the patient's voice features, such as anxiety, pain, weakness, etc., and displays the results to the doctor for reference when the doctor makes a diagnosis. In addition, the system, combined with the knowledge graph, can automatically match relevant medical cases, provide doctors with possible etiology analysis, and improve diagnostic efficiency.

[0165] During the three-month testing period, the system processed 150,000 patient voice data and evaluated key indicators.

[0166] Table 1 System performance comparison

[0167]

[0168] The completeness of the medical records was tested, and the completeness of the traditional manual recording method was 70.0%, mainly because doctors need to manually input data, which is prone to omissions or simplified descriptions. The speech transcription system based on automatic speech recognition has improved on this basis, with a completeness of 82.0%, but there is still a problem of partial information loss, especially when it involves multiple rounds of dialogue or complex medical terms. The system of the present invention uses multimodal fusion and knowledge graph enhancement to achieve a completeness of 95.5% for medical records, which is 25.5% higher than the traditional method, effectively reducing the omission of key information and improving the doctor's understanding of the patient's condition.

[0169] In terms of the accuracy of speech emotion analysis, the accuracy of the traditional automatic speech recognition speech transcription system is 58.0%, because the system only relies on text information and cannot recognize emotional signals in speech, such as pitch, speaking speed, pause patterns, etc. The system of the present invention combines speech feature analysis to capture the patient's anxiety, pain, confusion and other emotional states, increasing the accuracy to 90.0%, an increase of 32.0%, which helps doctors understand the patient's psychological state more accurately and make more appropriate communication and treatment decisions.

[0170] The accuracy of keyword extraction is a key indicator to measure the performance of speech analysis systems in automatically identifying the core information of patients' conditions. The accuracy of keyword extraction of traditional automatic speech recognition speech transcription systems is 68.0%. The main problem is that the system cannot correctly distinguish between medical terms and daily language, resulting in some key information being omitted or misidentified. The system of the present invention uses a multi-head attention mechanism to optimize keyword extraction, which increases the accuracy to 92.0%, an increase of 24.0% compared to traditional methods, ensuring that doctors can quickly obtain the core symptom description of patients and improve diagnostic efficiency.

[0171] In terms of the matching rate of diagnosis-assisted recommendations, the matching rate of the traditional automatic speech recognition speech transcription system is 72.0%, which is mainly limited by the system's insufficient understanding of medical knowledge and the inability to effectively associate patient symptoms with disease diagnosis. The system of the present invention combines knowledge graphs and optimizes the recommendation logic through context association and semantic reasoning, so that the matching rate reaches 94.0%, an increase of 22.0%, enabling doctors to obtain intelligent diagnosis suggestions based on patient descriptions more quickly, improving the accuracy and efficiency of clinical decision-making.

[0172] The time it takes for doctors to look up information is a key indicator for measuring how well a speech analysis system can improve doctors' work efficiency in a clinical setting. Traditional manual recording methods require doctors to search for patient medical history and current condition records in different systems, with an average lookup time of 7.5 minutes. The speech transcription system based on automatic speech recognition reduces some of the time through automatic transcription, reducing the average doctor's lookup time to 6.3 minutes. However, the system of the present invention further shortens the doctor's lookup time to 4.8 minutes through intelligent information organization and automatic summarization, which is 36.0% less than the traditional method, improving the doctor's work efficiency and enabling them to spend more time on patient communication and treatment plan formulation.

[0173] The waiting time for patients is an important indicator of the system's ability to optimize hospital resource management. The traditional manual recording method causes patients to wait an average of 32 minutes due to the long time it takes for doctors to record, check information, and make diagnoses. Although the traditional automatic speech recognition voice transcription system has reduced some of the recording time, the overall waiting time is still long, about 28 minutes. The system of the present invention reduces the average waiting time for patients to 22 minutes, a reduction of 31.3%, by optimizing medical records, intelligent recommendations, and automatic classification, significantly improving the hospital's operating efficiency and patients' medical experience.

[0174] Finally, in terms of computing resource consumption, the computing requirement benchmark of the traditional method is set at 100%, and the computing resource consumption of the ASR speech transcription system is reduced to 88.0% due to the speech-to-text task involved. The system of the present invention uses improved quantized low-rank adaptation and adapter fusion technology to optimize the large model, so that the computing resource consumption is further reduced to 60.0%, which is 40.0% less than the traditional method. While ensuring high accuracy, it optimizes computing efficiency, enabling the system to run efficiently in the existing server environment of the hospital without the need for additional expansion of high-performance computing equipment.

[0175] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A customer voice analysis system based on a large model, characterized in that: include: Voice data collection and preprocessing module, used to collect customer voice data and preprocess the customer voice data; A speech recognition module is used to recognize the pre-processed customer speech data and generate text data; The feature fusion module is used to extract the semantic features of text data and fuse the features of customer voice data to generate a fused voice-text representation; The large model fine-tuning module is used to fine-tune the parameters of the pre-trained Spark cognitive large model using the improved quantization low-rank adaptation method and the adapter fusion method, and use the error compensation method to reduce the impact of quantization error, calculate the gradient adjustment, and use the dynamic gradient adjustment strategy to correct when the error exceeds the threshold: ； ； in, represents the corrected gradient, represents the gradient of the parameter weight matrix update, represents the parameter weight matrix after inverse quantization, represents the error compensation coefficient, represents the error adjustment term, Represents the indicator function. When the error exceeds the threshold When the error correction is triggered, represents the quantization error threshold, Represents the dimension of the parameter weight matrix; The semantic analysis and emotion recognition module is used to process the fused speech-text representation, extract semantic information, identify the emotional state, and generate analysis results; The conversation context management module is used to associate customer historical voice data, establish conversation context information, and update voice analysis results.

2. A customer voice analysis system based on a large model according to claim 1, characterized in that: The modules are implemented in the following ways: S1. Collect customer voice data and pre-process the customer voice data; S2, recognize the pre-processed customer voice data and generate text data; S3, extracting semantic features of text data and obtaining features of customer voice data to generate fused voice-text representation; S4, using improved quantized low-rank adaptation method and adapter fusion method to fine-tune the parameters of the pre-trained Spark cognitive large model; S5. Use the Spark cognitive big model to process the fused speech and text representation, extract semantic information, identify emotional states, and generate speech analysis results; S6. Associate the customer's historical voice data, establish conversation context information, and update the voice analysis results.

3. A customer voice analysis system based on a large model according to claim 2, characterized in that: The customer voice data features include pitch, frequency, energy and speaking speed.

4. A customer voice analysis system based on a large model according to claim 2, characterized in that: The S4 comprises the following specific steps: S41. Constructing the parameter weight matrix of the Spark cognitive model ,right Perform low-rank decomposition and use dynamic low-rank optimization method to Perform optimal rank calculation, the parameter weight matrix Representation Dimensions: ； in, represents the parameter weight matrix, Represents the left matrix of low-rank decomposition, which is part of the weight after dimensionality reduction. , Represents the right matrix of the low-rank decomposition, which is responsible for restoring the complete weight information, , represents the low-rank dimension, which is the reduced parameter size, and is optimized using the least squares error minimization method: ； in, represents the parameter weight matrix OK, express No. OK, represents the norm, represents the dimension of the parameter weight matrix, represents the variable value that minimizes the function value; S42, quantize the parameter weight matrix using 4-bit quantization: ； in, represents the quantized parameter weight matrix, represents the minimum value in the parameter weight matrix, represents the maximum value in the parameter weight matrix, Indicates the number of quantization bits, Indicates that the value is rounded and integer; S43, using a group quantization method, independently quantizing the parameter weight matrices of different layers according to 4-bit quantization, and restoring the parameter weight matrices after group quantization: ； ； in, represents the parameter weight matrix after group quantization, represents the quantization function, which is the 4-bit quantization in step S42, Indicates The parameter weight matrix of the layer, Indicates the number of quantization bits, Represents the parameter weight matrix after inverse quantization; S44, using error compensation method to reduce the impact of quantization error, calculating gradient adjustment, using dynamic gradient adjustment strategy, and making corrections when the error exceeds the threshold: ； ； in, represents the corrected gradient, represents the gradient of the parameter weight matrix update, represents the parameter weight matrix after inverse quantization, represents the error compensation coefficient, represents the error adjustment term, Represents the indicator function. When the error exceeds the threshold When the error correction is triggered, represents the quantization error threshold, Represents the dimension of the parameter weight matrix; S45. Let the input feature be expressed as , the adapter layer parameter matrix is , calculate the adapted feature representation: ； in, represents the adapted feature representation, represents the input feature representation, represents the training parameters, represents the rectified linear unit activation function, represents the adapter layer parameter matrix; S46. Assume that the output probability distribution of the pre-trained Spark cognitive model is , the output probability distribution of the pre-trained Spark cognitive model after fine-tuning is , the Kullback-Leibler divergence is used to calculate the distillation loss: ； in, represents the distillation loss, represents the output probability distribution of the pre-trained Spark cognitive model, represents the output probability distribution of the pre-trained Spark Cognitive Big Model after fine-tuning, Represents the category index, Represents input data; S47, using distillation temperature parameters Smooth the output probability distribution of the pre-trained Spark Cognitive Big Model: ； in, represents the logits value of the pre-trained Spark cognitive model, represents the distillation temperature parameter, represents the natural exponential function, Represents a variable when traversing all categories, Represents the current time step.

5. A customer voice analysis system based on a large model according to claim 2, characterized in that: The S5 comprises the following specific steps: S51, assuming that the fused speech text is represented by The fused speech-text representation includes the semantic features of the text data: And the customer voice data features are , and They are the dimensions of the semantic features of text data and the features of customer voice data: ； in, represents the fused speech-text representation, represents the vector concatenation operation, Represents the semantic features of text data, Indicates the characteristics of customer voice data; S52. Use the embedding layer of the Spark cognitive model to encode the fused speech-text representation: ； in, Indicates the hidden state, represents the embedding matrix, which is determined by the pre-trained weights of the Spark Cognitive Big Model; S53. Use the multi-head attention mechanism to calculate the weight distribution of each input vector and calculate the attention score: ； ； in, Indicates The output of an attention head is represents normalization, , and vectors representing queries, keys, and values, , and weight matrices representing queries, keys, and values, Represents the query vector The dimension of represents the numerical stability correction term; S54, using the multi-layer Transformer encoder of the Spark cognitive large model to perform multi-head attention calculation on the hidden state: ； in, Indicates the current layer hidden state, represents the previous hidden state, Representation layer normalization, Represents multi-head attention calculation; S55, using gated network to optimize attention output: ； ； in, Indicates The gating factor of the layer, represents the activation function, represents the gating weight matrix, Indicates the current layer hidden state, represents the bias term, represents the optimized hidden state, Represents the hidden state of the previous layer; S56. The output layer of the Spark Cognitive Model is used to calculate the sentiment classification score, and the Softmax function is used to calculate the probability distribution of the sentiment category: ； ； in, represents the sentiment category score, represents the classification bias vector, represents the classification weight matrix, represents the probability distribution of sentiment categories, Represents the sentiment category index; S57, extracting keywords using an attention-guided keyword extraction method, and selecting keywords using an adaptive threshold method: ； ； in, Indicates The total attention weight of words, represents the keyword extraction transformation matrix, represents the rectified linear unit activation function, Indicates The attention score of each word, Indicates The hidden state of the layer, represents the number of layers of hidden states, represents the dynamic threshold, represents the mean value of keyword weight, Indicates the standard deviation of keyword weights, represents the adjustment factor; Finally, we get the keyword set : ； in, Indicates Keywords; S58. Combine sentiment classification results with keyword extraction results to generate structured data: ； in, Represents structured data.

Images